threads / discuss / 14222

Corruption: empty refs/heads in otherwise filled repo: cannot clone?

Subject: Corruption: empty refs/heads in otherwise filled repo: cannot clone?

## tl;dr

10 messages between Jun 30, 2008 and Jun 30, 2008.

replies: 9people: 4as markdown or json

Jan Wielemaker· Jun 30, 2008, 09:49 UTC · lore
Hi,

I'm a bit puzzled. I have a bare repository, somehow without any files in refs/heads.

It started using conversion from CVS via SVN using git-1.5.3.4. It is configured for group access and there are a number of SSH accounts on the machine in the proper group using git-shell for fellow developers.

I never have any problems, but after upgrading to git-git (1.5.6.rc3) fellow users get error:

error: no such remote ref refs/heads/JPL31 error: no such remote ref refs/heads/V57X error: no such remote ref refs/heads/V5_4_patches error: no such remote ref refs/heads/XML_UNICODE error: no such remote ref refs/heads/attvar error: no such remote ref refs/heads/gmp

Creating a new account and doing a fresh clone I even got the fatal error there are no remote branches!? Still, using my own user id, I could do anything I wanted. I checked permissions to see whether there are differences between group and owner permissions, but couldn't find anything suspicious. git fsck --full on the repo gives no errors.

It turns out the directory refs/heads is empty!?

I made a fresh clone of the repo as myself, creating all references nicely in refs/remotes/origin. Then I copied these files to refs/heads into the main bare repository and now all appears to work nicely again. Still, this is a bit worrying ... My questions:

    * Is my work-around safe and sound?
    * How can this have happened?
    * Why can I clone as myself and not as anyone else!?  Stranger:
    after a "sudo -u <someone> /bin/bash" I could not clone.  I can
    copy the entire tree using cp -a, but I still cannot clone the
    copy with this user, while as myself I can clone the copy. 
    Somehow `as myself' appears to get the branchheads from somewhere.
    There is only one copy of git installed, so both users use the
    same.  What can cause this difference?
	Thanks --- Jan
Jakub Narebski· Jun 30, 2008, 10:10 UTC · re: Jan Wielemaker · lore

Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?

Jan Wielemaker <J.Wielemaker@uva.nl> writes:
> I'm a bit puzzled. I have a bare repository, somehow without any files
> in refs/heads.

Do you have .git/packed-refs file? How do you have gc.packrefs set ("git config --get gc.packrefs")? What are the git version on the clients that have trouble accessing repository, and what protocol (transport merhod) do they use: "file", http(s)/ftp, git, ssh?

-- 
Jakub Narebski
Poland
ShadeHawk on #git
Jan Wielemaker· Jun 30, 2008, 10:30 UTC · re: Jakub Narebski · lore

Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?

Hi Jakub,
Thanks for the quick reply.  Info:
On Monday 30 June 2008 12:10, Jakub Narebski wrote:
Show 5 quoted lines
> Jan Wielemaker <J.Wielemaker@uva.nl> writes:
> > I'm a bit puzzled. I have a bare repository, somehow without any files
> > in refs/heads.
>
> Do you have .git/packed-refs file? How do you have gc.packrefs set

Its a bare repo, so I'll forget the .git. Yes, I have that file and it contains nice references, I checked a few by hand, and they have the same SHA1 as the files I copied.

> ("git config --get gc.packrefs")? What are the git version on the

This gives no output, so I guess the answer is 'no'. Is that wrong? This is indeed the case (config):

----------------------------------------------------------------
[core]
        repositoryformatversion = 0
        filemode = true
        bare = true
        sharedrepository = 1
[gitcvs]
        enabled = 1
        logfile = /home/git/pl.git/cvs.log
        allbinary = 1
----------------------------------------------------------------
> clients that have trouble accessing repository, and what protocol
> (transport merhod) do they use: "file", http(s)/ftp, git, ssh?

I could reproduce the problem using the same git that maintains the bare repository, which started as git-1.5.3.4 and is now 1.5.6.rc3. It reproduces both using local file access and ssh. Other people have a variety of versions. I know of concrete problems using the stable 1.5.5 over ssh.

The problem that I can't even clone is indeed after running a git-gc.

So, now I have all branch head refs double (in packed-refs and in refs/heads). That explains ar least where they come from, but it fails to explain why doing exactly the same using the same git on the same repo as two users yields a different result. Of course, besides the desire to understand how his works, I'm mostly interested in how to fix this :-)

	Thanks --- Jan
Jakub Narebski· Jun 30, 2008, 11:26 UTC · re: Jan Wielemaker · lore

Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?

Jan Wielemaker wrote:
Show 11 quoted lines
> On Monday 30 June 2008 12:10, Jakub Narebski wrote:
>> Jan Wielemaker <J.Wielemaker@uva.nl> writes:
>>>
>>> I'm a bit puzzled. I have a bare repository, somehow without any files
>>> in refs/heads.
>>
>> Do you have .git/packed-refs file?
> 
> Its a bare repo, so I'll forget the .git.  Yes, I have that file and
> it contains nice references,  I checked a few by hand, and they have
> the same SHA1 as the files I copied.

That is where the refs are instead of being in individual files under refs/ - those are so called "packed refs".

  When a ref is  missing from  the traditional $GIT_DIR/refs hierarchy,
  it is looked up in $GIT_DIR/packed-refs and used if found.

I'm not sure why branches are also packed, because git used to pack only tags and refs which were packed already.

>> How do you have gc.packrefs set ("git config --get gc.packrefs")?
> 
> This gives no output, so I guess the answer is 'no'.
In git-config(1) you can find the following:
  gc.packrefs::
        `git gc` does not run `git pack-refs` in a bare repository by
        default so that older dumb-transport clients can still fetch
        from the repository.  Setting this to `true` lets `git
        gc` to run `git pack-refs`.  Setting this to `false` tells
        `git gc` never to run `git pack-refs`. The default setting is
        `notbare`. Enable it only when you know you do not have to
        support such clients.  The default setting will change to `true`
        at some stage, and setting this to `false` will continue to
        prevent `git pack-refs` from being run from `git gc`.

Unless something changed (and git Documentation was not updated) git should not pack refs by default.

Show 11 quoted lines
>> What are the git version on the
>> clients that have trouble accessing repository, and what protocol
>> (transport method) do they use: "file", http(s)/ftp, git, ssh?
> 
> I could reproduce the problem using the same git that maintains the bare
> repository, which started as git-1.5.3.4 and is now 1.5.6.rc3. It
> reproduces both using local file access and ssh. Other people have a
> variety of versions. I know of concrete problems using the stable 1.5.5
> over ssh.
>
> The problem that I can't even clone is indeed after running a git-gc.
Hmmm... strange. I know that there can be problems with older clients
(those which do not understand packed-refs format) accessing
repositories with packed refs (without loose refs) via "dumb"
protocols.  Problems with accessing repositories locally and via ssh
might point to some troubles with permissions and ownership of
$GIT_DIR and $GIT_DIR/packed-refs.
 
git-gc can pack refs, that is what running it causes problems.
Show 6 quoted lines
> So, now I have all branch head refs double (in packed-refs and in
> refs/heads). That explains ar least where they come from, but it fails
> to explain why doing exactly the same using the same git on the same
> repo as two users yields a different result. Of course, besides the
> desire to understand how his works, I'm mostly interested in how to fix
> this :-)

Unfortunately as it is not caused by the issue I thought about I cannot help you further...

-- 
Jakub Narebski
Poland
Jan Wielemaker· Jun 30, 2008, 11:44 UTC · re: Jakub Narebski · lore

Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?

Hi Jakub,

Summarising, I think the conclusion is that git pack-refs has somehow been run on this repository, and being a bare one this is not a particulary good idea right now. I have the impression I should `unpack' them by putting the appriate files in heads (done) and tags (now still) and (re)move packed-refs.

	Cheers --- Jan
On Monday 30 June 2008 13:26, Jakub Narebski wrote:
Show 71 quoted lines
> Jan Wielemaker wrote:
> > On Monday 30 June 2008 12:10, Jakub Narebski wrote:
> >> Jan Wielemaker <J.Wielemaker@uva.nl> writes:
> >>> I'm a bit puzzled. I have a bare repository, somehow without any files
> >>> in refs/heads.
> >>
> >> Do you have .git/packed-refs file?
> >
> > Its a bare repo, so I'll forget the .git.  Yes, I have that file and
> > it contains nice references,  I checked a few by hand, and they have
> > the same SHA1 as the files I copied.
>
> That is where the refs are instead of being in individual files under
> refs/ - those are so called "packed refs".
>
>   When a ref is  missing from  the traditional $GIT_DIR/refs hierarchy,
>   it is looked up in $GIT_DIR/packed-refs and used if found.
>
> I'm not sure why branches are also packed, because git used to pack
> only tags and refs which were packed already.
>
> >> How do you have gc.packrefs set ("git config --get gc.packrefs")?
> >
> > This gives no output, so I guess the answer is 'no'.
>
> In git-config(1) you can find the following:
>
>   gc.packrefs::
>         `git gc` does not run `git pack-refs` in a bare repository by
>         default so that older dumb-transport clients can still fetch
>         from the repository.  Setting this to `true` lets `git
>         gc` to run `git pack-refs`.  Setting this to `false` tells
>         `git gc` never to run `git pack-refs`. The default setting is
>         `notbare`. Enable it only when you know you do not have to
>         support such clients.  The default setting will change to `true`
>         at some stage, and setting this to `false` will continue to
>         prevent `git pack-refs` from being run from `git gc`.
>
> Unless something changed (and git Documentation was not updated) git
> should not pack refs by default.
>
> >> What are the git version on the
> >> clients that have trouble accessing repository, and what protocol
> >> (transport method) do they use: "file", http(s)/ftp, git, ssh?
> >
> > I could reproduce the problem using the same git that maintains the bare
> > repository, which started as git-1.5.3.4 and is now 1.5.6.rc3. It
> > reproduces both using local file access and ssh. Other people have a
> > variety of versions. I know of concrete problems using the stable 1.5.5
> > over ssh.
> >
> > The problem that I can't even clone is indeed after running a git-gc.
>
> Hmmm... strange. I know that there can be problems with older clients
> (those which do not understand packed-refs format) accessing
> repositories with packed refs (without loose refs) via "dumb"
> protocols.  Problems with accessing repositories locally and via ssh
> might point to some troubles with permissions and ownership of
> $GIT_DIR and $GIT_DIR/packed-refs.
>
> git-gc can pack refs, that is what running it causes problems.
>
> > So, now I have all branch head refs double (in packed-refs and in
> > refs/heads). That explains ar least where they come from, but it fails
> > to explain why doing exactly the same using the same git on the same
> > repo as two users yields a different result. Of course, besides the
> > desire to understand how his works, I'm mostly interested in how to fix
> > this :-)
>
> Unfortunately as it is not caused by the issue I thought about I cannot
> help you further...
Jakub Narebski· Jun 30, 2008, 12:03 UTC · re: Jan Wielemaker · lore

Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?

Jan Wielemaker wrote:
Show 5 quoted lines
> Summarising, I think the conclusion is that git pack-refs has somehow
> been run on this repository, and being a bare one this is not a
> particulary good idea right now. I have the impression I should `unpack'
> them by putting the appriate files in heads (done) and tags (now still)
> and (re)move packed-refs.

If you use new enough git both on server and on clients it should not have problems with packed-refs. I would rather check permissions of $GIT_DIR and $GIT_DIR/packed-refs.

If "git show-ref" and "git for-each-ref" works, then "git ls-remote <repo>" should work, and git-fetch/git-clone shoulw work too...

-- 
Jakub Narebski
Poland
Jan Wielemaker· Jun 30, 2008, 12:20 UTC · re: Jakub Narebski · lore

Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?

On Monday 30 June 2008 14:03, Jakub Narebski wrote:
Show 10 quoted lines
> Jan Wielemaker wrote:
> > Summarising, I think the conclusion is that git pack-refs has somehow
> > been run on this repository, and being a bare one this is not a
> > particulary good idea right now. I have the impression I should `unpack'
> > them by putting the appriate files in heads (done) and tags (now still)
> > and (re)move packed-refs.
>
> If you use new enough git both on server and on clients it should
> not have problems with packed-refs. I would rather check permissions
> of $GIT_DIR and $GIT_DIR/packed-refs.

There is no permission problem, as a I proved by doing a recursive copy of the whole repo (cp -a, no errors) and the problem prevails on the copy. A serious scan for permission errors was my first step. Almost looks like something in the environment, but I can't find anything weird there either.

> If "git show-ref" and "git for-each-ref" works, then
> "git ls-remote <repo>" should work, and git-fetch/git-clone
> shoulw work too...

Thanks for the pointers. I'll leave it for now (busy ...). I now understand how it is supposed to work. If I can find time at some point I'll run strace in different setups to see whether I can spot the difference.

Anyway, thanks to your help I am fairly confident my repo is in good shape again.

	Cheers --- Jan
Daniel Barkalow· Jun 30, 2008, 19:33 UTC · re: Jan Wielemaker · lore

Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?

On Mon, 30 Jun 2008, Jan Wielemaker wrote:
Show 17 quoted lines
> On Monday 30 June 2008 14:03, Jakub Narebski wrote:
> > Jan Wielemaker wrote:
> > > Summarising, I think the conclusion is that git pack-refs has somehow
> > > been run on this repository, and being a bare one this is not a
> > > particulary good idea right now. I have the impression I should `unpack'
> > > them by putting the appriate files in heads (done) and tags (now still)
> > > and (re)move packed-refs.
> >
> > If you use new enough git both on server and on clients it should
> > not have problems with packed-refs. I would rather check permissions
> > of $GIT_DIR and $GIT_DIR/packed-refs.
> 
> There is no permission problem, as a I proved by doing a recursive copy
> of the whole repo (cp -a, no errors) and the problem prevails on the
> copy. A serious scan for permission errors was my first step. Almost
> looks like something in the environment, but I can't find anything weird
> there either.

That's a "cp -a" as somebody else, I assume? (If it were as you or root, you'd generate a copy of the repository with any permission problem unchanged, since -a includes -p.)

	-Daniel
*This .sig left intentionally blank*
Jan Wielemaker· Jun 30, 2008, 19:46 UTC · re: Daniel Barkalow · lore

Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?

On Monday 30 June 2008 21:33:50 Daniel Barkalow wrote:
Show 22 quoted lines
> On Mon, 30 Jun 2008, Jan Wielemaker wrote:
> > On Monday 30 June 2008 14:03, Jakub Narebski wrote:
> > > Jan Wielemaker wrote:
> > > > Summarising, I think the conclusion is that git pack-refs has somehow
> > > > been run on this repository, and being a bare one this is not a
> > > > particulary good idea right now. I have the impression I should
> > > > `unpack' them by putting the appriate files in heads (done) and tags
> > > > (now still) and (re)move packed-refs.
> > >
> > > If you use new enough git both on server and on clients it should
> > > not have problems with packed-refs. I would rather check permissions
> > > of $GIT_DIR and $GIT_DIR/packed-refs.
> >
> > There is no permission problem, as a I proved by doing a recursive copy
> > of the whole repo (cp -a, no errors) and the problem prevails on the
> > copy. A serious scan for permission errors was my first step. Almost
> > looks like something in the environment, but I can't find anything weird
> > there either.
>
> That's a "cp -a" as somebody else, I assume? (If it were as you or root,
> you'd generate a copy of the repository with any permission problem
> unchanged, since -a includes -p.)

I'm an old time Unix guy :-). The failing user has a git-shell. I did (reconstructing from memory):

	% sudo -u failinguser /bin/bash
	<passwd>
	% Use whoami to validate I was this user
	% git clone /home/git/pl.git
	<failed (no remote branches)>
	% cp -a /home/git/pl.git tmp.git
	<ok, no errors>
	% git clone tmp.git
	<failed (no remote branches)>

Funny enough, as myself I can clone tmp.git!? I have only one git installation in /usr/local/bin.

	Cheers --- Jan
Björn Steinbrink· Jun 30, 2008, 11:50 UTC · re: Jakub Narebski · lore

Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?

On 2008.06.30 13:26:10 +0200, Jakub Narebski wrote:
Show 41 quoted lines
> Jan Wielemaker wrote:
> > On Monday 30 June 2008 12:10, Jakub Narebski wrote:
> >> Jan Wielemaker <J.Wielemaker@uva.nl> writes:
> >>>
> >>> I'm a bit puzzled. I have a bare repository, somehow without any files
> >>> in refs/heads.
> >>
> >> Do you have .git/packed-refs file?
> > 
> > Its a bare repo, so I'll forget the .git.  Yes, I have that file and
> > it contains nice references,  I checked a few by hand, and they have
> > the same SHA1 as the files I copied.
> 
> That is where the refs are instead of being in individual files under
> refs/ - those are so called "packed refs".
> 
>   When a ref is  missing from  the traditional $GIT_DIR/refs hierarchy,
>   it is looked up in $GIT_DIR/packed-refs and used if found.
> 
> I'm not sure why branches are also packed, because git used to pack
> only tags and refs which were packed already.
> 
> >> How do you have gc.packrefs set ("git config --get gc.packrefs")?
> > 
> > This gives no output, so I guess the answer is 'no'.
> 
> In git-config(1) you can find the following:
> 
>   gc.packrefs::
>         `git gc` does not run `git pack-refs` in a bare repository by
>         default so that older dumb-transport clients can still fetch
>         from the repository.  Setting this to `true` lets `git
>         gc` to run `git pack-refs`.  Setting this to `false` tells
>         `git gc` never to run `git pack-refs`. The default setting is
>         `notbare`. Enable it only when you know you do not have to
>         support such clients.  The default setting will change to `true`
>         at some stage, and setting this to `false` will continue to
>         prevent `git pack-refs` from being run from `git gc`.
> 
> Unless something changed (and git Documentation was not updated) git
> should not pack refs by default.

I had a quick glance over the log for builtin-gc.c, and that changed last year in:

56752391 Make "git gc" pack all refs by default
Björn - who lacks time to provide a doc update patch :-(

← back to recent threads