Re: [BUG] Incorrect/misleading error when `git rev-list --objects --all` hits the max open files limit

From: Pete Wyckoff <pw@padd.com>
Date: Mar 5, 2013, 15:26 UTC
Message-ID: <20130305152625.GA20562@padd.com>
In-Reply-To: <20130303232927.GA16606@rabbit.us>

rabbit@rabbit.us wrote on Mon, 04 Mar 2013 10:29 +1100:

Show 35 quoted lines

> I was tinkering with a massive git repository (actually a bup 
> repository, but it is a standard valid git repo underneath). While 
> validating that a repack ran succesfully I executed the command:
> 
>  git rev-list --objects --all > rev.list
> 
> And got back this:
> 
> error: packfile ./objects/pack/pack-d9808b7515419737806d0c621a0a1910f71c5cba.pack cannot be accessed
> fatal: missing blob object '27a8cf44da85b958aef2b5074931e7913e08ae95'
> 
> Several hours later after successful fsck, and after chasing down trees 
> blobs etc, I realized that the problem is too many open files. The hint 
> came from ls-tree which lists the correct error (among a lot of spurious 
> junk):
> 
>  git ls-tree -r c636a5f51d4e > /dev/null
> 
>  error: packfile ./objects/pack/pack-d9808b7515419737806d0c621a0a1910f71c5cba.pack cannot be accessed
>  error: packfile ./objects/pack/pack-841e375f5e6c793a52fd1a3a2aea0b76219c4cdd.pack cannot be accessed
>  error: packfile ./objects/pack/pack-e67d9bf75e0840fc6113170b314d2d5a32cbb45a.pack cannot be accessed
>  error: packfile ./objects/pack/pack-b8fd8f083461c391fe6ec396840c328620d912e2.pack cannot be accessed
>  error: packfile ./objects/pack/pack-d9808b7515419737806d0c621a0a1910f71c5cba.pack cannot be accessed
>  error: packfile ./objects/pack/pack-804e0fadf56e2a165c157ef257620369adeea595.pack cannot be accessed
>  error: unable to open object pack directory: ./objects/pack: Too many open files
>  error: packfile ./objects/pack/pack-804e0fadf56e2a165c157ef257620369adeea595.pack cannot be accessed
>  error: Could not read 32a050cb7e54a1e817d135d25ab251480e8d9e3c
> 
> Failure to report the correct message verified with git 1.7.2.5 and 
> 1.8.2 (debian testing and experimental).
> 
> I hope this is sufficient description to address the underlying issue. I 
> will keep the un-repacked "many files" repo around just in case you need 
> more info/testing (though lowering the ulimit works equally well on 
> normal-size repos).

I've run into this twice:

    1.  user with restrictive ulimit on #files.
    2.  forgot to do "git gc" after regular fast-imports

Here's one thread:

    http://thread.gmane.org/gmane.comp.version-control.git/179231/focus=179292

I've got a patch in cold storage. It's not beautiful because there are too many paths that can end up hitting EMFILE. I'll dust it off and see if it is worth considering.

		-- Pete