Intriguing error with Git 1.6.3.1: Too many open files

We use Git (1.6.3.1) exclusively here at Box.

Starting this afternoon, we started getting errors when pushing to one of the repos. It is a very active, bare shared repo, shared among multiple developers. The error is:

*"error: unable to open object pack directory: ./objects/pack: Too many open files"

Things I tried/observed to resolve it:

1) ulimit -H = unlimited, when run as root (original error seems
misleading?). I also killed other proceses that had some open files, and
brought down the total count of open files (roughly lsof | wc -l) to <1000.

2) I can push to several other repos on the same machine just fine

3) git fsck, git gc, git-branch and git-status return the same error

4) I copied the repo (using scp -r, git-clone failed) to a different
machine, but got the same errors when pushing to it

5) I asked #git and the suggestion was that the repo is somehow corrupt and
I should re-push to it

One more thing worth mentioning: Gerrit (code review) uses the same repo (lots of stuff in refs/changes)

If the repo is indeed not recoverable, I know how to restore it. But I'd rather fix this if possible. Wondering if it is worth pursuing and what might be wrong?

Thanks. -Soham

Hi,

Soham Mehta wrote:

Do current git versions (v1.7.4.2 or later) behave the same way?

Sorry for the trouble, Jonathan

On Tue, Jan 10, 2012 at 7:09 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:

I haven't played with versions yet. Are you alluding that #2 on this list may have addressed it? https://raw.github.com/gitster/git/master/Documentation/RelNotes/1.7.4.2.txt

 * We used to keep one file descriptor open for each and every packfile
   that we have a mmap window on it (read: "in use"), even when for very
   tiny packfiles.  We now close the file descriptor early when the entire
   packfile fits inside one mmap window.

The corrupt repo does have a 1000+ pack files, anywhere from 650M to 2.5K.

Soham Mehta wrote:

For almost all purposes, the latest stable version should work fine (v1.7.8.3 at the moment). The people on this list tend to be very keen to hear about cases where that's not true --- if people are stuck using old versions because of regressions, that means we're failing.

Not exactly. I was thinking of commit c7934306 (Limit file descriptors used by packs, 2011-02-28).

In git time, 1.6.3.1 is ancient history. There are many little fixes and performance improvements you are missing out on.

Just my two cents, Jonathan