Re: Funny error with git gc...

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: May 15, 2009, 18:46 UTC
Message-ID: <alpine.LFD.2.01.0905151128230.3343@localhost.localdomain>
In-Reply-To: <7vmy9etey8.fsf@alter.siamese.dyndns.org>

On Fri, 15 May 2009, Junio C Hamano wrote:

Show 12 quoted lines

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> > if you need a chuckle, like me, you might appreciate this story: in one of 
> > my repositories, "git gc" dies with
> >
> > 	unable to open object pack directory: ...: Too many open files
> >
> > turns out that there are a whopping 1088 packs in that repository...
> 
> Isn't it a more serious problem than a mere chuckle?  How would one
> recover from such a situation (other than "mv .git/objects/pack-*;
> for p in pack-*.pack; do git unpack-objects <$p; done")?

Well, you can probably just increase the file limits and try again. Depending on setup, you may need root to do so, though.

I also think you _should_ be able to avoid this by just limiting the pack size usage. IOW, with some packed_git_limit, something like

	[core]
		packedGitWindowSize = 16k
		packedGitLimit = 1M

you should hopefully be able to repack (slowly) even with a low file descriptor limit, because of the total limit on the size.

That said, I do agree that ulimit doesn't always work on all systems (whether due to hard system limits or due to not having permission to raise the limits), and playing games with pack limits is non-obvious. We should really try to avoid getting into such a situation. But I think git by default avoids it by the auto-gc, no? So you have to disable that explicitly to get into this bad situation.

One solution - which I think may be the right one regardless - is to not use "mmap()" for small packs or small SHA1 files.

mmap is great for random-access multi-use scenarios (and to avoid some memory pressure by allowing sharing of pages), but for anything that is just a couple of pages in size, mmap() just adds big overhead with little upside.

So if we use malloc+read for small things, we'd probably avoid this. Now, if you have a few thousand _large_ packs, you'd still be screwed, but the most likely reason for having a thousand packfiles is that you did daily "git pull"s, and have lots and lots of packs that are pretty small.

Dscho? What are your pack-file statistics in this case?

		Linus