[RFC PATCH] test showing EMFILE error with too many packs

From: Pete Wyckoff <pw@padd.com>
Date: Aug 13, 2011, 20:25 UTC
Message-ID: <20110813202521.GA9958@arf.padd.com>
In-Reply-To: <20110812172600.GA15896@padd.com>

pw@padd.com wrote on Fri, 12 Aug 2011 10:26 -0700:

Show 37 quoted lines

> spearce@spearce.org wrote on Fri, 12 Aug 2011 09:09 -0700:
> > On Fri, Aug 12, 2011 at 08:15, Pete Wyckoff <pw@padd.com> wrote:
> > > Somebody at $work found this problem:
> > >
> > >    $ git ls-files -s | wc
> > >    error: packfile .git/objects/pack/pack-1627e77da82bbb36118762649c8aa88c05664b1e.pack cannot be accessed
> > >    [..lots more similar errors..]
> > >
> > > Turns out his shell's open file descriptor limit was 500.  And
> > > there are 1600 pack files in the repo.
> > >
> > > Increasing the descriptor limit to 1024 fixed it.  I could
> > > probably get him to repack, which may also fix it.
> > >
> > > Does it seem feasible to look for EMFILE errors and close
> > > some packs?  Or at least spit out a more intuitive error?
> > 
> > What version of Git? I remember fixing this already.... :-)
> 
> Initially 1.7.5.4.  Same problem on 1.7.6 and master.  
> 
> I have your "Limit file descriptors used by packs" (c793430, 28
> feb 2011).
> 
> It fails here:
> 
> 	if (!is_pack_valid(p)) {
> 		error("packfile %s cannot be accessed", p->pack_name);
> 		goto next;
> 	}
> 
> because p->pack_fd is -1, because an earlier git_open_noatime()
> got EMFILE.  The function unuse_one_window() is never able to
> find anything to close.
> 
> I'll do some more debugging this weekend.  Thanks for pointing
> out that it _should_ be fixed.

Here's a test case (on master). It is easy to repack the repository to avoid this situation, so I'm not convinced this is a bug.

But it would be nice at least to report what's going on when we run out of file descriptors.

The current error is:

    $ ../git commit -m fileN
    error: packfile .git/objects/pack/pack-7bff59b72a19f371ccd3e4ffefaac6cd1e07c6ed.pack cannot be accessed
    error: invalid object 100644 a7d02de3336c72e6d2d24f382b82195dbf625404 for 'file47'
    error: Error building trees

		-- Pete

--------8<--------

From: Pete Wyckoff <pw@padd.com>
Date: Sat, 13 Aug 2011 15:57:55 -0400
Subject: [RFC PATCH] test showing EMFILE error with too many packs

In a repository with too many pack files, it is possible to have too many open at once and exceed the process file descriptor limit.

This test shows one failure with an artificially low limit of 50. The same failure happens at a "normal" value like 1024, but takes longer.

Signed-off-by: Pete Wyckoff <pw@padd.com>
---
 t/t5308-pack-emfile.sh |   29 +++++++++++++++++++++++++++++
 1 files changed, 29 insertions(+), 0 deletions(-)
 create mode 100755 t/t5308-pack-emfile.sh

diff --git a/t/t5308-pack-emfile.sh b/t/t5308-pack-emfile.sh
new file mode 100755
index 0000000..66c62a7
--- /dev/null
+++ b/t/t5308-pack-emfile.sh
@@ -0,0 +1,29 @@
+#!/bin/sh
+
+test_description='repos with too many packs should not fail'
+
+. ./test-lib.sh
+
+test_expect_success setup '
+	for i in {1..60}
+	do
+		echo "$i" >"file$i" &&
+		git add "file$i" &&
+		test_tick &&
+		git commit -q -m "$i" &&
+		git repack -q
+	done
+	git prune-packed -q
+'
+
+# run in a subshell since limits cannot be increased
+test_expect_failure 'commit fail with too many packs' '
+	(
+	ulimit -n 50 &&
+	>"fileN" &&
+	git add "fileN" &&
+	git commit -q -m "fileN"
+	)
+'
+
+test_done

-- 
1.7.5.4