{"thread":{"id":"28079","subject":"open files limit","startedAt":"2011-08-12T15:15:48Z","lastAt":"2011-08-13T20:25:21Z","messageCount":4,"participants":["Pete Wyckoff","Shawn Pearce"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"173402","messageId":"20110812151548.GA14385@padd.com","threadId":"28079","inReplyTo":null,"subject":"open files limit","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2011-08-12T15:15:48Z","receivedAt":"2011-08-12T15:15:48Z","isPatch":false,"sender":{"key":"pw@padd.com","avatar":null},"body":"Somebody at $work found this problem:\n\n $ git ls-files -s | wc\n error: packfile .git/objects/pack/pack-1627e77da82bbb36118762649c8aa88c05664b1e.pack cannot be accessed\n [..lots more similar errors..]\n\nTurns out his shell's open file descriptor limit was 500. And\nthere are 1600 pack files in the repo.\n\nIncreasing the descriptor limit to 1024 fixed it. I could\nprobably get him to repack, which may also fix it.\n\nDoes it seem feasible to look for EMFILE errors and close\nsome packs? Or at least spit out a more intuitive error?\n\n\t\t-- Pete\n"},{"id":"173405","messageId":"CAJo=hJsj8_VjD5wN9Gge_Me-eXKK-P7nLAxERiiLp0+ayiEBbg@mail.gmail.com","threadId":"28079","inReplyTo":"20110812151548.GA14385@padd.com","subject":"Re: open files limit","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-08-12T16:09:53Z","receivedAt":"2011-08-12T16:09:53Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Fri, Aug 12, 2011 at 08:15, Pete Wyckoff wrote:\n> Somebody at $work found this problem:\n>\n>    $ git ls-files -s | wc\n>    error: packfile .git/objects/pack/pack-1627e77da82bbb36118762649c8aa88c05664b1e.pack cannot be accessed\n>    [..lots more similar errors..]\n>\n> Turns out his shell's open file descriptor limit was 500.  And\n> there are 1600 pack files in the repo.\n>\n> Increasing the descriptor limit to 1024 fixed it.  I could\n> probably get him to repack, which may also fix it.\n>\n> Does it seem feasible to look for EMFILE errors and close\n> some packs?  Or at least spit out a more intuitive error?\n\nWhat version of Git? I remember fixing this already.... :-)\n\n-- \nShawn.\n"},{"id":"173409","messageId":"20110812172600.GA15896@padd.com","threadId":"28079","inReplyTo":"CAJo=hJsj8_VjD5wN9Gge_Me-eXKK-P7nLAxERiiLp0+ayiEBbg@mail.gmail.com","subject":"Re: open files limit","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2011-08-12T17:26:00Z","receivedAt":"2011-08-12T17:26:00Z","isPatch":false,"sender":{"key":"pw@padd.com","avatar":null},"body":"spearce@spearce.org wrote on Fri, 12 Aug 2011 09:09 -0700:\n> On Fri, Aug 12, 2011 at 08:15, Pete Wyckoff wrote:\n> > Somebody at $work found this problem:\n> >\n> >    $ git ls-files -s | wc\n> >    error: packfile .git/objects/pack/pack-1627e77da82bbb36118762649c8aa88c05664b1e.pack cannot be accessed\n> >    [..lots more similar errors..]\n> >\n> > Turns out his shell's open file descriptor limit was 500.  And\n> > there are 1600 pack files in the repo.\n> >\n> > Increasing the descriptor limit to 1024 fixed it.  I could\n> > probably get him to repack, which may also fix it.\n> >\n> > Does it seem feasible to look for EMFILE errors and close\n> > some packs?  Or at least spit out a more intuitive error?\n> \n> What version of Git? I remember fixing this already.... :-)\n\nInitially 1.7.5.4. Same problem on 1.7.6 and master. \n\nI have your \"Limit file descriptors used by packs\" (c793430, 28\nfeb 2011).\n\nIt fails here:\n\n\tif (!is_pack_valid(p)) {\n\t\terror(\"packfile %s cannot be accessed\", p->pack_name);\n\t\tgoto next;\n\t}\n\nbecause p->pack_fd is -1, because an earlier git_open_noatime()\ngot EMFILE. The function unuse_one_window() is never able to\nfind anything to close.\n\nI'll do some more debugging this weekend. Thanks for pointing\nout that it _should_ be fixed.\n\n\t\t-- Pete\n"},{"id":"173462","messageId":"20110813202521.GA9958@arf.padd.com","threadId":"28079","inReplyTo":"20110812172600.GA15896@padd.com","subject":"[RFC PATCH] test showing EMFILE error with too many packs","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2011-08-13T20:25:21Z","receivedAt":"2011-08-13T20:25:21Z","isPatch":true,"sender":{"key":"pw@padd.com","avatar":null},"body":"pw@padd.com wrote on Fri, 12 Aug 2011 10:26 -0700:\n> spearce@spearce.org wrote on Fri, 12 Aug 2011 09:09 -0700:\n> > On Fri, Aug 12, 2011 at 08:15, Pete Wyckoff wrote:\n> > > Somebody at $work found this problem:\n> > >\n> > >    $ git ls-files -s | wc\n> > >    error: packfile .git/objects/pack/pack-1627e77da82bbb36118762649c8aa88c05664b1e.pack cannot be accessed\n> > >    [..lots more similar errors..]\n> > >\n> > > Turns out his shell's open file descriptor limit was 500.  And\n> > > there are 1600 pack files in the repo.\n> > >\n> > > Increasing the descriptor limit to 1024 fixed it.  I could\n> > > probably get him to repack, which may also fix it.\n> > >\n> > > Does it seem feasible to look for EMFILE errors and close\n> > > some packs?  Or at least spit out a more intuitive error?\n> > \n> > What version of Git? I remember fixing this already.... :-)\n> \n> Initially 1.7.5.4. Same problem on 1.7.6 and master. \n> \n> I have your \"Limit file descriptors used by packs\" (c793430, 28\n> feb 2011).\n> \n> It fails here:\n> \n> \tif (!is_pack_valid(p)) {\n> \t\terror(\"packfile %s cannot be accessed\", p->pack_name);\n> \t\tgoto next;\n> \t}\n> \n> because p->pack_fd is -1, because an earlier git_open_noatime()\n> got EMFILE. The function unuse_one_window() is never able to\n> find anything to close.\n> \n> I'll do some more debugging this weekend. Thanks for pointing\n> out that it _should_ be fixed.\n\nHere's a test case (on master). It is easy to repack the\nrepository to avoid this situation, so I'm not convinced this is\na bug.\n\nBut it would be nice at least to report what's going on when we\nrun out of file descriptors.\n\nThe current error is:\n\n $ ../git commit -m fileN\n error: packfile .git/objects/pack/pack-7bff59b72a19f371ccd3e4ffefaac6cd1e07c6ed.pack cannot be accessed\n error: invalid object 100644 a7d02de3336c72e6d2d24f382b82195dbf625404 for 'file47'\n error: Error building trees\n\n\n\t\t-- Pete\n\n--------8<--------\n\nFrom: Pete Wyckoff \nDate: Sat, 13 Aug 2011 15:57:55 -0400\nSubject: [RFC PATCH] test showing EMFILE error with too many packs\n\nIn a repository with too many pack files, it is\npossible to have too many open at once and exceed\nthe process file descriptor limit.\n\nThis test shows one failure with an artificially low limit\nof 50. The same failure happens at a \"normal\" value\nlike 1024, but takes longer.\n\nSigned-off-by: Pete Wyckoff \n---\n t/t5308-pack-emfile.sh | 29 +++++++++++++++++++++++++++++\n 1 files changed, 29 insertions(+), 0 deletions(-)\n create mode 100755 t/t5308-pack-emfile.sh\n\ndiff --git a/t/t5308-pack-emfile.sh b/t/t5308-pack-emfile.sh\nnew file mode 100755\nindex 0000000..66c62a7\n--- /dev/null\n+++ b/t/t5308-pack-emfile.sh\n@@ -0,0 +1,29 @@\n+#!/bin/sh\n+\n+test_description='repos with too many packs should not fail'\n+\n+. ./test-lib.sh\n+\n+test_expect_success setup '\n+\tfor i in {1..60}\n+\tdo\n+\t\techo \"$i\" >\"file$i\" &&\n+\t\tgit add \"file$i\" &&\n+\t\ttest_tick &&\n+\t\tgit commit -q -m \"$i\" &&\n+\t\tgit repack -q\n+\tdone\n+\tgit prune-packed -q\n+'\n+\n+# run in a subshell since limits cannot be increased\n+test_expect_failure 'commit fail with too many packs' '\n+\t(\n+\tulimit -n 50 &&\n+\t>\"fileN\" &&\n+\tgit add \"fileN\" &&\n+\tgit commit -q -m \"fileN\"\n+\t)\n+'\n+\n+test_done\n-- \n1.7.5.4\n"}]}