{"thread":{"id":"33061","subject":"[BUG] Incorrect/misleading error when `git rev-list --objects --all` hits the max open files limit","startedAt":"2013-03-03T23:29:27Z","lastAt":"2013-03-05T15:26:25Z","messageCount":2,"participants":["Peter Rabbitson","Pete Wyckoff"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"210552","messageId":"20130303232927.GA16606@rabbit.us","threadId":"33061","inReplyTo":null,"subject":"[BUG] Incorrect/misleading error when `git rev-list --objects --all` hits the max open files limit","fromName":"Peter Rabbitson","fromEmail":"rabbit@rabbit.us","sentAt":"2013-03-03T23:29:27Z","receivedAt":"2013-03-03T23:29:27Z","isPatch":false,"sender":{"key":"rabbit@rabbit.us","avatar":null},"body":"Hi!\n\nI was tinkering with a massive git repository (actually a bup \nrepository, but it is a standard valid git repo underneath). While \nvalidating that a repack ran succesfully I executed the command:\n\n git rev-list --objects --all > rev.list\n\nAnd got back this:\n\nerror: packfile ./objects/pack/pack-d9808b7515419737806d0c621a0a1910f71c5cba.pack cannot be accessed\nfatal: missing blob object '27a8cf44da85b958aef2b5074931e7913e08ae95'\n\nSeveral hours later after successful fsck, and after chasing down trees \nblobs etc, I realized that the problem is too many open files. The hint \ncame from ls-tree which lists the correct error (among a lot of spurious \njunk):\n\n git ls-tree -r c636a5f51d4e > /dev/null\n\n error: packfile ./objects/pack/pack-d9808b7515419737806d0c621a0a1910f71c5cba.pack cannot be accessed\n error: packfile ./objects/pack/pack-841e375f5e6c793a52fd1a3a2aea0b76219c4cdd.pack cannot be accessed\n error: packfile ./objects/pack/pack-e67d9bf75e0840fc6113170b314d2d5a32cbb45a.pack cannot be accessed\n error: packfile ./objects/pack/pack-b8fd8f083461c391fe6ec396840c328620d912e2.pack cannot be accessed\n error: packfile ./objects/pack/pack-d9808b7515419737806d0c621a0a1910f71c5cba.pack cannot be accessed\n error: packfile ./objects/pack/pack-804e0fadf56e2a165c157ef257620369adeea595.pack cannot be accessed\n error: unable to open object pack directory: ./objects/pack: Too many open files\n error: packfile ./objects/pack/pack-804e0fadf56e2a165c157ef257620369adeea595.pack cannot be accessed\n error: Could not read 32a050cb7e54a1e817d135d25ab251480e8d9e3c\n\nFailure to report the correct message verified with git 1.7.2.5 and \n1.8.2 (debian testing and experimental).\n\nI hope this is sufficient description to address the underlying issue. I \nwill keep the un-repacked \"many files\" repo around just in case you need \nmore info/testing (though lowering the ulimit works equally well on \nnormal-size repos).\n\nCheers\n"},{"id":"210627","messageId":"20130305152625.GA20562@padd.com","threadId":"33061","inReplyTo":"20130303232927.GA16606@rabbit.us","subject":"Re: [BUG] Incorrect/misleading error when `git rev-list --objects --all` hits the max open files limit","fromName":"Pete Wyckoff","fromEmail":"pw@padd.com","sentAt":"2013-03-05T15:26:25Z","receivedAt":"2013-03-05T15:26:25Z","isPatch":false,"sender":{"key":"pw@padd.com","avatar":null},"body":"rabbit@rabbit.us wrote on Mon, 04 Mar 2013 10:29 +1100:\n> I was tinkering with a massive git repository (actually a bup \n> repository, but it is a standard valid git repo underneath). While \n> validating that a repack ran succesfully I executed the command:\n> \n>  git rev-list --objects --all > rev.list\n> \n> And got back this:\n> \n> error: packfile ./objects/pack/pack-d9808b7515419737806d0c621a0a1910f71c5cba.pack cannot be accessed\n> fatal: missing blob object '27a8cf44da85b958aef2b5074931e7913e08ae95'\n> \n> Several hours later after successful fsck, and after chasing down trees \n> blobs etc, I realized that the problem is too many open files. The hint \n> came from ls-tree which lists the correct error (among a lot of spurious \n> junk):\n> \n>  git ls-tree -r c636a5f51d4e > /dev/null\n> \n>  error: packfile ./objects/pack/pack-d9808b7515419737806d0c621a0a1910f71c5cba.pack cannot be accessed\n>  error: packfile ./objects/pack/pack-841e375f5e6c793a52fd1a3a2aea0b76219c4cdd.pack cannot be accessed\n>  error: packfile ./objects/pack/pack-e67d9bf75e0840fc6113170b314d2d5a32cbb45a.pack cannot be accessed\n>  error: packfile ./objects/pack/pack-b8fd8f083461c391fe6ec396840c328620d912e2.pack cannot be accessed\n>  error: packfile ./objects/pack/pack-d9808b7515419737806d0c621a0a1910f71c5cba.pack cannot be accessed\n>  error: packfile ./objects/pack/pack-804e0fadf56e2a165c157ef257620369adeea595.pack cannot be accessed\n>  error: unable to open object pack directory: ./objects/pack: Too many open files\n>  error: packfile ./objects/pack/pack-804e0fadf56e2a165c157ef257620369adeea595.pack cannot be accessed\n>  error: Could not read 32a050cb7e54a1e817d135d25ab251480e8d9e3c\n> \n> Failure to report the correct message verified with git 1.7.2.5 and \n> 1.8.2 (debian testing and experimental).\n> \n> I hope this is sufficient description to address the underlying issue. I \n> will keep the un-repacked \"many files\" repo around just in case you need \n> more info/testing (though lowering the ulimit works equally well on \n> normal-size repos).\n\nI've run into this twice:\n\n    1.  user with restrictive ulimit on #files.\n    2.  forgot to do \"git gc\" after regular fast-imports\n\nHere's one thread:\n\n    http://thread.gmane.org/gmane.comp.version-control.git/179231/focus=179292\n\nI've got a patch in cold storage.  It's not beautiful because\nthere are too many paths that can end up hitting EMFILE.  I'll\ndust it off and see if it is worth considering.\n\n\t\t-- Pete\n"}]}