git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [MAKESHIFT PATCH] Cope better with a _lot_ of packs

From
Junio C Hamano <gitster@pobox.com>
Date
May 21, 2009, 01:22 UTC
Message-ID
<7vzld7tfl2.fsf@alter.siamese.dyndns.org>
In-Reply-To
<20090515191735.GG30527@spearce.org>
"Shawn O. Pearce" <spearce@spearce.org> writes:
Show 25 quoted lines
> Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
>> 
>> You might end up with a situation where you have tons of pack files, e.g.
>> when using hg2git.  In this situation, all kinds of operations may 
>> end up with a "too many files open" error.  Let's recover gracefully from 
>> that.
>> 
>> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
>
> Yea, this looks right to me.  JGit had a similar problem and now
> maintains a core.packedGitOpenFiles parameter, set to 128 by default,
> as one of the rules it uses to limit the size of its pack cache area.
>
>> diff --git a/sha1_file.c b/sha1_file.c
>> index 28bd908..bd5edd8 100644
>> --- a/sha1_file.c
>> +++ b/sha1_file.c
>> @@ -720,6 +720,8 @@ static int open_packed_git_1(struct packed_git *p)
>>  		return error("packfile %s index unavailable", p->pack_name);
>>  
>>  	p->pack_fd = open(p->pack_name, O_RDONLY);
>> +	while (p->pack_fd < 0 && errno == EMFILE && unuse_one_window(p, -1))
>> +		p->pack_fd = open(p->pack_name, O_RDONLY);
>>  	if (p->pack_fd < 0 || fstat(p->pack_fd, &st))
>>  		return -1;

I wonder if this use of unuse_one_window() can be made more clever to drop windows from the same packfile first. If you have 200 packfiles with 2 windows open for each one when you ran out the file descriptors, you do not want to drop one window from each packfile (which would not give you any free file descriptor) and then finally release another window (at which point we have one packfile with no window and we can close the descriptor).

But at the same time I do not know if it matters in practice. Presumably you have too many packfiles because many of them are very small.

Previous: Shawn O. Pearce
Message 18 of 18 in “Funny error with git gc...”
  1. Johannes SchindelinMay 15, 2009
  2. Junio C HamanoMay 15, 2009
  3. Avery PennarunMay 15, 2009
  4. Linus TorvaldsMay 15, 2009
  5. Johannes SchindelinMay 15, 2009
  6. Johannes SchindelinMay 15, 2009
  7. Johannes SchindelinMay 15, 2009
  8. Linus TorvaldsMay 15, 2009
  9. Johannes SchindelinMay 23, 2009
  10. Linus TorvaldsMay 23, 2009
  11. Johannes SchindelinMay 24, 2009
  12. Linus TorvaldsMay 15, 2009
  13. Johannes SchindelinMay 15, 2009
  14. Daniel BarkalowMay 15, 2009
  15. Matthias AndreeMay 18, 2009
  16. Cope better with a _lot_ of packsJohannes Schindelin, May 15, 2009
  17. Shawn O. PearceMay 15, 2009
  18. Junio C HamanoMay 21, 2009

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.