git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [MAKESHIFT PATCH] Cope better with a _lot_ of packs

From
Shawn O. Pearce <spearce@spearce.org>
Date
May 15, 2009, 19:17 UTC
Message-ID
<20090515191735.GG30527@spearce.org>
In-Reply-To
<alpine.DEB.1.00.0905152040590.26154@pacific.mpi-cbg.de>
Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
Show 7 quoted lines
> 
> You might end up with a situation where you have tons of pack files, e.g.
> when using hg2git.  In this situation, all kinds of operations may 
> end up with a "too many files open" error.  Let's recover gracefully from 
> that.
> 
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Yea, this looks right to me. JGit had a similar problem and now maintains a core.packedGitOpenFiles parameter, set to 128 by default, as one of the rules it uses to limit the size of its pack cache area.

Show 32 quoted lines
> diff --git a/sha1_file.c b/sha1_file.c
> index 28bd908..bd5edd8 100644
> --- a/sha1_file.c
> +++ b/sha1_file.c
> @@ -720,6 +720,8 @@ static int open_packed_git_1(struct packed_git *p)
>  		return error("packfile %s index unavailable", p->pack_name);
>  
>  	p->pack_fd = open(p->pack_name, O_RDONLY);
> +	while (p->pack_fd < 0 && errno == EMFILE && unuse_one_window(p, -1))
> +		p->pack_fd = open(p->pack_name, O_RDONLY);
>  	if (p->pack_fd < 0 || fstat(p->pack_fd, &st))
>  		return -1;
>  
> @@ -937,6 +939,8 @@ static void prepare_packed_git_one(char *objdir, int local)
>  	sprintf(path, "%s/pack", objdir);
>  	len = strlen(path);
>  	dir = opendir(path);
> +	while (!dir && errno == EMFILE && unuse_one_window(packed_git, -1))
> +		dir = opendir(path);
>  	if (!dir) {
>  		if (errno != ENOENT)
>  			error("unable to open object pack directory: %s: %s",
> @@ -2339,6 +2343,8 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,
>  
>  	filename = sha1_file_name(sha1);
>  	fd = create_tmpfile(tmpfile, sizeof(tmpfile), filename);
> +	while (fd < 0 && errno == EMFILE && unuse_one_window(packed_git, -1))
> +		fd = create_tmpfile(tmpfile, sizeof(tmpfile), filename);
>  	if (fd < 0) {
>  		if (errno == EACCES)
>  			return error("insufficient permission for adding an object to repository database %s\n", get_object_directory());
> -- 
-- 
Shawn.
Previous: Johannes SchindelinNext: Junio C Hamano
Message 17 of 18 in “Funny error with git gc...”
  1. Johannes SchindelinMay 15, 2009
  2. Junio C HamanoMay 15, 2009
  3. Avery PennarunMay 15, 2009
  4. Linus TorvaldsMay 15, 2009
  5. Johannes SchindelinMay 15, 2009
  6. Johannes SchindelinMay 15, 2009
  7. Johannes SchindelinMay 15, 2009
  8. Linus TorvaldsMay 15, 2009
  9. Johannes SchindelinMay 23, 2009
  10. Linus TorvaldsMay 23, 2009
  11. Johannes SchindelinMay 24, 2009
  12. Linus TorvaldsMay 15, 2009
  13. Johannes SchindelinMay 15, 2009
  14. Daniel BarkalowMay 15, 2009
  15. Matthias AndreeMay 18, 2009
  16. Cope better with a _lot_ of packsJohannes Schindelin, May 15, 2009
  17. Shawn O. PearceMay 15, 2009
  18. Junio C HamanoMay 21, 2009

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.