Re: [MAKESHIFT PATCH] Cope better with a _lot_ of packs
- From
Shawn O. Pearce <spearce@spearce.org>
- Date
- May 15, 2009, 19:17 UTC
- Message-ID
- <20090515191735.GG30527@spearce.org>
- In-Reply-To
- <alpine.DEB.1.00.0905152040590.26154@pacific.mpi-cbg.de>
Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
Show 7 quoted lines
> > You might end up with a situation where you have tons of pack files, e.g. > when using hg2git. In this situation, all kinds of operations may > end up with a "too many files open" error. Let's recover gracefully from > that. > > Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Yea, this looks right to me. JGit had a similar problem and now maintains a core.packedGitOpenFiles parameter, set to 128 by default, as one of the rules it uses to limit the size of its pack cache area.
Show 32 quoted lines
> diff --git a/sha1_file.c b/sha1_file.c
> index 28bd908..bd5edd8 100644
> --- a/sha1_file.c
> +++ b/sha1_file.c
> @@ -720,6 +720,8 @@ static int open_packed_git_1(struct packed_git *p)
> return error("packfile %s index unavailable", p->pack_name);
>
> p->pack_fd = open(p->pack_name, O_RDONLY);
> + while (p->pack_fd < 0 && errno == EMFILE && unuse_one_window(p, -1))
> + p->pack_fd = open(p->pack_name, O_RDONLY);
> if (p->pack_fd < 0 || fstat(p->pack_fd, &st))
> return -1;
>
> @@ -937,6 +939,8 @@ static void prepare_packed_git_one(char *objdir, int local)
> sprintf(path, "%s/pack", objdir);
> len = strlen(path);
> dir = opendir(path);
> + while (!dir && errno == EMFILE && unuse_one_window(packed_git, -1))
> + dir = opendir(path);
> if (!dir) {
> if (errno != ENOENT)
> error("unable to open object pack directory: %s: %s",
> @@ -2339,6 +2343,8 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,
>
> filename = sha1_file_name(sha1);
> fd = create_tmpfile(tmpfile, sizeof(tmpfile), filename);
> + while (fd < 0 && errno == EMFILE && unuse_one_window(packed_git, -1))
> + fd = create_tmpfile(tmpfile, sizeof(tmpfile), filename);
> if (fd < 0) {
> if (errno == EACCES)
> return error("insufficient permission for adding an object to repository database %s\n", get_object_directory());
> -- -- Shawn.