Re: git repack leaks disk space on ENOSPC

From: Jeff King <peff@peff.net>
Date: Oct 12, 2017, 13:27 UTC
Message-ID: <20171012132730.bvglyiar4h6win4b@sigill.intra.peff.net>
In-Reply-To: <20171012031702.GB155740@aiede.mtv.corp.google.com>

On Wed, Oct 11, 2017 at 08:17:03PM -0700, Jonathan Nieder wrote:

Show 19 quoted lines

> I can imagine this behavior of retaining tmp_pack being useful for
> debugging in some circumstances, but I agree with you that it is
> certainly not a good default.
> 
> Chasing this down, I find:
> 
>   pack-write.c::create_tmp_packfile chooses the filename
>   builtin/pack-objects.c::write_pack_file writes to it and the .bitmap, calling
>   pack-write.c::finish_tmp_packfile to rename it into place
> 
> Nothing tries to install an atexit handler to do anything special to it
> on exit.
> 
> The natural thing, I'd expect, would be for pack-write to use the
> tempfile API (see tempfile.h) to create and finish the file.  That way,
> we'd get such atexit handlers for free.  If we want a way to keep temp
> files for debugging on abnormal exit, we could set that up separately as
> a generic feature of the tempfile API (e.g. an envvar
> GIT_KEEP_TEMPFILES_ON_FAILURE), making that an orthogonal topic.

Yes, I think this is the right direction. I've had a patch in GitHub's fork for years that does so (since otherwise failures can fill up your disk and need manual intervention).

The main reason that I hadn't submitted it upstream was because of the "you can never free a struct tempfile" requirement. So my patch just leaks the tempfile structs. That's OK for packs, of which we tend to create only a few in a given process, but doesn't scale for loose objects.

Now that 89563ec379 (Merge branch 'jk/incore-lockfile-removal', 2017-09-19) has landed, I think it makes sense to pursue that direction.

My patch roughly looks like:

  diff --git a/builtin/index-pack.c b/builtin/index-pack.c
  index 4ff567db47..7f261e56c4 100644
  --- a/builtin/index-pack.c
  +++ b/builtin/index-pack.c

  @@ -308,9 +348,11 @@ static const char *open_pack_file(const char *pack_name)
                  input_fd = 0;
                  if (!pack_name) {
                          struct strbuf tmp_file = STRBUF_INIT;
  +                       struct tempfile *t = xcalloc(1, sizeof(*t));
                          output_fd = odb_mkstemp(&tmp_file,
                                                  "pack/tmp_pack_XXXXXX");
                          pack_name = strbuf_detach(&tmp_file, NULL);
  +                       register_tempfile(t, pack_name);
                  } else {
                          output_fd = open(pack_name, O_CREAT|O_EXCL|O_RDWR, 0600);
                          if (output_fd < 0)

but note that's not quite what we'd want. It never closes the tempfile, so:

  1. Under the new regime, we'd still leak the struct!

  2. Git will still try to unlink the tempfile on exit, even if we
     successfully moved it into place.

So I think all the code around open_pack_file() needs to learn to pass around the tempfile struct, and eventually use rename_tempfile() to cement it in place. I also suspect that odb_mkstemp should just take a "struct tempfile".

-Peff