[PATCH 0/3] reducing fsck memory usage

From: Jeff King <peff@peff.net>
Date: Sep 22, 2022, 10:09 UTC
Message-ID: <Yyw0PSVe3YTQGgRS@coredump.intra.peff.net>
In-Reply-To: <YyoljwDIn7PxRlC9@coredump.intra.peff.net>

On Tue, Sep 20, 2022 at 04:41:51PM -0400, Jeff King wrote:

Show 11 quoted lines

> And this is data we've allocated for trees. It kind of looks like
> fsck_walk_tree() never bothers to clean up the trees it parses, leaving
> the buffers attached to the tree structs. But that can't be the case,
> because linux.git has something like 16GB of trees. These may be entries
> we keep in the internal delta cache, though it should be a bit smaller
> than what you have here (the default is 96MB; you can drop it with
> core.deltaBaseCacheLimit, but runtime may suffer).
> 
> There's a call to free_tree_buffer() in builtin/fsck.c:traverse_one();
> that may be what ends up freeing things. It's been a while since I've
> traced through the call paths for fsck.

I dug into this a bit more. We do indeed free the trees correctly in most of the code paths, but there's one where we don't, and you can convince it to hold all of the trees in memory at once (though in practice few people are likely to hit this case).

So here's a fix for that, plus a cleaned up version of the patch I sent earlier which should drop your memory usage, plus a bonus small memory reduction and cleanup in the third patch.

  [1/3]: fsck: free tree buffers after walking unreachable objects
  [2/3]: fsck: turn off save_commit_buffer
  [3/3]: parse_object_buffer(): respect save_commit_buffer

 builtin/fsck.c | 6 +++---
 object.c       | 3 ++-
 2 files changed, 5 insertions(+), 4 deletions(-)

-Peff