Re: [PATCH] upload-pack: disable commit graph more gently for shallow traversal

From: Jeff King <peff@peff.net>
Date: Sep 12, 2019, 14:23 UTC
Message-ID: <20190912142306.GE23031@sigill.intra.peff.net>
In-Reply-To: <fdb81ce0-20ee-5880-2a6c-0c8b3f1739b9@gmail.com>

On Thu, Sep 12, 2019 at 08:23:49AM -0400, Derrick Stolee wrote:

Show 9 quoted lines

> > That creates an interesting problem for commits that have _already_ been
> > parsed using the commit graph. Their commit->object.parsed flag is set,
> > their commit->graph_pos is set, but their commit->maybe_tree may still
> > be NULL. When somebody later calls repo_get_commit_tree(), we see that
> > we haven't loaded the tree oid yet and try to get it from the commit
> > graph. But since it has been freed, we segfault!
> 
> OOPS! That is certainly a bad thing. I'm glad you found it, but I
> am sorry for how you (probably) found it.

Heh. I'll admit it was quite a slog of debugging, but _most_ of that was figuring out in which circumstance we'd have actually parsed the object. Finding the problematic end state was pretty easy from a coredump. :)

Show 11 quoted lines

> > diff --git a/commit-graph.c b/commit-graph.c
> > index 9b02d2c426..bc5dd5913f 100644
> > --- a/commit-graph.c
> > +++ b/commit-graph.c
> > @@ -41,6 +41,8 @@
> >  #define GRAPH_MIN_SIZE (GRAPH_HEADER_SIZE + 4 * GRAPH_CHUNKLOOKUP_WIDTH \
> >  			+ GRAPH_FANOUT_SIZE + the_hash_algo->rawsz)
> >  
> > +static int commit_graph_disabled;
> 
> Should we be putting this inside the repository struct instead?

Probably. The only caller will just pass the_repository, but it doesn't hurt to scope it down now.

It could potentially go into the commit_graph itself, but it looks like with the incremental work we may have multiple such structs. It could also go into raw_object_store, but I think conceptually it's a repo-level thing.

So I put it straight into "struct repository".

> Your patch does not seem to actually cover the "I've already parsed some commits"
> case, as you are only preventing the commit-graph from being prepared. Instead,
> we need to have a short-circuit inside parse_commit() to avoid future parsing
> from the commit-graph file.

Maybe I was too clever, then. :)

I didn't want to have to sprinkle "are we disabled" in parse_commit(), etc. But any such uses of the commit graph have to do:

  if (!prepare_commit_graph(r))
	return;

to lazy-load it. So the logic to prepare becomes (roughly):

  if (disabled)
	return 0;
  if (already_loaded)
	return 1;
  return actually_load() ? 1 : 0;

and "disabled" takes precedence.

I've added this comment in prepare_commit_graph():

        /*
         * This must come before the "already attempted?" check below, because
         * we want to disable even an already-loaded graph file.
         */
        if (r->commit_graph_disabled)
                return 0;

        if (r->objects->commit_graph_attempted)
                return !!r->objects->commit_graph;
        r->objects->commit_graph_attempted = 1;

Does that make more sense?

Unrelated, but I also notice the top of prepare_commit_graph() has:

        if (git_env_bool(GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD, 0))
                die("dying as requested by the '%s' variable on commit-graph load!",
                    GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD);

as the very first thing. Meaning we're calling getenv() as part of every single parse_commit(), rather than just once per process. Seems like an easy efficiency win.

-Peff