[PATCH v3 0/2] [v2.24.0-rc0 BUG] fetch.writeCommitGraph fails on first fetch

UPDATE for V2: We now know the full repro, and a test is added. Thanks Szeder and Peff for your insights!

UPDATE in V3: Cleaned up the commit messages and some test details.

While dogfooding, Johannes found a bug in the fetch.writeCommitGraph config behavior. While his example initially happened during a clone with --recurse-submodules, (UPDATE) and the submodule is important, but --recurse-submodules is not:

$ git clone <url> test
$ cd test
$ git -c fetch.writeCommitGraph=true fetch origin
Computing commit graph generation numbers: 100% (12/12), done.
BUG: commit-graph.c:886: missing parent <hash1> for commit <hash2>
Aborted (core dumped)

In the repo I had cloned, there were really 60 commits to scan, but only 12 were in the list to write when calling compute_generation_numbers(). A commit in the list expects to see a parent, but that parent is not in the list. The simple example I used for my testing was https://github.com/derrickstolee/numbers. Thie repo HAS A SUBMODULE, I just forgot. Sorry for derailing the investigation somewhat.

The details above are the start of the commit message for [PATCH 1/2], including a test that fails when fetching after cloning a repo with a submodule.

In [PATCH 2/2], I actually have the fix. I tried to include as much detail as I could for how I investigated the problem and why I think this is the right solution. I added details that have come from the on-list discussion, including what the submodule code is doing and why REACHABLE is no longer used in commit-reach.c.

Thanks, -Stolee

Derrick Stolee (2):
  t5510-fetch.sh: demonstrate fetch.writeCommitGraph bug
  commit-graph: fix writing first commit-graph during fetch

 commit-graph.c   | 11 +++++++----
 commit-reach.c   |  1 -
 object.h         |  3 ++-
 t/t5510-fetch.sh | 16 ++++++++++++++++
 4 files changed, 25 insertions(+), 6 deletions(-)

base-commit: d966095db01190a2196e31195ea6fa0c722aa732
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-415%2Fderrickstolee%2Ffetch-first-write-fail-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-415/derrickstolee/fetch-first-write-fail-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/415

Range-diff vs v2:

 1:  6ac0a05746 ! 1:  ce53b5a7bf t5510-fetch.sh: demonstrate fetch.writeCommitGraph bug
     @@ -22,10 +22,6 @@
          A follow-up will fix the bug, but first we create a test that
          demonstrates the problem.
      
     -    I used "test_expect_failure" for the entire test instead of
     -    "test_must_fail" only on the command that I expect to fail. This is
     -    because the BUG() returns an exit code so test_must_fail complains.
     -
          Helped-by: Jeff King <peff@peff.net>
          Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
          Helped-by: Szeder Gábor <szeder.dev@gmail.com>
     @@ -39,16 +35,15 @@
       '
       
      +test_expect_failure 'fetch.writeCommitGraph with submodules' '
     -+	pwd="$(pwd)" &&
      +	git clone dups super &&
      +	(
      +		cd super &&
     -+		git submodule add "file://$pwd/three" &&
     ++		git submodule add "file://$TRASH_DIRECTORY/three" &&
      +		git commit -m "add submodule"
      +	) &&
     -+	git clone "super" writeError &&
     ++	git clone "super" super-clone &&
      +	(
     -+		cd writeError &&
     ++		cd super-clone &&
      +		test_path_is_missing .git/objects/info/commit-graphs/commit-graph-chain &&
      +		git -c fetch.writeCommitGraph=true fetch origin &&
      +		test_path_is_file .git/objects/info/commit-graphs/commit-graph-chain
 2:  ca59b118f1 ! 2:  edacfff490 commit-graph: fix writing first commit-graph during fetch
     @@ -6,7 +6,7 @@
          fetch.writeCommitGraph and fetching in a repo with a submodule. Here, we
          fix that bug and set the test to "test_expect_success".
      
     -    The prolem arises with this set of commands when the remote repo at
     +    The problem arises with this set of commands when the remote repo at
          <url> has a submodule. Note that --recurse-submodules is not needed to
          demonstrate the bug.
      
     @@ -44,16 +44,6 @@
          negotiation is comparing the remote refs with the local refs and marking
          some commits as UNINTERESTING.
      
     -    You may ask: did this feature ever work at all? Yes, it did, as long as
     -    you had a commit-graph covering all of your local refs. My testing was
     -    unfortunately limited to this scenario. The UNINTERESTING commits are
     -    always part of the "old" commit-graph, and when we add new commits to a
     -    top layer of the commit-graph chain those are not needed. If we happen
     -    to merge layers of the chain, then the commits are added as a list, not
     -    using a commit walk. Further, the test added for this config option in
     -    t5510-fetch.sh uses local filesystem clones, which somehow avoids this
     -    logic.
     -
          I tested running clear_commit_marks_many() to clear the UNINTERESTING
          flag inside close_reachable(), but the tips did not have the flag, so
          that did nothing.
     @@ -62,7 +52,7 @@
          fault. Thanks, Peff, for pointing out this detail! More specifically,
          for each submodule, the collect_changed_submodules() runs a revision
          walk to essentially do file-history on the list of submodules. That
     -    revision walk marks commits UNININTERESTING if they are simiplified away
     +    revision walk marks commits UNININTERESTING if they are simplified away
          by not changing the submodule.
      
          Instead, I finally arrived on the conclusion that I should use a flag
     @@ -163,6 +153,6 @@
       
      -test_expect_failure 'fetch.writeCommitGraph with submodules' '
      +test_expect_success 'fetch.writeCommitGraph with submodules' '
     - 	pwd="$(pwd)" &&
       	git clone dups super &&
       	(
     + 		cd super &&