Re: [PATCH v4 08/10] commit-graph: use generation v2 only if entire chain does

Hi Abhishek,

"Abhishek Kumar via GitGitGadget" <gitgitgadget@gmail.com> writes:

All right.

I think this paragraphs tries too much to be concise, with the result it is less clear than it could be. Perhaps it would be better to separate "what-if" from the current behavior.

  If each layer of split commit-graph is treated independently, as it
  were the case before this commit, with Git inspecting only the current
  layer for chunk_generation_data pointer, commits in the lower layer
  (one with GDAT) would have corrected commit date as their generation
  number, while commits in the upper layer would have topological levels
  as their generation.  Corrected commit dates have usually much larger
  values than topological levels.  This means that if we take two
  commits, one from the upper layer, and one reachable from it in the
  lower layer, then the expectation that the generation of a parent is
  smaller than the generation of a child would be violated.

I don't quite understand the issue here. Unless none of the following query commands short-circuit and all walk the commit graph regardless of what generation numbers tell them, they should give different results with and without the commit graph, if we take two commits one from lower layer of split commit graph with GDAT, and one commit from the higher layer without GDAT, one lower reachable from the other higher.

We have the following query commands that we can check:
  $ git merge-base --is-ancestor <lower> <higher>
  $ git merge-base --independent <lower> <higher>
  
  $ git tag --contains <tag-to-lower>
  $ git tag --merged <tag-to-higher>
  $ git branch --contains <branch-to-lower>
  $ git branch --merged <branch-to-higher>

The second set of queries require for those commits to be tagged, or have branch pointing at them, respectively.

Also, shouldn't `git commit-graph verify` fail with split commit graph where the top layer is created with GIT_TEST_COMMIT_GRAPH_NO_GDAT=1?

Let's assume that we have the following history, with newer commits shown on top like in `git log --graph --oneline --all`:

          topological     corrected         generation
          level           commit date       number^*

      d    3                                3
      |
   c  |    3                                3
   |  |                                                 without GDAT
 ..|..|.....[layer.boundary]........................................
   |  |                                                    with GDAT
   |  b    2              1112912113        1112912113
   |  |
   a  |    2              1112912053        1112912053
   | /
   |/
   r       1              1112911993        1112911993

*) each layer inspected individually.

With such history, we can for example reach 'a' from 'c', thus `git merge-base --is-ancestor a b` should return true value, but without this commit gen(a) > gen(c), instead of gen(a) <= gen(c); I use here weaker reachability condition, but the one that works also for commits outside the commit-graph (and those for which generation numbers overflows).

All right, that's good explanation.

All right, very good explanation; the only minor suggestion would be to add some 'intro' to the first of those two paragraphs, for example:

  Therefore, when writing the new layer in split commit-graph...

Though I am not sure if it is necessary.

All right, this function checks assumedly topmost layer if it is GDAT-less, and if it is propagates this status down the layers of split commit graph. This is needed because if we have mixed-generation commit graph, then for each and every layer we need to use topological levels as generation number.

The only minor issue is the name of this function (it does not hint that it propagates the GDAT status downwards), but I don't have a better idea, unfortunately. And it does reflect what this function is used for.

All right, this looks like a good place to put this new check: just after reading commit-graph chain.

All right, instead of simply checking if the current layer has generation data chunk, we need to also check if the whole graph allows for it (if there are no mixed-generation layers).

The g->read_generation_data should be filled correctly, because fill_commit_graph_info() is always preceded by read_commit_graph(), if I understand it correctly.

This needs more careful examination, and looking at larger context of those lines.

At this point, unless `--split=replace` option is used, 'g' points to the bottom layer out of all topmost layers being merged. We know that if there are GDAT-less layers then these must be top layers, so this means that we can write GDAT chunk in the result of the merge -- because we would be replacing all possible GDAT-less layers (and maybe some with GDAT) with a single layer with the GDAT chunk.

The ctx->write_generation_data is set to true unless environment variable GIT_TEST_COMMIT_GRAPH_NO_GDAT is true, and that in write_commit_graph() it would be set to false if topmost layer doesn't have GDAT chunk, and to true if `--split=replace` option is used; see below.

Looks good to me.

NOTE that this means that GIT_TEST_COMMIT_GRAPH_NO_GDAT prevents from writing GDAT chunk with generation data v2 unless we are merging layers, or replacing all of them with a single layer: then it is _ignored_.

Should we clarify this fact in the description of GIT_TEST_COMMIT_GRAPH_NO_GDAT in t/README? Currently it reads:

  GIT_TEST_COMMIT_GRAPH_NO_GDAT=<boolean>, when true, forces the
  commit-graph to be written without generation data chunk.

All right, when writing the commit graph we want to make use of existing generation data chunks. This is safe, because when computing generation numbers for writing we have separate place to store topoogical levels (`topo_levels`) so they would not be mixed with corrected commit dates: generation number v1 and v2 are kept separate.

All right, if current (topmost) layed does not have GDAT, then when creating a new layer do not create GDAT layer either (merging layers and rewriting the commit-graph is handled separately).

All right, when replacing all layers (`git commit-graph write --split=replace`), then we can safely write the GDAT chunk.

Note however that here we don't take into account the value of the environment variable GIT_TEST_COMMIT_GRAPH_NO_GDAT. Which maybe is what we want...

All right, this new field is here to propagate to each layer the information whether we can read from the generation number v2 data chunk.

Though I am not sure whether this field should be added here, and whether it should be `unsigned int` (we don't have to be that careful about saving space for this type).

This should probably go just before cd-ing into just created subdirectory.

Minor nitpick: Why use "EOM", which is used only twice in Git the test suite, and not the conventional "EOF" (used at least 4000 times)?

The t/README says:

   - Don't chdir around in tests.  It is not sufficient to chdir to
     somewhere and then chdir back to the original location later in
     the test, as any intermediate step can fail and abort the test,
     causing the next test to start in an unexpected directory.  Do so
     inside a subshell if necessary.

Though I am not sure if it should apply also to this situation.

All right.

Is there a reason why we do not check just written commit-graph file with `test-tool read-graph >output-layer-1`?

Shouldn't we use `test_merge` instead of `git merge`; I am not sure when to use one or the other?

It would be nice to have ASCII-art of the history (of the graph of revisions) created here for subsequent tests:

                                        
           /- 6 <-- 7 <-- 8 <-- 9 <-- 10*
          /    \-\
         /        \
  1 <-- 2 <-- 3*   \--\
  |      \             \ 
  |       \-----\       \
   \             \       \
    \-- 4*<------ M/1     M/2
        |\               /  
        | \-- 5*        /
        \              /
         \------------/

  * - 1st layer  

Though I am not sure if what I have created is readable; I think a better way to draw this graph is possible, for example:

               /- 3*
              /
             /
  1 <------ 2 <---- 6 <-- 7 <-- 8 <-- 9 <-- 10*
   \         \       \
    \         \       \
     \         \       \
      \- 4* <-- M/1     \    
         |\              \
         | \------------- M/2
         \
          \---- 5*

Edit: as I see the history gets even more complicated, so perhaps
ASCII-art diagram of the history with layers marked would be too
complicated, and wouldn't bring much.

Why do we need such shape of the history in the repository?

All right, we check that we have 2 commits, and that there is no GDAT chunk.

All right, we verify commit-graph as a whole (both layers).

Hmmm... I wonder if we can come up with a better name for this test; for example should it be "does not write" or "do not write"?

The commit graph gets complicated, so it would not be easy to visualize it with ASCII-art diagram without any crossed lines. Maybe `git log --graph --oneline --all` would help:

*   (merge/3) Merge branch 'merge/2'
|\
| *   (merge/2) Merge branch 'commits/6'
| |\
* | \   Merge branch 'commits/5'
|\ \ \
| * | | (commits/5) 5
| |/ /
* | |   Merge branch 'merge/1'
|\ \ \
| * | | (merge/1) Merge branch 'commits/4'
| |\| |
| | * | (commits/4) 4
* | | | (commits/3) 3
|/ / /
| | | * (commits/10) 10
| | | * (commits/9) 9
| | | * (commits/8) 8
| | | * (commits/7) 7
| | |/
| | * (commits/6) 6
| |/
|/|
* | (commits/2) 2
|/
* (commits/1) 1

All right, so here we check that we have layer without GDAT at the top, and we request not to merge layers thus new layer will be created, then the new layer also does not have GDAT chunk (and has 3 commits).

Minor nitpick: shouldn't those test be indented?

All right, this checks that we have split commit-graph chain that consist of a single layer, and that the commit-graph file for this single layer exists.

Shouldn't we use `test-tool read-graph` to check whether generation_data chunk is present... ah, sorry, I have realized that after previous patches `graph_read_expect 15` implicitly checks the latter, because in its' use of `test-tool read-graph` it does expect generation_data chunk.

So we use `test-tool read-graph` manually to check that generation_data chunk is absent, and we use graph_read_expect to check that it is present (and in both cases that the number of commits matches). I wonder if it would be possible to simplify that...

All right.

What it is meant to test? That adding single-commit to a 15 commit commit-graph file in split mode does not result in layers merging, and actually adds a new layer: we check that we have exactly two layers and that they are all OK.

We don't check here that the newly created top layer commit-graph does have GDAT chunk, as it should be if the top layer (in this case the only layer) has GDAT chunk.

One test we are missing is testing that merging layers is done correctly, namely that if we are merging layers in split commit-graph file, and the layer below the ones we are merging lacks GDAT chunk, then the result of the merge should also be without GDAT chunk. This would require at least two GDAT-less layers in a setup.

I'm not sure how difficult writing such test should be.

Best,