{"thread":{"id":"57472","subject":"[PATCH 0/7] Commit-graph: Generation Number v2 Fixes, v3 implementation","startedAt":"2022-02-24T20:38:43Z","lastAt":"2022-03-10T17:18:15Z","messageCount":70,"participants":["Derrick Stolee via GitGitGadget","Junio C Hamano","Derrick Stolee","Ævar Arnfjörð Bjarmason","Patrick Steinhardt"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"449504","messageId":"pull.1163.git.1645735117.gitgitgadget@gmail.com","threadId":"57472","inReplyTo":null,"subject":"[PATCH 0/7] Commit-graph: Generation Number v2 Fixes, v3 implementation","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-24T20:38:29Z","receivedAt":"2022-02-24T20:38:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This patch series includes two distinct, but similarly-motivated parts:\n\n * Patches 1-4 fix some bugs in the commit-graph generation number v2.\n * Patches 5-7 add a new generation number v3 by incrementing the\n commit-graph file format.\n\nI had been thinking about generation number v3, which is the same corrected\ncommit date as generation number v2, but it is stored in the Commit Data\nchunk, requiring a new commit-graph file format version. This breaks\ncompatibility with older versions of Git, so it requires opt-in via the\ncommitGraph.generationVersion config value. The only improvement over\nversion 2 is that the commit-graph file is smaller, so I/O time is reduced.\n\nHowever, while exploring this idea I found bugs in generation number v2. In\nparticular, Git has been ignoring them since shortly after they were\nintroduced. This is due to a bug I introduced when trying to make split\ncommit-graphs safer with mixed generation number versions. I also noticed an\nissue with the offset overflows that I only noticed after writing generation\nnumber v3 using a smaller offset size, actually triggering the bug in the\ntest suite.\n\nI'm submitting these two things together so we can see them all at once, but\nI'd be happy to split this into two series. The first four patches are\nimportant bug fixes, so we can consider them as higher-priority.\n\nThanks, -Stolee\n\nDerrick Stolee (7):\n test-read-graph: include extra post-parse info\n commit-graph: fix ordering bug in generation numbers\n commit-graph: start parsing generation v2 (again)\n commit-graph: fix generation number v2 overflow values\n commit-graph: document file format v2\n commit-graph: parse file format v2\n commit-graph: write file format v2\n\n Documentation/config/commitgraph.txt | 4 +-\n .../technical/commit-graph-format.txt | 22 ++++-\n commit-graph.c | 98 +++++++++++++++----\n commit-graph.h | 6 ++\n commit.h | 1 +\n t/helper/test-read-graph.c | 13 +++\n t/t4216-log-bloom.sh | 1 +\n t/t5318-commit-graph.sh | 65 ++++++++++--\n t/t5324-split-commit-graph.sh | 10 ++\n 9 files changed, 189 insertions(+), 31 deletions(-)\n\n\nbase-commit: dab1b7905d0b295f1acef9785bb2b9cbb0fdec84\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1163%2Fderrickstolee%2Fgen-v3-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1163/derrickstolee/gen-v3-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1163\n-- \ngitgitgadget\n"},{"id":"449505","messageId":"6e47ffed25795260c2b8614d4589fb58d892c8df.1645735117.git.gitgitgadget@gmail.com","threadId":"57472","inReplyTo":"pull.1163.git.1645735117.gitgitgadget@gmail.com","subject":"[PATCH 2/7] commit-graph: fix ordering bug in generation numbers","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-24T20:38:31Z","receivedAt":"2022-02-24T20:38:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen computing the generation numbers for a commit-graph, we compute\nthe corrected commit dates and then check if their offsets from the\nactual dates is too large to fit in the 32-bit Generation Data chunk.\nHowever, there is a problem with this approach: if we have parsed the\ngeneration data from the previous commit-graph, then we continue the\nloop because the corrected commit date is already computed.\n\nIt is incorrect to add an increment to num_generation_data_overflows\nhere, because we might start double-counting commits that are computed\nbecause of the depth-first search walk from a commit with an earlier\nOID.\n\nInstead, iterate over the full commit list at the end, checking the\noffsets to see how many grow beyond the maximum value.\n\nUpdate a test in t5318 to use a larger time value, which will help\ndemonstrate this bug in more cases. It still won't hit all potential\ncases until the next change, which reenables reading generation numbers.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 10 +++++++---\n t/t5318-commit-graph.sh | 4 ++--\n 2 files changed, 9 insertions(+), 5 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 265c010122e..a19bd96c2ee 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1556,12 +1556,16 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \t\t\t\tif (current->date && current->date > max_corrected_commit_date)\n \t\t\t\t\tmax_corrected_commit_date = current->date - 1;\n \t\t\t\tcommit_graph_data_at(current)->generation = max_corrected_commit_date + 1;\n-\n-\t\t\t\tif (commit_graph_data_at(current)->generation - current->date > GENERATION_NUMBER_V2_OFFSET_MAX)\n-\t\t\t\t\tctx->num_generation_data_overflows++;\n \t\t\t}\n \t\t}\n \t}\n+\n+\tfor (i = 0; i < ctx->commits.nr; i++) {\n+\t\tstruct commit *c = ctx->commits.list[i];\n+\t\ttimestamp_t offset = commit_graph_data_at(c)->generation - c->date;\n+\t\tif (offset > GENERATION_NUMBER_V2_OFFSET_MAX)\n+\t\t\tctx->num_generation_data_overflows++;\n+\t}\n \tstop_progress(&ctx->progress);\n }\n \ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 2b05026cf6d..f9bffe38013 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -467,10 +467,10 @@ test_expect_success 'warn on improper hash version' '\n \t)\n '\n \n-test_expect_success 'lower layers have overflow chunk' '\n+test_expect_success TIME_IS_64BIT,TIME_T_IS_64BIT 'lower layers have overflow chunk' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tUNIX_EPOCH_ZERO=\"@0 +0000\" &&\n-\tFUTURE_DATE=\"@2147483646 +0000\" &&\n+\tFUTURE_DATE=\"@4147483646 +0000\" &&\n \trm -f .git/objects/info/commit-graph &&\n \ttest_commit --date \"$FUTURE_DATE\" future-1 &&\n \ttest_commit --date \"$UNIX_EPOCH_ZERO\" old-1 &&\n-- \ngitgitgadget\n\n"},{"id":"449506","messageId":"2f89275314b4a2a89a18d14e41602bbe2e1988dc.1645735117.git.gitgitgadget@gmail.com","threadId":"57472","inReplyTo":"pull.1163.git.1645735117.gitgitgadget@gmail.com","subject":"[PATCH 1/7] test-read-graph: include extra post-parse info","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-24T20:38:30Z","receivedAt":"2022-02-24T20:38:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIt can be helpful to verify that the 'struct commit_graph' that results\nfrom parsing a commit-graph is correctly structured. The existence of\ndifferent chunks is not enough to verify that all of the optional\nfeatures are correctly enabled.\n\nUpdate 'test-tool read-graph' to output an \"options:\" line that includes\ninformation for different parts of the struct commit_graph.\n\nIn particular, this change demonstrates that the read_generation_data\noption is never being enabled, which will be fixed in a later change.\n\nSigned-off-by: Derrick Stolee \n---\n t/helper/test-read-graph.c | 13 +++++++++++++\n t/t4216-log-bloom.sh | 1 +\n t/t5318-commit-graph.sh | 1 +\n t/t5324-split-commit-graph.sh | 5 +++++\n 4 files changed, 20 insertions(+)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex 75927b2c81d..c3b6b8d1734 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -3,6 +3,7 @@\n #include \"commit-graph.h\"\n #include \"repository.h\"\n #include \"object-store.h\"\n+#include \"bloom.h\"\n \n int cmd__read_graph(int argc, const char **argv)\n {\n@@ -45,6 +46,18 @@ int cmd__read_graph(int argc, const char **argv)\n \t\tprintf(\" bloom_data\");\n \tprintf(\"\\n\");\n \n+\tprintf(\"options:\");\n+\tif (graph->bloom_filter_settings)\n+\t\tprintf(\" bloom(%d,%d,%d)\",\n+\t\t graph->bloom_filter_settings->hash_version,\n+\t\t graph->bloom_filter_settings->bits_per_entry,\n+\t\t graph->bloom_filter_settings->num_hashes);\n+\tif (graph->read_generation_data)\n+\t\tprintf(\" read_generation_data\");\n+\tif (graph->topo_levels)\n+\t\tprintf(\" topo_levels\");\n+\tprintf(\"\\n\");\n+\n \tUNLEAK(graph);\n \n \treturn 0;\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex cc3cebf6722..5ed6d2a21c1 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -48,6 +48,7 @@ graph_read_expect () {\n \theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata generation_data bloom_indexes bloom_data\n+\toptions: bloom(1,10,7)\n \tEOF\n \ttest-tool read-graph >actual &&\n \ttest_cmp expect actual\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex edb728f77c3..2b05026cf6d 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -104,6 +104,7 @@ graph_read_expect() {\n \theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata$OPTIONAL\n+\toptions:\n \tEOF\n \ttest-tool read-graph >output &&\n \ttest_cmp expect output\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 847b8097109..778fa418de2 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -34,6 +34,7 @@ graph_read_expect() {\n \theader: 43475048 1 $(test_oid oid_version) 4 $NUM_BASE\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata generation_data\n+\toptions:\n \tEOF\n \ttest-tool read-graph >output &&\n \ttest_cmp expect output\n@@ -508,6 +509,7 @@ test_expect_success 'setup repo for mixed generation commit-graph-chain' '\n \t\theader: 43475048 1 $(test_oid oid_version) 4 1\n \t\tnum_commits: $NUM_SECOND_LAYER_COMMITS\n \t\tchunks: oid_fanout oid_lookup commit_metadata\n+\t\toptions:\n \t\tEOF\n \t\ttest_cmp expect output &&\n \t\tgit commit-graph verify &&\n@@ -540,6 +542,7 @@ test_expect_success 'do not write generation data chunk if not present on existi\n \t\theader: 43475048 1 $(test_oid oid_version) 4 2\n \t\tnum_commits: $NUM_THIRD_LAYER_COMMITS\n \t\tchunks: oid_fanout oid_lookup commit_metadata\n+\t\toptions:\n \t\tEOF\n \t\ttest_cmp expect output &&\n \t\tgit commit-graph verify\n@@ -581,6 +584,7 @@ test_expect_success 'do not write generation data chunk if the topmost remaining\n \t\theader: 43475048 1 $(test_oid oid_version) 4 2\n \t\tnum_commits: $(($NUM_THIRD_LAYER_COMMITS + $NUM_FOURTH_LAYER_COMMITS))\n \t\tchunks: oid_fanout oid_lookup commit_metadata\n+\t\toptions:\n \t\tEOF\n \t\ttest_cmp expect output &&\n \t\tgit commit-graph verify\n@@ -620,6 +624,7 @@ test_expect_success 'write generation data chunk if topmost remaining layer has\n \t\theader: 43475048 1 $(test_oid oid_version) 5 1\n \t\tnum_commits: $(($NUM_SECOND_LAYER_COMMITS + $NUM_THIRD_LAYER_COMMITS + $NUM_FOURTH_LAYER_COMMITS + $NUM_FIFTH_LAYER_COMMITS))\n \t\tchunks: oid_fanout oid_lookup commit_metadata generation_data\n+\t\toptions:\n \t\tEOF\n \t\ttest_cmp expect output\n \t)\n-- \ngitgitgadget\n\n"},{"id":"449507","messageId":"a3436b92a32f7f6dd02ad61eb2337a4d088d5e9c.1645735117.git.gitgitgadget@gmail.com","threadId":"57472","inReplyTo":"pull.1163.git.1645735117.gitgitgadget@gmail.com","subject":"[PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-24T20:38:32Z","receivedAt":"2022-02-24T20:38:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'read_generation_data' member of 'struct commit_graph' was\nintroduced by 1fdc383c5 (commit-graph: use generation v2 only if entire\nchain does, 2021-01-16). The intention was to avoid using corrected\ncommit dates if not all layers of a commit-graph had that data stored.\nThe logic in validate_mixed_generation_chain() at that point incorrectly\ninitialized read_generation_data to 1 if and only if the tip\ncommit-graph contained the Corrected Commit Date chunk.\n\nThis was \"fixed\" in 448a39e65 (commit-graph: validate layers for\ngeneration data, 2021-02-02) to validate that read_generation_data was\neither non-zero for all layers, or it would set read_generation_data to\nzero for all layers.\n\nThe problem here is that read_generation_data is not initialized to be\nnon-zero anywhere!\n\nThis change initializes read_generation_data immediately after the chunk\nis parsed, so each layer will have its value present as soon as\npossible.\n\nThe read_generation_data member is used in fill_commit_graph_info() to\ndetermine if we should use the corrected commit date or the topological\nlevels stored in the Commit Data chunk. Due to this bug, all previous\nversions of Git were defaulting to topological levels in all cases!\n\nThis can be measured with some performance tests. Using the Linux kernel\nas a testbed, I generated a complete commit-graph containing corrected\ncommit dates and tested the 'new' version against the previous, 'old'\nversion.\n\nFirst, rev-list with --topo-order demonstrates a 26% improvement using\ncorrected commit dates:\n\nhyperfine \\\n\t-n \"old\" \"$OLD_GIT rev-list --topo-order -1000 v3.6\" \\\n\t-n \"new\" \"$NEW_GIT rev-list --topo-order -1000 v3.6\" \\\n\t--warmup=10\n\nBenchmark 1: old\n Time (mean ± σ): 57.1 ms ± 3.1 ms\n Range (min … max): 52.9 ms … 62.0 ms 55 runs\n\nBenchmark 2: new\n Time (mean ± σ): 45.5 ms ± 3.3 ms\n Range (min … max): 39.9 ms … 51.7 ms 59 runs\n\nSummary\n 'new' ran\n 1.26 ± 0.11 times faster than 'old'\n\nThese performance improvements are due to the algorithmic improvements\ngiven by walking fewer commits due to the higher cutoffs from corrected\ncommit dates.\n\nHowever, this comes at a cost. The additional I/O cost of parsing the\ncorrected commit dates is visible in case of merge-base commands that do\nnot reduce the overall number of walked commits.\n\nhyperfine \\\n -n \"old\" \"$OLD_GIT merge-base v4.8 v4.9\" \\\n -n \"new\" \"$NEW_GIT merge-base v4.8 v4.9\" \\\n --warmup=10\n\nBenchmark 1: old\n Time (mean ± σ): 110.4 ms ± 6.4 ms\n Range (min … max): 96.0 ms … 118.3 ms 25 runs\n\nBenchmark 2: new\n Time (mean ± σ): 150.7 ms ± 1.1 ms\n Range (min … max): 149.3 ms … 153.4 ms 19 runs\n\nSummary\n 'old' ran\n 1.36 ± 0.08 times faster than 'new'\n\nPerformance issues like this are what motivated 702110aac (commit-graph:\nuse config to specify generation type, 2021-02-25).\n\nIn the future, we could fix this performance problem by inserting the\ncorrected commit date offsets into the Commit Date chunk instead of\nhaving that data in an extra chunk.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 3 +++\n t/t4216-log-bloom.sh | 2 +-\n t/t5318-commit-graph.sh | 14 ++++++++++++--\n t/t5324-split-commit-graph.sh | 9 +++++++--\n 4 files changed, 23 insertions(+), 5 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a19bd96c2ee..8e52bb09552 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -407,6 +407,9 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t\t&graph->chunk_generation_data);\n \t\tpair_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW,\n \t\t\t&graph->chunk_generation_data_overflow);\n+\n+\t\tif (graph->chunk_generation_data)\n+\t\t\tgraph->read_generation_data = 1;\n \t}\n \n \tif (r->settings.commit_graph_read_changed_paths) {\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 5ed6d2a21c1..fa9d32facfb 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -48,7 +48,7 @@ graph_read_expect () {\n \theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata generation_data bloom_indexes bloom_data\n-\toptions: bloom(1,10,7)\n+\toptions: bloom(1,10,7) read_generation_data\n \tEOF\n \ttest-tool read-graph >actual &&\n \ttest_cmp expect actual\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex f9bffe38013..1afee1c2705 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -100,11 +100,21 @@ graph_read_expect() {\n \t\tOPTIONAL=\" $2\"\n \t\tNUM_CHUNKS=$((3 + $(echo \"$2\" | wc -w)))\n \tfi\n+\tGENERATION_VERSION=2\n+\tif test ! -z \"$3\"\n+\tthen\n+\t\tGENERATION_VERSION=$3\n+\tfi\n+\tOPTIONS=\n+\tif test $GENERATION_VERSION -gt 1\n+\tthen\n+\t\tOPTIONS=\" read_generation_data\"\n+\tfi\n \tcat >expect <<- EOF\n \theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata$OPTIONAL\n-\toptions:\n+\toptions:$OPTIONS\n \tEOF\n \ttest-tool read-graph >output &&\n \ttest_cmp expect output\n@@ -498,7 +508,7 @@ test_expect_success 'git commit-graph verify' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit rev-parse commits/8 | git -c commitGraph.generationVersion=1 commit-graph write --stdin-commits &&\n \tgit commit-graph verify >output &&\n-\tgraph_read_expect 9 extra_edges\n+\tgraph_read_expect 9 extra_edges 1\n '\n \n NUM_COMMITS=9\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 778fa418de2..669ddc645fa 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -30,11 +30,16 @@ graph_read_expect() {\n \tthen\n \t\tNUM_BASE=$2\n \tfi\n+\tOPTIONS=\n+\tif test -z \"$3\"\n+\tthen\n+\t\tOPTIONS=\" read_generation_data\"\n+\tfi\n \tcat >expect <<- EOF\n \theader: 43475048 1 $(test_oid oid_version) 4 $NUM_BASE\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata generation_data\n-\toptions:\n+\toptions:$OPTIONS\n \tEOF\n \ttest-tool read-graph >output &&\n \ttest_cmp expect output\n@@ -624,7 +629,7 @@ test_expect_success 'write generation data chunk if topmost remaining layer has\n \t\theader: 43475048 1 $(test_oid oid_version) 5 1\n \t\tnum_commits: $(($NUM_SECOND_LAYER_COMMITS + $NUM_THIRD_LAYER_COMMITS + $NUM_FOURTH_LAYER_COMMITS + $NUM_FIFTH_LAYER_COMMITS))\n \t\tchunks: oid_fanout oid_lookup commit_metadata generation_data\n-\t\toptions:\n+\t\toptions: read_generation_data\n \t\tEOF\n \t\ttest_cmp expect output\n \t)\n-- \ngitgitgadget\n\n"},{"id":"449508","messageId":"de7ab2f39d90c6b33b21a5adf126ec2ef5ce27e6.1645735117.git.gitgitgadget@gmail.com","threadId":"57472","inReplyTo":"pull.1163.git.1645735117.gitgitgadget@gmail.com","subject":"[PATCH 4/7] commit-graph: fix generation number v2 overflow values","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-24T20:38:33Z","receivedAt":"2022-02-24T20:38:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe Generation Data Chunk was implemented and tested in e8b63005c\n(commit-graph: implement generation data chunk, 2021-01-16), but the\ntest was carefully constructed to work on systems with 32-bit dates.\nSince the corrected commit date offsets still required more than 31\nbits, this triggered writing the generation_data_overflow chunk.\n\nHowever, upon closer look, the\nwrite_graph_chunk_generation_data_overflow() method writes the offsets\nto the chunk (as dictated by the format) but fill_commit_graph_info()\ntreats the value in the chunk as if it is the full corrected commit date\n(not an offset). For some reason, this does not cause an issue when\nusing the FUTURE_DATE specified in t5318-commit-graph.sh, but it does\nshow up as a failure in 'git commit-graph verify' if we increase that\nFUTURE_DATE to be above four billion.\n\nFix this error and update the test to require 64-bit dates so we can\nsafely use this large value in our test.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 2 +-\n t/t5318-commit-graph.sh | 21 +++++++++++++++++++--\n 2 files changed, 20 insertions(+), 3 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8e52bb09552..b86a6a634fe 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -806,7 +806,7 @@ static void fill_commit_graph_info(struct commit *item, struct commit_graph *g,\n \t\t\t\tdie(_(\"commit-graph requires overflow generation data but has none\"));\n \n \t\t\toffset_pos = offset ^ CORRECTED_COMMIT_DATE_OFFSET_OVERFLOW;\n-\t\t\tgraph_data->generation = get_be64(g->chunk_generation_data_overflow + 8 * offset_pos);\n+\t\t\tgraph_data->generation = item->date + get_be64(g->chunk_generation_data_overflow + 8 * offset_pos);\n \t\t} else\n \t\t\tgraph_data->generation = item->date + offset;\n \t} else\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 1afee1c2705..5e4b0216fa6 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -815,6 +815,19 @@ test_expect_success 'corrupt commit-graph write (missing tree)' '\n \t)\n '\n \n+# The remaining tests check timestamps that flow over\n+# 32-bits. The graph_git_behavior checks can't take a\n+# prereq, so just stop here if we are on a 32-bit machine.\n+\n+if ! test_have_prereq TIME_IS_64BIT\n+then\n+\ttest_done\n+fi\n+if ! test_have_prereq TIME_T_IS_64BIT\n+then\n+\ttest_done\n+fi\n+\n # We test the overflow-related code with the following repo history:\n #\n # 4:F - 5:N - 6:U\n@@ -832,10 +845,10 @@ test_expect_success 'corrupt commit-graph write (missing tree)' '\n # The largest offset observed is 2 ^ 31, just large enough to overflow.\n #\n \n-test_expect_success 'set up and verify repo with generation data overflow chunk' '\n+test_expect_success TIME_IS_64BIT,TIME_T_IS_64BIT 'set up and verify repo with generation data overflow chunk' '\n \tobjdir=\".git/objects\" &&\n \tUNIX_EPOCH_ZERO=\"@0 +0000\" &&\n-\tFUTURE_DATE=\"@2147483646 +0000\" &&\n+\tFUTURE_DATE=\"@4000000000 +0000\" &&\n \ttest_oid_cache <<-EOF &&\n \toid_version sha1:1\n \toid_version sha256:2\n@@ -867,4 +880,8 @@ test_expect_success 'set up and verify repo with generation data overflow chunk'\n \n graph_git_behavior 'generation data overflow chunk repo' repo left right\n \n+# Do not add tests at the end of this file, unless they require 64-bit\n+# timestamps, since this portion of the script is only executed when\n+# time data types have 64 bits.\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"449509","messageId":"7f9b65bd22551fd7fd5d2f0bf18aee8c25f1db99.1645735117.git.gitgitgadget@gmail.com","threadId":"57472","inReplyTo":"pull.1163.git.1645735117.gitgitgadget@gmail.com","subject":"[PATCH 5/7] commit-graph: document file format v2","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-24T20:38:34Z","receivedAt":"2022-02-24T20:38:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe corrected commit date was first documented in 5a3b130ca (doc: add\ncorrected commit date info, 2021-01-16) and it used an optional chunk to\naugment the commit-graph format without modifying the file format\nversion.\n\nOne major benefit to this approach is that corrected commit dates could\nbe written without causing a backwards compatibility issue with Git\nversions that do not understand them. The topological level was still\navailable in the CDAT chunk as it was before.\n\nHowever, this causes a different issue: more data needs to be loaded\nfrom disk when parsing commits from the commit-graph. In cases where\nthere is no significant algorithmic gain from using corrected commit\ndates, commit walks take up to 20% longer because of this extra data.\n\nCreate a new file format version for the commit-graph format that\ndiffers only in the CDAT chunk: it now stores corrected commit date\noffsets. This brings our data back to normal and will demonstrate\nperformance gains in almost all cases.\n\nSigned-off-by: Derrick Stolee \n---\n .../technical/commit-graph-format.txt | 22 ++++++++++++++-----\n 1 file changed, 17 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex 87971c27dd7..2cb48993314 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -36,7 +36,7 @@ HEADER:\n The signature is: {'C', 'G', 'P', 'H'}\n \n 1-byte version number:\n- Currently, the only valid version is 1.\n+ This version number can be 1 or 2.\n \n 1-byte Hash Version\n We infer the hash length (H) from this value:\n@@ -85,13 +85,22 @@ CHUNK DATA:\n position. If there are more than two parents, the second value\n has its most-significant bit on and the other bits store an array\n position into the Extra Edge List chunk.\n- * The next 8 bytes store the topological level (generation number v1)\n- of the commit and\n- the commit time in seconds since EPOCH. The generation number\n- uses the higher 30 bits of the first 4 bytes, while the commit\n+ * The next 8 bytes store the generation number information of the\n+ commit and the commit time in seconds since EPOCH. The generation\n+ number uses the higher 30 bits of the first 4 bytes, while the commit\n time uses the 32 bits of the second 4 bytes, along with the lowest\n 2 bits of the lowest byte, storing the 33rd and 34th bit of the\n commit time.\n+ - If the commit-graph file format is version 1, then the higher 30\n+\tbits contain the topological level (generation number v1) for the\n+\tcommit.\n+ - If the commit-graph file format is version 2, then the higher 30\n+\tbits contain the corrected commit date offset (generation number\n+\tv2) for the commit, except if the offset cannot be stored within\n+\t29 bits. If the offset is too large for 29 bits, then the value\n+\tstored here has its most-significant bit on and the other bits\n+\tstore the position of the corrected commit date in the Generation\n+\tDate Overflow chunk.\n \n Generation Data (ID: {'G', 'D', 'A', 'T' }) (N * 4 bytes) [Optional]\n * This list of 4-byte values store corrected commit date offsets for the\n@@ -103,6 +112,9 @@ CHUNK DATA:\n * Generation Data chunk is present only when commit-graph file is written\n by compatible versions of Git and in case of split commit-graph chains,\n the topmost layer also has Generation Data chunk.\n+ * This chunk does not exist if the commit-graph file format version is 2,\n+ because the corrected commit date offset data is stored in the Commit\n+ Data chunk.\n \n Generation Data Overflow (ID: {'G', 'D', 'O', 'V' }) [Optional]\n * This list of 8-byte values stores the corrected commit date offsets\n-- \ngitgitgadget\n\n"},{"id":"449510","messageId":"28fe8824ba71ff9cda5ec5c034b6a6fb3c857654.1645735117.git.gitgitgadget@gmail.com","threadId":"57472","inReplyTo":"pull.1163.git.1645735117.gitgitgadget@gmail.com","subject":"[PATCH 6/7] commit-graph: parse file format v2","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-24T20:38:35Z","receivedAt":"2022-02-24T20:38:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe commit-graph file format v2 alters how it stores the corrected\ncommit date offsets within the Commit Data chunk instead of a separate\nchunk. The idea is to significantly reduce the amount of data loaded\nfrom disk while parsing the commit-graph.\n\nWe need to alter the error message when we see a file format version\noutside of our range now that multiple are possible. This has a\nnon-functional side-effect of altering a use of GRAPH_VERSION within\nwrite_commit_graph().\n\nBy storing the file format version in 'struct commit_graph', we can\nalter the parsing code to depend on that version value. This involves\nchanging where we look for the corrected commit date offset, but also\nwhich constants we use for jumping into the Generation Data Overflow\nchunk. The Commit Data chunk only has 30 bits available for the offset\nwhile the Generation Data chunk has 32 bits. This only makes a\nmeaningful difference in very malformed repositories.\n\nAlso, we need to be careful about how we enable using corrected commit\ndates and generation numbers to rely upon the read_generation_data value\ninstead of a non-zero value in the Commit Date chunk. In\ngeneration_numbers_enabled(), the first_generation variable is\nattemptint to look for the first topological level stored to see that it\nis nonzero. However, for a v2 commit-graph, this value is actually\nlikely to be zero because the corrected commit date offset is probably\nzero.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 43 +++++++++++++++++++++++++++++------------\n commit-graph.h | 6 ++++++\n t/t5318-commit-graph.sh | 2 +-\n 3 files changed, 38 insertions(+), 13 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex b86a6a634fe..366fc4d6e41 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -49,7 +49,9 @@ void git_test_write_commit_graph_or_die(void)\n #define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n \n #define GRAPH_VERSION_1 0x1\n-#define GRAPH_VERSION GRAPH_VERSION_1\n+#define GRAPH_VERSION_2 0x2\n+#define GRAPH_VERSION_MIN GRAPH_VERSION_1\n+#define GRAPH_VERSION_MAX GRAPH_VERSION_2\n \n #define GRAPH_EXTRA_EDGES_NEEDED 0x80000000\n #define GRAPH_EDGE_LAST_MASK 0x7fffffff\n@@ -63,6 +65,7 @@ void git_test_write_commit_graph_or_die(void)\n \t\t\t+ GRAPH_FANOUT_SIZE + the_hash_algo->rawsz)\n \n #define CORRECTED_COMMIT_DATE_OFFSET_OVERFLOW (1ULL << 31)\n+#define CORRECTED_COMMIT_DATE_OFFSET_OVERFLOW_V3 (1ULL << 29)\n \n /* Remember to update object flag allocation in object.h */\n #define REACHABLE (1u<<15)\n@@ -358,9 +361,10 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t}\n \n \tgraph_version = *(unsigned char*)(data + 4);\n-\tif (graph_version != GRAPH_VERSION) {\n-\t\terror(_(\"commit-graph version %X does not match version %X\"),\n-\t\t graph_version, GRAPH_VERSION);\n+\tif (graph_version < GRAPH_VERSION_MIN ||\n+\t graph_version > GRAPH_VERSION_MAX) {\n+\t\terror(_(\"commit-graph version %X is not understood\"),\n+\t\t graph_version);\n \t\treturn NULL;\n \t}\n \n@@ -375,6 +379,7 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \n \tgraph = alloc_commit_graph();\n \n+\tgraph->version = graph_version;\n \tgraph->hash_len = the_hash_algo->rawsz;\n \tgraph->num_chunks = *(unsigned char*)(data + 6);\n \tgraph->data = graph_map;\n@@ -402,13 +407,17 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \tpair_chunk(cf, GRAPH_CHUNKID_EXTRAEDGES, &graph->chunk_extra_edges);\n \tpair_chunk(cf, GRAPH_CHUNKID_BASE, &graph->chunk_base_graphs);\n \n-\tif (get_configured_generation_version(r) >= 2) {\n-\t\tpair_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA,\n-\t\t\t&graph->chunk_generation_data);\n+\tif (graph_version >= GRAPH_VERSION_2 ||\n+\t get_configured_generation_version(r) >= 2) {\n+\t\t/* Skip this chunk if GRAPH_VERSION_2 or higher. */\n+\t\tif (graph_version == GRAPH_VERSION_1)\n+\t\t\tpair_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA,\n+\t\t\t\t &graph->chunk_generation_data);\n \t\tpair_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW,\n \t\t\t&graph->chunk_generation_data_overflow);\n \n-\t\tif (graph->chunk_generation_data)\n+\t\tif (graph_version >= GRAPH_VERSION_2 ||\n+\t\t graph->chunk_generation_data)\n \t\t\tgraph->read_generation_data = 1;\n \t}\n \n@@ -683,6 +692,9 @@ int generation_numbers_enabled(struct repository *r)\n \tif (!g->num_commits)\n \t\treturn 0;\n \n+\tif (g->version >= GRAPH_VERSION_2)\n+\t\treturn g->read_generation_data;\n+\n \tfirst_generation = get_be32(g->chunk_commit_data +\n \t\t\t\t g->hash_len + 8) >> 2;\n \n@@ -799,13 +811,20 @@ static void fill_commit_graph_info(struct commit *item, struct commit_graph *g,\n \titem->date = (timestamp_t)((date_high << 32) | date_low);\n \n \tif (g->read_generation_data) {\n-\t\toffset = (timestamp_t)get_be32(g->chunk_generation_data + sizeof(uint32_t) * lex_index);\n+\t\ttimestamp_t overflow_bit;\n+\t\tif (g->version == GRAPH_VERSION_2) {\n+\t\t\toffset = (timestamp_t)(get_be32(commit_data + g->hash_len + 8) >> 2);\n+\t\t\toverflow_bit = CORRECTED_COMMIT_DATE_OFFSET_OVERFLOW_V3;\n+\t\t} else {\n+\t\t\toffset = (timestamp_t)get_be32(g->chunk_generation_data + sizeof(uint32_t) * lex_index);\n+\t\t\toverflow_bit = CORRECTED_COMMIT_DATE_OFFSET_OVERFLOW;\n+\t\t}\n \n-\t\tif (offset & CORRECTED_COMMIT_DATE_OFFSET_OVERFLOW) {\n+\t\tif (offset & overflow_bit) {\n \t\t\tif (!g->chunk_generation_data_overflow)\n \t\t\t\tdie(_(\"commit-graph requires overflow generation data but has none\"));\n \n-\t\t\toffset_pos = offset ^ CORRECTED_COMMIT_DATE_OFFSET_OVERFLOW;\n+\t\t\toffset_pos = offset ^ overflow_bit;\n \t\t\tgraph_data->generation = item->date + get_be64(g->chunk_generation_data_overflow + 8 * offset_pos);\n \t\t} else\n \t\t\tgraph_data->generation = item->date + offset;\n@@ -1917,7 +1936,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \n \thashwrite_be32(f, GRAPH_SIGNATURE);\n \n-\thashwrite_u8(f, GRAPH_VERSION);\n+\thashwrite_u8(f, GRAPH_VERSION_1);\n \thashwrite_u8(f, oid_version());\n \thashwrite_u8(f, get_num_chunks(cf));\n \thashwrite_u8(f, ctx->num_commit_graphs_after - 1);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 04a94e18302..b379b8eae25 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -63,6 +63,12 @@ struct commit_graph {\n \tconst unsigned char *data;\n \tsize_t data_len;\n \n+\t/**\n+\t * The 'version' byte mirrors the file format version. This is\n+\t * necessary to consider when parsing commits.\n+\t */\n+\tunsigned version;\n+\n \tunsigned char hash_len;\n \tunsigned char num_chunks;\n \tuint32_t num_commits;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 5e4b0216fa6..a14a13e5f7b 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -605,7 +605,7 @@ test_expect_success 'detect bad signature' '\n '\n \n test_expect_success 'detect bad version' '\n-\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\02\" \\\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\03\" \\\n \t\t\"graph version\"\n '\n \n-- \ngitgitgadget\n\n"},{"id":"449511","messageId":"ade697c4d34a43d09db5be686591b2dc7d68f012.1645735117.git.gitgitgadget@gmail.com","threadId":"57472","inReplyTo":"pull.1163.git.1645735117.gitgitgadget@gmail.com","subject":"[PATCH 7/7] commit-graph: write file format v2","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-24T20:38:36Z","receivedAt":"2022-02-24T20:39:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe commit-graph file format v2 exists so we can modify the meaning of\nthe Commit Data chunk to store corrected commit date offsets. Thus, we\ntrigger the write to use this different file format only if the\nconfigured generation number version is 3.\n\nThe implementation needs to be careful of a few things to ensure we\nenable computing corrected commit dates and do not compute topological\nlevels. We also still need the Generation Data Overflow chunk, but we\ncompute the offsets into that chunk while writing the Commit Data chunk\ninstead of the generation Data chunk.\n\nTesting 'git merge-base v4.8 v4.9' in the Linux kernel with corrected\ncommit dates, but the only difference being the file format (between\ngeneration number v2 and v3) we get these results:\n\nBenchmark 1: generation number v2\n Time (mean ± σ): 144.4 ms ± 8.3 ms\n Range (min … max): 127.4 ms … 154.6 ms 20 runs\n\nBenchmark 2: generation number v3\n Time (mean ± σ): 139.3 ms ± 7.3 ms\n Range (min … max): 125.1 ms … 148.1 ms 20 runs\n\nThis provides a 3.6% improvement, and the only reason is the reduced\nI/O. This test was run with hot caches, so I re-ran it in the cold-cache\ncase, trying to demonstrate that this I/O cost is higher when reading\ndirectly from disk every time:\n\nBenchmark 1: generation number v2\n Time (mean ± σ): 469.9 ms ± 14.8 ms\n Range (min … max): 434.5 ms … 494.4 ms 10 runs\n\nBenchmark 2: generation number v3\n Time (mean ± σ): 413.4 ms ± 18.9 ms\n Range (min … max): 372.8 ms … 428.3 ms 10 runs\n\nWith cold caches, the improvement increases to 13.4%.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/commitgraph.txt | 4 ++-\n commit-graph.c | 46 +++++++++++++++++++++++++---\n commit.h | 1 +\n t/t5318-commit-graph.sh | 25 ++++++++++++++-\n 4 files changed, 69 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex 30604e4a4c2..79d57d06a67 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -1,7 +1,9 @@\n commitGraph.generationVersion::\n \tSpecifies the type of generation number version to use when writing\n \tor reading the commit-graph file. If version 1 is specified, then\n-\tthe corrected commit dates will not be written or read. Defaults to\n+\tthe corrected commit dates will not be written or read. If version\n+\t3 is specified, then the commit-graph file will be slightly smaller,\n+\tbut will be incompatible with some old versions of Git. Defaults to\n \t2.\n \n commitGraph.maxNewFilters::\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 366fc4d6e41..82f7401b283 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1035,6 +1035,7 @@ struct write_commit_graph_context {\n \tstruct progress *progress;\n \tint progress_done;\n \tuint64_t progress_cnt;\n+\tint version;\n \n \tchar *base_graph_name;\n \tint num_commit_graphs_before;\n@@ -1118,12 +1119,14 @@ static int write_graph_chunk_data(struct hashfile *f,\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tuint32_t num_extra_edges = 0;\n+\tint num_generation_data_overflows = 0;\n \n \twhile (list < last) {\n \t\tstruct commit_list *parent;\n \t\tstruct object_id *tree;\n \t\tint edge_value;\n \t\tuint32_t packedDate[2];\n+\t\tuint32_t generation_data;\n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n \n \t\tif (repo_parse_commit_no_graph(ctx->r, *list))\n@@ -1203,7 +1206,18 @@ static int write_graph_chunk_data(struct hashfile *f,\n \t\telse\n \t\t\tpackedDate[0] = 0;\n \n-\t\tpackedDate[0] |= htonl(*topo_level_slab_at(ctx->topo_levels, *list) << 2);\n+\t\tif (ctx->version == GRAPH_VERSION_1)\n+\t\t\tgeneration_data = *topo_level_slab_at(ctx->topo_levels, *list);\n+\t\telse {\n+\t\t\tgeneration_data = commit_graph_data_at(*list)->generation - (*list)->date;\n+\t\t\tif (generation_data > GENERATION_NUMBER_V3_OFFSET_MAX) {\n+\t\t\t\tgeneration_data = CORRECTED_COMMIT_DATE_OFFSET_OVERFLOW_V3 |\n+\t\t\t\t\t\t num_generation_data_overflows;\n+\t\t\t\tnum_generation_data_overflows++;\n+\t\t\t}\n+\t\t}\n+\n+\t\tpackedDate[0] |= htonl(generation_data << 2);\n \n \t\tpackedDate[1] = htonl((*list)->date);\n \t\thashwrite(f, packedDate, 8);\n@@ -1243,12 +1257,16 @@ static int write_graph_chunk_generation_data_overflow(struct hashfile *f,\n {\n \tstruct write_commit_graph_context *ctx = data;\n \tint i;\n+\ttimestamp_t offset_max = ctx->version >= 2 ?\n+\t\t\t\t\tGENERATION_NUMBER_V3_OFFSET_MAX :\n+\t\t\t\t\tGENERATION_NUMBER_V2_OFFSET_MAX;\n+\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tstruct commit *c = ctx->commits.list[i];\n \t\ttimestamp_t offset = commit_graph_data_at(c)->generation - c->date;\n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n \n-\t\tif (offset > GENERATION_NUMBER_V2_OFFSET_MAX) {\n+\t\tif (offset > offset_max) {\n \t\t\thashwrite_be32(f, offset >> 32);\n \t\t\thashwrite_be32(f, (uint32_t) offset);\n \t\t}\n@@ -1474,6 +1492,13 @@ static void compute_topological_levels(struct write_commit_graph_context *ctx)\n \tint i;\n \tstruct commit_list *list = NULL;\n \n+\t/*\n+\t * Skip topological levels if file format version is two or more,\n+\t * since the Commit Data chunk uses corrected commit date offsets.\n+\t */\n+\tif (ctx->version >= 2)\n+\t\treturn;\n+\n \tif (ctx->report_progress)\n \t\tctx->progress = start_delayed_progress(\n \t\t\t\t\t_(\"Computing commit graph topological levels\"),\n@@ -1526,6 +1551,9 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n {\n \tint i;\n \tstruct commit_list *list = NULL;\n+\ttimestamp_t offset_max = ctx->version >= 2 ?\n+\t\t\t\t\tGENERATION_NUMBER_V3_OFFSET_MAX :\n+\t\t\t\t\tGENERATION_NUMBER_V2_OFFSET_MAX;\n \n \tif (ctx->report_progress)\n \t\tctx->progress = start_delayed_progress(\n@@ -1585,7 +1613,7 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tstruct commit *c = ctx->commits.list[i];\n \t\ttimestamp_t offset = commit_graph_data_at(c)->generation - c->date;\n-\t\tif (offset > GENERATION_NUMBER_V2_OFFSET_MAX)\n+\t\tif (offset > offset_max)\n \t\t\tctx->num_generation_data_overflows++;\n \t}\n \tstop_progress(&ctx->progress);\n@@ -1908,7 +1936,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tadd_chunk(cf, GRAPH_CHUNKID_DATA, (hashsz + 16) * ctx->commits.nr,\n \t\t write_graph_chunk_data);\n \n-\tif (ctx->write_generation_data)\n+\tif (ctx->write_generation_data && ctx->version == GRAPH_VERSION_1)\n \t\tadd_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA,\n \t\t\t sizeof(uint32_t) * ctx->commits.nr,\n \t\t\t write_graph_chunk_generation_data);\n@@ -1936,7 +1964,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \n \thashwrite_be32(f, GRAPH_SIGNATURE);\n \n-\thashwrite_u8(f, GRAPH_VERSION_1);\n+\thashwrite_u8(f, ctx->version);\n \thashwrite_u8(f, oid_version());\n \thashwrite_u8(f, get_num_chunks(cf));\n \thashwrite_u8(f, ctx->num_commit_graphs_after - 1);\n@@ -2317,6 +2345,14 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->write_generation_data = (get_configured_generation_version(r) == 2);\n \tctx->num_generation_data_overflows = 0;\n \n+\tif (get_configured_generation_version(r) == 3)\n+\t\tctx->version = GRAPH_VERSION_2;\n+\telse\n+\t\tctx->version = GRAPH_VERSION_1;\n+\n+\tif (ctx->version >= GRAPH_VERSION_2)\n+\t\tctx->write_generation_data = 1;\n+\n \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n \t\t\t\t\t\t bloom_settings.bits_per_entry);\n \tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\ndiff --git a/commit.h b/commit.h\nindex 38cc5426615..a668b5cdec0 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -15,6 +15,7 @@\n #define GENERATION_NUMBER_V1_MAX 0x3FFFFFFF\n #define GENERATION_NUMBER_ZERO 0\n #define GENERATION_NUMBER_V2_OFFSET_MAX ((1ULL << 31) - 1)\n+#define GENERATION_NUMBER_V3_OFFSET_MAX ((1ULL << 29) - 1)\n \n struct commit_list {\n \tstruct commit *item;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex a14a13e5f7b..77e130ef63e 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -110,8 +110,13 @@ graph_read_expect() {\n \tthen\n \t\tOPTIONS=\" read_generation_data\"\n \tfi\n+\tVERSION=1\n+\tif test $GENERATION_VERSION -gt 2\n+\tthen\n+\t\tVERSION=2\n+\tfi\n \tcat >expect <<- EOF\n-\theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n+\theader: 43475048 $VERSION $(test_oid oid_version) $NUM_CHUNKS 0\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata$OPTIONAL\n \toptions:$OPTIONS\n@@ -343,6 +348,15 @@ test_expect_success 'build graph using --reachable' '\n graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n \n+test_expect_success 'write file format v2 with generation number v3' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit -c commitGraph.generationVersion=3 commit-graph write --reachable &&\n+\tgraph_read_expect \"11\" \"extra_edges\" 3\n+'\n+\n+graph_git_behavior 'graph v2, commit 8 vs merge 1' full commits/8 merge/1\n+graph_git_behavior 'graph v2, commit 8 vs merge 2' full commits/8 merge/2\n+\n test_expect_success 'setup bare repo' '\n \tcd \"$TRASH_DIRECTORY\" &&\n \tgit clone --bare --no-local full bare &&\n@@ -880,6 +894,15 @@ test_expect_success TIME_IS_64BIT,TIME_T_IS_64BIT 'set up and verify repo with g\n \n graph_git_behavior 'generation data overflow chunk repo' repo left right\n \n+test_expect_success TIME_IS_64BIT,TIME_T_IS_64BIT 'set up and verify repo with generation data overflow chunk (v3)' '\n+\tcd \"$TRASH_DIRECTORY/repo\" &&\n+\tgit -c commitGraph.generationVersion=3 commit-graph write --reachable &&\n+\tgraph_read_expect 10 \"generation_data_overflow\" 3 &&\n+\tgit commit-graph verify\n+'\n+\n+graph_git_behavior 'generation data overflow chunk repo' repo left right\n+\n # Do not add tests at the end of this file, unless they require 64-bit\n # timestamps, since this portion of the script is only executed when\n # time data types have 64 bits.\n-- \ngitgitgadget\n"},{"id":"449513","messageId":"xmqqsfs8j60g.fsf@gitster.g","threadId":"57472","inReplyTo":"pull.1163.git.1645735117.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/7] Commit-graph: Generation Number v2 Fixes, v3 implementation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-24T21:42:39Z","receivedAt":"2022-02-24T21:42:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> This patch series includes two distinct, but similarly-motivated parts:\n>\n> * Patches 1-4 fix some bugs in the commit-graph generation number v2.\n> * Patches 5-7 add a new generation number v3 by incrementing the\n> commit-graph file format.\n>\n> I had been thinking about generation number v3, which is the same corrected\n> commit date as generation number v2, but it is stored in the Commit Data\n> chunk, requiring a new commit-graph file format version. This breaks\n> compatibility with older versions of Git, so it requires opt-in via the\n> commitGraph.generationVersion config value. The only improvement over\n> version 2 is that the commit-graph file is smaller, so I/O time is reduced.\n\nSounds exciting. Locality of on-disk data does matter.\n\n> However, while exploring this idea I found bugs in generation number v2. In\n> particular, Git has been ignoring them since shortly after they were\n> introduced. This is due to a bug I introduced when trying to make split\n> commit-graphs safer with mixed generation number versions. I also noticed an\n> issue with the offset overflows that I only noticed after writing generation\n> number v3 using a smaller offset size, actually triggering the bug in the\n> test suite.\n>\n> I'm submitting these two things together so we can see them all at once, but\n> I'd be happy to split this into two series. The first four patches are\n> important bug fixes, so we can consider them as higher-priority.\n>\n> Thanks, -Stolee\n\nThanks, will take a look.\n\n"},{"id":"449515","messageId":"xmqqh78nkj2x.fsf@gitster.g","threadId":"57472","inReplyTo":"6e47ffed25795260c2b8614d4589fb58d892c8df.1645735117.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/7] commit-graph: fix ordering bug in generation numbers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-24T22:15:02Z","receivedAt":"2022-02-24T22:15:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> When computing the generation numbers for a commit-graph, we compute\n> the corrected commit dates and then check if their offsets from the\n> actual dates is too large to fit in the 32-bit Generation Data chunk.\n> However, there is a problem with this approach: if we have parsed the\n> generation data from the previous commit-graph, then we continue the\n> loop because the corrected commit date is already computed.\n>\n> It is incorrect to add an increment to num_generation_data_overflows\n> here, because we might start double-counting commits that are computed\n> because of the depth-first search walk from a commit with an earlier\n> OID.\n>\n> Instead, iterate over the full commit list at the end, checking the\n> offsets to see how many grow beyond the maximum value.\n\nHmph, I can see how the new code correctly counts the commits that\nrequire offsets that are too large, but I am not sure why the fix is\nneeded. The overall loop structure is\n\n for each commit ctx->commits.list[i]:\n continue if generation number has been computed for it already\n\n\tset up a commit-list for depth first search\n\twhile (we are still digging) {\n\t\tfor each parent {\n\t\t\tif generation for the parent is not known yet:\n\t\t\t\tpush it down and redo\n\t\t\telse\n\t\t\t\tcompute max of parents' generation number\n\t\t}\n if (all parents' generation number is known) {\n\t\t\tset the generation number for ourselves\n\t\t\tcount if we needed an offset that is too big\n\t\t}\n\t}\n }\n\nThe only case where we may double-count near the end of inner loop I\ncan think of is when we end up computing generation for the same\ncommit in the while () loop. But isn't that \"we dig the same thing\ntwice\" by itself something we want to fix, regardless of the\ndouble-counting issue?\n\nIOW,\n\n> \t\t\t\tif (current->date && current->date > max_corrected_commit_date)\n> \t\t\t\t\tmax_corrected_commit_date = current->date - 1;\n> \t\t\t\tcommit_graph_data_at(current)->generation = max_corrected_commit_date + 1;\n> -\n> -\t\t\t\tif (commit_graph_data_at(current)->generation - current->date > GENERATION_NUMBER_V2_OFFSET_MAX)\n> -\t\t\t\t\tctx->num_generation_data_overflows++;\n> \t\t\t}\n> \t\t}\n> \t}\n\nhere, before doing the assignment for the \"current\" commit's\ngeneration number, if we added\n\n\t\tif (commit_graph_data_at(current)->generation !=\n\t\t GENERATION_NUMBER_ZERO)\n\t\t\tBUG(\"why are we digging it twice?\");\n\nwould it trigger? If so, isn't that already a bug worth fixing?\n\nPerhaps avoiding the second round, perhaps like this, may be a\nbetter fix?\n\n\twhile (list) {\n\t\tstruct commit *current = list->item;\n\t\tstruct commit_list *parent;\n\t\tint all_parents_computed = 1;\n\t\ttimestamp_t max_corrected_commit_date = 0;\n\n+\t\tif (commit_graph_data_at(current)->generation !=\n+\t\t GENERATION_NUMBER_ZERO) {\n+\t\t\tpop_commit(&list);\n+\t\t\tcontinue;\n+\t\t}\n+\n\t\tfor (parent = current->parents; parent; parent = parent->next) {\n\nOr am I grossly misunderstanding why the original code is incorrect\nto have the counting at this place?\n\n> +\n> +\tfor (i = 0; i < ctx->commits.nr; i++) {\n> +\t\tstruct commit *c = ctx->commits.list[i];\n> +\t\ttimestamp_t offset = commit_graph_data_at(c)->generation - c->date;\n> +\t\tif (offset > GENERATION_NUMBER_V2_OFFSET_MAX)\n> +\t\t\tctx->num_generation_data_overflows++;\n> +\t}\n> \tstop_progress(&ctx->progress);\n> }\n> \n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 2b05026cf6d..f9bffe38013 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -467,10 +467,10 @@ test_expect_success 'warn on improper hash version' '\n> \t)\n> '\n> \n> -test_expect_success 'lower layers have overflow chunk' '\n> +test_expect_success TIME_IS_64BIT,TIME_T_IS_64BIT 'lower layers have overflow chunk' '\n> \tcd \"$TRASH_DIRECTORY/full\" &&\n> \tUNIX_EPOCH_ZERO=\"@0 +0000\" &&\n> -\tFUTURE_DATE=\"@2147483646 +0000\" &&\n> +\tFUTURE_DATE=\"@4147483646 +0000\" &&\n> \trm -f .git/objects/info/commit-graph &&\n> \ttest_commit --date \"$FUTURE_DATE\" future-1 &&\n> \ttest_commit --date \"$UNIX_EPOCH_ZERO\" old-1 &&\n"},{"id":"449524","messageId":"xmqqbkyvki54.fsf@gitster.g","threadId":"57472","inReplyTo":"de7ab2f39d90c6b33b21a5adf126ec2ef5ce27e6.1645735117.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/7] commit-graph: fix generation number v2 overflow values","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-24T22:35:19Z","receivedAt":"2022-02-24T22:35:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> -\t\t\tgraph_data->generation = get_be64(g->chunk_generation_data_overflow + 8 * offset_pos);\n> +\t\t\tgraph_data->generation = item->date + get_be64(g->chunk_generation_data_overflow + 8 * offset_pos);\n\nWow, that's embarrassing.\n\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 1afee1c2705..5e4b0216fa6 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -815,6 +815,19 @@ test_expect_success 'corrupt commit-graph write (missing tree)' '\n> \t)\n> '\n> \n> +# The remaining tests check timestamps that flow over\n> +# 32-bits. The graph_git_behavior checks can't take a\n> +# prereq, so just stop here if we are on a 32-bit machine.\n> +\n> +if ! test_have_prereq TIME_IS_64BIT\n> +then\n> +\ttest_done\n> +fi\n> +if ! test_have_prereq TIME_T_IS_64BIT\n> +then\n> +\ttest_done\n> +fi\n\nThe above is OK but is there a reason why we cannot do\n\n\tif A || B\n\tthen\n\t\ttest_done\n\tfi\n\nhere (I am assuming not, but in case I am missing the reason why it\nhas to be separate)?\n\n> @@ -832,10 +845,10 @@ test_expect_success 'corrupt commit-graph write (missing tree)' '\n> # The largest offset observed is 2 ^ 31, just large enough to overflow.\n> #\n> \n> -test_expect_success 'set up and verify repo with generation data overflow chunk' '\n> +test_expect_success TIME_IS_64BIT,TIME_T_IS_64BIT 'set up and verify repo with generation data overflow chunk' '\n> \tobjdir=\".git/objects\" &&\n> \tUNIX_EPOCH_ZERO=\"@0 +0000\" &&\n> -\tFUTURE_DATE=\"@2147483646 +0000\" &&\n> +\tFUTURE_DATE=\"@4000000000 +0000\" &&\n\nOK. 16#EE6B2800 too large to fit and will cause wrapping around with\nsigned 32-bit integer.\n\n> @@ -867,4 +880,8 @@ test_expect_success 'set up and verify repo with generation data overflow chunk'\n> \n> graph_git_behavior 'generation data overflow chunk repo' repo left right\n> \n> +# Do not add tests at the end of this file, unless they require 64-bit\n> +# timestamps, since this portion of the script is only executed when\n> +# time data types have 64 bits.\n> +\n> test_done\n"},{"id":"449527","messageId":"xmqq5yp3kh7o.fsf@gitster.g","threadId":"57472","inReplyTo":"7f9b65bd22551fd7fd5d2f0bf18aee8c25f1db99.1645735117.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/7] commit-graph: document file format v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-24T22:55:23Z","receivedAt":"2022-02-24T22:55:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> The corrected commit date was first documented in 5a3b130ca (doc: add\n> corrected commit date info, 2021-01-16) and it used an optional chunk to\n> augment the commit-graph format without modifying the file format\n> version.\n>\n> One major benefit to this approach is that corrected commit dates could\n> be written without causing a backwards compatibility issue with Git\n> versions that do not understand them. The topological level was still\n> available in the CDAT chunk as it was before.\n>\n> However, this causes a different issue: more data needs to be loaded\n> from disk when parsing commits from the commit-graph. In cases where\n> there is no significant algorithmic gain from using corrected commit\n> dates, commit walks take up to 20% longer because of this extra data.\n>\n> Create a new file format version for the commit-graph format that\n> differs only in the CDAT chunk: it now stores corrected commit date\n> offsets. This brings our data back to normal and will demonstrate\n> performance gains in almost all cases.\n\nOK.\n"},{"id":"449528","messageId":"xmqq1qzrkgxg.fsf@gitster.g","threadId":"57472","inReplyTo":"28fe8824ba71ff9cda5ec5c034b6a6fb3c857654.1645735117.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 6/7] commit-graph: parse file format v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-24T23:01:31Z","receivedAt":"2022-02-24T23:01:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> The commit-graph file format v2 alters how it stores the corrected\n> commit date offsets within the Commit Data chunk instead of a separate\n> chunk. The idea is to significantly reduce the amount of data loaded\n> from disk while parsing the commit-graph.\n>\n> We need to alter the error message when we see a file format version\n> outside of our range now that multiple are possible. This has a\n> non-functional side-effect of altering a use of GRAPH_VERSION within\n> write_commit_graph().\n>\n> By storing the file format version in 'struct commit_graph', we can\n> alter the parsing code to depend on that version value. This involves\n> changing where we look for the corrected commit date offset, but also\n> which constants we use for jumping into the Generation Data Overflow\n> chunk. The Commit Data chunk only has 30 bits available for the offset\n> while the Generation Data chunk has 32 bits. This only makes a\n> meaningful difference in very malformed repositories.\n>\n> Also, we need to be careful about how we enable using corrected commit\n> dates and generation numbers to rely upon the read_generation_data value\n> instead of a non-zero value in the Commit Date chunk. In\n> generation_numbers_enabled(), the first_generation variable is\n> attemptint to look for the first topological level stored to see that it\n> is nonzero. However, for a v2 commit-graph, this value is actually\n> likely to be zero because the corrected commit date offset is probably\n> zero.\n\nI see references to OVERFLOW_V3 that comes after OVERFLOW, but there\nis no OVERFLOW_V2. Intended, or should it be V2 to match the \"file\nformat v2\" of \"generation number v2\"? It is getting awkward to have\ntwo different version scheme (\"gen v2\" means corrected committer\ntimestamp, whose on-disk representation is different between \"file\nv1\" and \"file v2\", and this OVERFLOW vs OVERFLOW_V3 is about the\ndifference between \"file v1\" and \"file v2\" if I am following the\nseries correctly).\n"},{"id":"449530","messageId":"xmqqwnhjj24x.fsf@gitster.g","threadId":"57472","inReplyTo":"xmqqsfs8j60g.fsf@gitster.g","subject":"Re: [PATCH 0/7] Commit-graph: Generation Number v2 Fixes, v3 implementation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-24T23:06:22Z","receivedAt":"2022-02-24T23:06:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> I'm submitting these two things together so we can see them all at once, but\n>> I'd be happy to split this into two series. The first four patches are\n>> important bug fixes, so we can consider them as higher-priority.\n>>\n>> Thanks, -Stolee\n>\n> Thanks, will take a look.\n\nOverall it was a pleasant read, even though my reading hiccupped in\na few places. It does look like two separate topics, one of which\nbuilds on the other.\n"},{"id":"449579","messageId":"756fd005-637d-a067-9949-e87fc15b3bf6@github.com","threadId":"57472","inReplyTo":"xmqqh78nkj2x.fsf@gitster.g","subject":"Re: [PATCH 2/7] commit-graph: fix ordering bug in generation numbers","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-25T13:51:11Z","receivedAt":"2022-02-25T13:51:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/24/2022 5:15 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> From: Derrick Stolee \n>>\n>> When computing the generation numbers for a commit-graph, we compute\n>> the corrected commit dates and then check if their offsets from the\n>> actual dates is too large to fit in the 32-bit Generation Data chunk.\n>> However, there is a problem with this approach: if we have parsed the\n>> generation data from the previous commit-graph, then we continue the\n>> loop because the corrected commit date is already computed.\n>>\n>> It is incorrect to add an increment to num_generation_data_overflows\n>> here, because we might start double-counting commits that are computed\n>> because of the depth-first search walk from a commit with an earlier\n>> OID.\n>>\n>> Instead, iterate over the full commit list at the end, checking the\n>> offsets to see how many grow beyond the maximum value.\n> \n> Hmph, I can see how the new code correctly counts the commits that\n> require offsets that are too large, but I am not sure why the fix is\n> needed. The overall loop structure is\n\nIt is very subtle, which is why it took me a while to debug this\nissue once I managed to trigger it.\n\n> for each commit ctx->commits.list[i]:\n> continue if generation number has been computed for it already\n\nThis is the critical line in the current version. This includes\n\"continue if the generation number was loaded from the previous\ncommit-graph file.\" This means we under-count when building from\nan existing commit-graph with overflows.\n\nIf we insert an increment here, then we risk double-counting. I\nshould have described this better.\n\n> \tset up a commit-list for depth first search\n> \twhile (we are still digging) {\n> \t\tfor each parent {\n> \t\t\tif generation for the parent is not known yet:\n> \t\t\t\tpush it down and redo\n> \t\t\telse\n> \t\t\t\tcompute max of parents' generation number\n> \t\t}\n> if (all parents' generation number is known) {\n> \t\t\tset the generation number for ourselves\n> \t\t\tcount if we needed an offset that is too big\n> \t\t}\n> \t}\n> }\n> \n> The only case where we may double-count near the end of inner loop I\n> can think of is when we end up computing generation for the same\n> commit in the while () loop. But isn't that \"we dig the same thing\n> twice\" by itself something we want to fix, regardless of the\n> double-counting issue?\n\nBy \"we dig the same thing twice\" I think you mean \"we look across\nevery edge in the commit-graph, and some commits have multiple\ndirect children.\" There is no way around this, but we do skip\nrecalculating generation numbers for parents that are already\ncomputed.\n\n> IOW,\n> \n>> \t\t\t\tif (current->date && current->date > max_corrected_commit_date)\n>> \t\t\t\t\tmax_corrected_commit_date = current->date - 1;\n>> \t\t\t\tcommit_graph_data_at(current)->generation = max_corrected_commit_date + 1;\n>> -\n>> -\t\t\t\tif (commit_graph_data_at(current)->generation - current->date > GENERATION_NUMBER_V2_OFFSET_MAX)\n>> -\t\t\t\t\tctx->num_generation_data_overflows++;\n>> \t\t\t}\n>> \t\t}\n>> \t}\n> \n> here, before doing the assignment for the \"current\" commit's\n> generation number, if we added\n> \n> \t\tif (commit_graph_data_at(current)->generation !=\n> \t\t GENERATION_NUMBER_ZERO)\n> \t\t\tBUG(\"why are we digging it twice?\");\n> \n> would it trigger? If so, isn't that already a bug worth fixing?\n\nThis would not trigger, since 'current' did not have its\ngeneration when adding to the stack and it could not possibly\nhave been added a second time when doing a depth-first search\nfrom that commit.\n\n> Perhaps avoiding the second round, perhaps like this, may be a\n> better fix?\n> \n> \twhile (list) {\n> \t\tstruct commit *current = list->item;\n> \t\tstruct commit_list *parent;\n> \t\tint all_parents_computed = 1;\n> \t\ttimestamp_t max_corrected_commit_date = 0;\n> \n> +\t\tif (commit_graph_data_at(current)->generation !=\n> +\t\t GENERATION_NUMBER_ZERO) {\n> +\t\t\tpop_commit(&list);\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> \t\tfor (parent = current->parents; parent; parent = parent->next) {\n> \n> Or am I grossly misunderstanding why the original code is incorrect\n> to have the counting at this place?\n\nHopefully I cleared up the issue earlier in my reply. Let me\nknow if this is still confusing.\n\nThanks,\n-Stolee\n"},{"id":"449580","messageId":"3d303ba3-36ed-4bda-e499-8b7da0dc1413@github.com","threadId":"57472","inReplyTo":"xmqqbkyvki54.fsf@gitster.g","subject":"Re: [PATCH 4/7] commit-graph: fix generation number v2 overflow values","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-25T13:53:08Z","receivedAt":"2022-02-25T13:53:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/24/2022 5:35 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> -\t\t\tgraph_data->generation = get_be64(g->chunk_generation_data_overflow + 8 * offset_pos);\n>> +\t\t\tgraph_data->generation = item->date + get_be64(g->chunk_generation_data_overflow + 8 * offset_pos);\n> \n> Wow, that's embarrassing.\n> \n>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n>> index 1afee1c2705..5e4b0216fa6 100755\n>> --- a/t/t5318-commit-graph.sh\n>> +++ b/t/t5318-commit-graph.sh\n>> @@ -815,6 +815,19 @@ test_expect_success 'corrupt commit-graph write (missing tree)' '\n>> \t)\n>> '\n>> \n>> +# The remaining tests check timestamps that flow over\n>> +# 32-bits. The graph_git_behavior checks can't take a\n>> +# prereq, so just stop here if we are on a 32-bit machine.\n>> +\n>> +if ! test_have_prereq TIME_IS_64BIT\n>> +then\n>> +\ttest_done\n>> +fi\n>> +if ! test_have_prereq TIME_T_IS_64BIT\n>> +then\n>> +\ttest_done\n>> +fi\n> \n> The above is OK but is there a reason why we cannot do\n> \n> \tif A || B\n> \tthen\n> \t\ttest_done\n> \tfi\n> \n> here (I am assuming not, but in case I am missing the reason why it\n> has to be separate)?\n\nDoes not need to be separate. I just discovered the two different\nprereqs for similar, but not exact, checks. I can swap this to an\nor statement.\n \n>> @@ -832,10 +845,10 @@ test_expect_success 'corrupt commit-graph write (missing tree)' '\n>> # The largest offset observed is 2 ^ 31, just large enough to overflow.\n>> #\n>> \n>> -test_expect_success 'set up and verify repo with generation data overflow chunk' '\n>> +test_expect_success TIME_IS_64BIT,TIME_T_IS_64BIT 'set up and verify repo with generation data overflow chunk' '\n>> \tobjdir=\".git/objects\" &&\n>> \tUNIX_EPOCH_ZERO=\"@0 +0000\" &&\n>> -\tFUTURE_DATE=\"@2147483646 +0000\" &&\n>> +\tFUTURE_DATE=\"@4000000000 +0000\" &&\n> \n> OK. 16#EE6B2800 too large to fit and will cause wrapping around with\n> signed 32-bit integer.\n\nRight. I wanted it to be right on that boundary of needing the 32nd\nbit but not being over that on its own. I did check that without\nthe prereqs this code fails on 32-bit systems due to parsing the\ntime.\n\nThanks,\n-Stolee\n"},{"id":"449581","messageId":"a78f0808-3a58-3bab-c958-ee623b2c43fc@github.com","threadId":"57472","inReplyTo":"xmqq1qzrkgxg.fsf@gitster.g","subject":"Re: [PATCH 6/7] commit-graph: parse file format v2","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-25T13:54:08Z","receivedAt":"2022-02-25T13:54:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/24/2022 6:01 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> From: Derrick Stolee \n>>\n>> The commit-graph file format v2 alters how it stores the corrected\n>> commit date offsets within the Commit Data chunk instead of a separate\n>> chunk. The idea is to significantly reduce the amount of data loaded\n>> from disk while parsing the commit-graph.\n>>\n>> We need to alter the error message when we see a file format version\n>> outside of our range now that multiple are possible. This has a\n>> non-functional side-effect of altering a use of GRAPH_VERSION within\n>> write_commit_graph().\n>>\n>> By storing the file format version in 'struct commit_graph', we can\n>> alter the parsing code to depend on that version value. This involves\n>> changing where we look for the corrected commit date offset, but also\n>> which constants we use for jumping into the Generation Data Overflow\n>> chunk. The Commit Data chunk only has 30 bits available for the offset\n>> while the Generation Data chunk has 32 bits. This only makes a\n>> meaningful difference in very malformed repositories.\n>>\n>> Also, we need to be careful about how we enable using corrected commit\n>> dates and generation numbers to rely upon the read_generation_data value\n>> instead of a non-zero value in the Commit Date chunk. In\n>> generation_numbers_enabled(), the first_generation variable is\n>> attemptint to look for the first topological level stored to see that it\n>> is nonzero. However, for a v2 commit-graph, this value is actually\n>> likely to be zero because the corrected commit date offset is probably\n>> zero.\n> \n> I see references to OVERFLOW_V3 that comes after OVERFLOW, but there\n> is no OVERFLOW_V2. Intended, or should it be V2 to match the \"file\n> format v2\" of \"generation number v2\"? It is getting awkward to have\n> two different version scheme (\"gen v2\" means corrected committer\n> timestamp, whose on-disk representation is different between \"file\n> v1\" and \"file v2\", and this OVERFLOW vs OVERFLOW_V3 is about the\n> difference between \"file v1\" and \"file v2\" if I am following the\n> series correctly).\n\nYou're right that it would be clearer to rename OVERFLOW to\nOVERFLOW_V2. I'll add that to my next version when these patches\nappear on their own.\n\nThanks,\n-Stolee\n"},{"id":"449582","messageId":"1322bf87-e61b-5fcd-53eb-c98602efdc07@github.com","threadId":"57472","inReplyTo":"xmqqwnhjj24x.fsf@gitster.g","subject":"Re: [PATCH 0/7] Commit-graph: Generation Number v2 Fixes, v3 implementation","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-25T13:55:01Z","receivedAt":"2022-02-25T13:55:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/24/2022 6:06 PM, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n>>> I'm submitting these two things together so we can see them all at once, but\n>>> I'd be happy to split this into two series. The first four patches are\n>>> important bug fixes, so we can consider them as higher-priority.\n>>>\n>>> Thanks, -Stolee\n>>\n>> Thanks, will take a look.\n> \n> Overall it was a pleasant read, even though my reading hiccupped in\n> a few places. It does look like two separate topics, one of which\n> builds on the other.\n\nThanks. I'll split them. The next version will drop the last three\npatches and I'll re-send them after the first four merge down.\n\nThanks,\n-Stolee\n"},{"id":"449612","messageId":"xmqqwnhietnz.fsf@gitster.g","threadId":"57472","inReplyTo":"756fd005-637d-a067-9949-e87fc15b3bf6@github.com","subject":"Re: [PATCH 2/7] commit-graph: fix ordering bug in generation numbers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-25T17:35:12Z","receivedAt":"2022-02-25T17:35:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> It is very subtle, which is why it took me a while to debug this\n> issue once I managed to trigger it.\n>\n>> for each commit ctx->commits.list[i]:\n>> continue if generation number has been computed for it already\n>\n> This is the critical line in the current version. This includes\n> \"continue if the generation number was loaded from the previous\n> commit-graph file.\" This means we under-count when building from\n> an existing commit-graph with overflows.\n>\n> If we insert an increment here, then we risk double-counting. I\n> should have described this better.\n\nAh, that obviously I missed. Thanks.\n\n"},{"id":"449613","messageId":"xmqqr17qeths.fsf@gitster.g","threadId":"57472","inReplyTo":"3d303ba3-36ed-4bda-e499-8b7da0dc1413@github.com","subject":"Re: [PATCH 4/7] commit-graph: fix generation number v2 overflow values","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-02-25T17:38:55Z","receivedAt":"2022-02-25T17:39:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>>> +# The remaining tests check timestamps that flow over\n>>> +# 32-bits. The graph_git_behavior checks can't take a\n>>> +# prereq, so just stop here if we are on a 32-bit machine.\n>>> +\n>>> +if ! test_have_prereq TIME_IS_64BIT\n>>> +then\n>>> +\ttest_done\n>>> +fi\n>>> +if ! test_have_prereq TIME_T_IS_64BIT\n>>> +then\n>>> +\ttest_done\n>>> +fi\n>> \n>> The above is OK but is there a reason why we cannot do\n>> \n>> \tif A || B\n>> \tthen\n>> \t\ttest_done\n>> \tfi\n>> \n>> here (I am assuming not, but in case I am missing the reason why it\n>> has to be separate)?\n>\n> Does not need to be separate. I just discovered the two different\n> prereqs for similar, but not exact, checks. I can swap this to an\n> or statement.\n\nI do not think a single condition with single test_done is\nnecessarily better. I was just curious if there was anything subtle\ngoing on.\n\nThanks.\n"},{"id":"449637","messageId":"220225.86a6ee7eid.gmgdl@evledraar.gmail.com","threadId":"57472","inReplyTo":"7f9b65bd22551fd7fd5d2f0bf18aee8c25f1db99.1645735117.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/7] commit-graph: document file format v2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-25T22:31:59Z","receivedAt":"2022-02-25T22:44:35Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Feb 24 2022, Derrick Stolee via GitGitGadget wrote:\n\n> The corrected commit date was first documented in 5a3b130ca (doc: add\n> corrected commit date info, 2021-01-16) and it used an optional chunk to\n> augment the commit-graph format without modifying the file format\n> version.\n>\n> One major benefit to this approach is that corrected commit dates could\n> be written without causing a backwards compatibility issue with Git\n> versions that do not understand them. The topological level was still\n> available in the CDAT chunk as it was before.\n>\n> However, this causes a different issue: more data needs to be loaded\n> from disk when parsing commits from the commit-graph. In cases where\n> there is no significant algorithmic gain from using corrected commit\n> dates, commit walks take up to 20% longer because of this extra data.\n>\n> Create a new file format version for the commit-graph format that\n> differs only in the CDAT chunk: it now stores corrected commit date\n> offsets. This brings our data back to normal and will demonstrate\n> performance gains in almost all cases.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> .../technical/commit-graph-format.txt | 22 ++++++++++++++-----\n> 1 file changed, 17 insertions(+), 5 deletions(-)\n>\n> diff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\n> index 87971c27dd7..2cb48993314 100644\n> --- a/Documentation/technical/commit-graph-format.txt\n> +++ b/Documentation/technical/commit-graph-format.txt\n> @@ -36,7 +36,7 @@ HEADER:\n> The signature is: {'C', 'G', 'P', 'H'}\n> \n> 1-byte version number:\n> - Currently, the only valid version is 1.\n> + This version number can be 1 or 2.\n> \n> 1-byte Hash Version\n> We infer the hash length (H) from this value:\n> @@ -85,13 +85,22 @@ CHUNK DATA:\n> position. If there are more than two parents, the second value\n> has its most-significant bit on and the other bits store an array\n> position into the Extra Edge List chunk.\n> - * The next 8 bytes store the topological level (generation number v1)\n> - of the commit and\n> - the commit time in seconds since EPOCH. The generation number\n> - uses the higher 30 bits of the first 4 bytes, while the commit\n> + * The next 8 bytes store the generation number information of the\n> + commit and the commit time in seconds since EPOCH. The generation\n> + number uses the higher 30 bits of the first 4 bytes, while the commit\n> time uses the 32 bits of the second 4 bytes, along with the lowest\n> 2 bits of the lowest byte, storing the 33rd and 34th bit of the\n> commit time.\n> + - If the commit-graph file format is version 1, then the higher 30\n> +\tbits contain the topological level (generation number v1) for the\n> +\tcommit.\n> + - If the commit-graph file format is version 2, then the higher 30\n> +\tbits contain the corrected commit date offset (generation number\n> +\tv2) for the commit, except if the offset cannot be stored within\n> +\t29 bits. If the offset is too large for 29 bits, then the value\n> +\tstored here has its most-significant bit on and the other bits\n> +\tstore the position of the corrected commit date in the Generation\n> +\tDate Overflow chunk.\n> \n> Generation Data (ID: {'G', 'D', 'A', 'T' }) (N * 4 bytes) [Optional]\n> * This list of 4-byte values store corrected commit date offsets for the\n> @@ -103,6 +112,9 @@ CHUNK DATA:\n> * Generation Data chunk is present only when commit-graph file is written\n> by compatible versions of Git and in case of split commit-graph chains,\n> the topmost layer also has Generation Data chunk.\n> + * This chunk does not exist if the commit-graph file format version is 2,\n> + because the corrected commit date offset data is stored in the Commit\n> + Data chunk.\n> \n> Generation Data Overflow (ID: {'G', 'D', 'O', 'V' }) [Optional]\n> * This list of 8-byte values stores the corrected commit date offsets\n\nWe talked a while ago now about how we do commit-graph format changes\nand this is partially echoing those earlier questions[1] from 2019.\n\nI fully understand why we're writing this amended CDAT chunk in a\ndifferent layout. By not having the GDAT side-chunk to look up in the\ndata is more local, that part of the file is more compact etc.\n\nWhat I don't understand is why getting those performance improvements\nrequires the breaking version change & the writing of the incompatible\nversion number.\n\nI.e. couldn't the differently formatted CDAT chunk be written instead to a new\nchunk name (say \"2DAT\") instead? Per [1] we'd pay a small fixed cost for\na possibly empty chunk (I didn't re-do those numbers), but surely the\nperformance improvements will be about the same for that miniscule\noverhead.\n\nIt will give you something you can't have here, which is optional\ncompatibility with older clients by writing both versions. That'll be a\n~2x as large file on disk, but with the page cache & each client version\nskipping to the data it needs caching characteristics & data locality\nshould work out to about the same thing.\n\nOr maybe they won't. I just found it surprising when reviewing this to\nnot find an answer to why that approach wasn't\nconsidered.\n\nE.g. 76ffbca71a9 (commit-graph: write Bloom filters to commit graph\nfile, 2020-04-06) is a commit adding such new optional and\nbackwards-compatible data.\n\n1. https://lore.kernel.org/git/87h8acivkh.fsf@evledraar.gmail.com/\n"},{"id":"449753","messageId":"d19f5ee8-af92-805f-7ea2-8285862c1123@github.com","threadId":"57472","inReplyTo":"220225.86a6ee7eid.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 5/7] commit-graph: document file format v2","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-28T13:44:39Z","receivedAt":"2022-02-28T13:44:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/25/2022 5:31 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Feb 24 2022, Derrick Stolee via GitGitGadget wrote:\n> \n...\n>> Generation Data (ID: {'G', 'D', 'A', 'T' }) (N * 4 bytes) [Optional]\n>> * This list of 4-byte values store corrected commit date offsets for the\n>> @@ -103,6 +112,9 @@ CHUNK DATA:\n>> * Generation Data chunk is present only when commit-graph file is written\n>> by compatible versions of Git and in case of split commit-graph chains,\n>> the topmost layer also has Generation Data chunk.\n>> + * This chunk does not exist if the commit-graph file format version is 2,\n>> + because the corrected commit date offset data is stored in the Commit\n>> + Data chunk.\n>> \n>> Generation Data Overflow (ID: {'G', 'D', 'O', 'V' }) [Optional]\n>> * This list of 8-byte values stores the corrected commit date offsets\n> \n> We talked a while ago now about how we do commit-graph format changes\n> and this is partially echoing those earlier questions[1] from 2019.\n> \n> I fully understand why we're writing this amended CDAT chunk in a\n> different layout. By not having the GDAT side-chunk to look up in the\n> data is more local, that part of the file is more compact etc.\n> \n> What I don't understand is why getting those performance improvements\n> requires the breaking version change & the writing of the incompatible\n> version number.\n> \n> I.e. couldn't the differently formatted CDAT chunk be written instead to a new\n> chunk name (say \"2DAT\") instead? Per [1] we'd pay a small fixed cost for\n> a possibly empty chunk (I didn't re-do those numbers), but surely the\n> performance improvements will be about the same for that miniscule\n> overhead.\n\nCDAT is a required chunk. It is part of the v1 spec that CDAT exists\nand is correct. All other Git clients will error out when reading a\n\"v1\" graph without such a chunk, and in a way that is less helpful to\nusers. Instead of clearly indicating \"file version is too new\" it will\nsay \"commit-graph is missing the Commit Data chunk\" which is not\nhelpful.\n\n> It will give you something you can't have here, which is optional\n> compatibility with older clients by writing both versions. That'll be a\n> ~2x as large file on disk, but with the page cache & each client version\n> skipping to the data it needs caching characteristics & data locality\n> should work out to about the same thing.\n\nWriting both is the only way that this could work without incrementing\nthe graph version number, but I'd rather just update the number and\navoid wasting the effort to write that extra data.\n\nIt seems you are hyper-focused on \"we don't _need_ to update the version\nnumber\" and you are willing to recommend wasteful approaches in order to\nsupport that stance.\n\nSo: you're right. We don't _need_ to update the version number. But this\nis the best choice among the options available.\n\n> Or maybe they won't. I just found it surprising when reviewing this to\n> not find an answer to why that approach wasn't\n> considered.\n\nThe point is to create a new format that can be chosen when deployed\nin an environment where older Git versions will not exist (such as\na Git server). The new version is not chosen by default and instead\nis opt-in through the commitGraph.generationVersion config option.\n\nPerhaps in a year or two we would consider making this the new\ndefault, but there is no rush to do so.\n\nThanks,\n-Stolee\n"},{"id":"449755","messageId":"pull.1163.v2.git.1646056423.gitgitgadget@gmail.com","threadId":"57472","inReplyTo":"pull.1163.git.1645735117.gitgitgadget@gmail.com","subject":"[PATCH v2 0/4] Commit-graph: Generation Number v2 Fixes","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-28T13:53:39Z","receivedAt":"2022-02-28T13:53:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This patch series fixes some bugs in generation number v2. They were\ndiscovered while building generation number v3, but that implementation will\nbe delayed until these fixes are merged.\n\nIn particular, Git has been ignoring corrected commit dates since shortly\nafter they were introduced. This is due to a bug I introduced when trying to\nmake split commit-graphs safer with mixed generation number versions. I also\nnoticed an issue with the offset overflows that I only noticed after writing\ngeneration number v3 using a smaller offset size, actually triggering the\nbug in the test suite.\n\n\nUpdates in v2\n=============\n\n * Dropped generation v3 patches, saving them for later.\n * Updated a commit message to more clearly describe the problem with the\n old code.\n * Used an || instead of two if statements in test script.\n\nThanks, -Stolee\n\nDerrick Stolee (4):\n test-read-graph: include extra post-parse info\n commit-graph: fix ordering bug in generation numbers\n commit-graph: start parsing generation v2 (again)\n commit-graph: fix generation number v2 overflow values\n\n commit-graph.c | 15 +++++++++++----\n t/helper/test-read-graph.c | 13 +++++++++++++\n t/t4216-log-bloom.sh | 1 +\n t/t5318-commit-graph.sh | 34 +++++++++++++++++++++++++++++-----\n t/t5324-split-commit-graph.sh | 10 ++++++++++\n 5 files changed, 64 insertions(+), 9 deletions(-)\n\n\nbase-commit: dab1b7905d0b295f1acef9785bb2b9cbb0fdec84\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1163%2Fderrickstolee%2Fgen-v3-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1163/derrickstolee/gen-v3-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1163\n\nRange-diff vs v1:\n\n 1: 2f89275314b = 1: 2f89275314b test-read-graph: include extra post-parse info\n 2: 6e47ffed257 ! 2: cbcbf10e699 commit-graph: fix ordering bug in generation numbers\n @@ Commit message\n actual dates is too large to fit in the 32-bit Generation Data chunk.\n However, there is a problem with this approach: if we have parsed the\n generation data from the previous commit-graph, then we continue the\n - loop because the corrected commit date is already computed.\n + loop because the corrected commit date is already computed. This causes\n + an under-count in the number of overflow values.\n \n It is incorrect to add an increment to num_generation_data_overflows\n - here, because we might start double-counting commits that are computed\n - because of the depth-first search walk from a commit with an earlier\n - OID.\n + next to this 'continue' statement, because we might start\n + double-counting commits that are computed because of the depth-first\n + search walk from a commit with an earlier OID.\n \n Instead, iterate over the full commit list at the end, checking the\n offsets to see how many grow beyond the maximum value.\n 3: a3436b92a32 = 3: 5bc6a7660d8 commit-graph: start parsing generation v2 (again)\n 4: de7ab2f39d9 ! 4: 193217c71e0 commit-graph: fix generation number v2 overflow values\n @@ t/t5318-commit-graph.sh: test_expect_success 'corrupt commit-graph write (missin\n +# 32-bits. The graph_git_behavior checks can't take a\n +# prereq, so just stop here if we are on a 32-bit machine.\n +\n -+if ! test_have_prereq TIME_IS_64BIT\n -+then\n -+\ttest_done\n -+fi\n -+if ! test_have_prereq TIME_T_IS_64BIT\n ++if ! test_have_prereq TIME_IS_64BIT || ! test_have_prereq TIME_T_IS_64BIT\n +then\n +\ttest_done\n +fi\n 5: 7f9b65bd225 < -: ----------- commit-graph: document file format v2\n 6: 28fe8824ba7 < -: ----------- commit-graph: parse file format v2\n 7: ade697c4d34 < -: ----------- commit-graph: write file format v2\n\n-- \ngitgitgadget\n"},{"id":"449756","messageId":"2f89275314b4a2a89a18d14e41602bbe2e1988dc.1646056423.git.gitgitgadget@gmail.com","threadId":"57472","inReplyTo":"pull.1163.v2.git.1646056423.gitgitgadget@gmail.com","subject":"[PATCH v2 1/4] test-read-graph: include extra post-parse info","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-28T13:53:40Z","receivedAt":"2022-02-28T13:53:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIt can be helpful to verify that the 'struct commit_graph' that results\nfrom parsing a commit-graph is correctly structured. The existence of\ndifferent chunks is not enough to verify that all of the optional\nfeatures are correctly enabled.\n\nUpdate 'test-tool read-graph' to output an \"options:\" line that includes\ninformation for different parts of the struct commit_graph.\n\nIn particular, this change demonstrates that the read_generation_data\noption is never being enabled, which will be fixed in a later change.\n\nSigned-off-by: Derrick Stolee \n---\n t/helper/test-read-graph.c | 13 +++++++++++++\n t/t4216-log-bloom.sh | 1 +\n t/t5318-commit-graph.sh | 1 +\n t/t5324-split-commit-graph.sh | 5 +++++\n 4 files changed, 20 insertions(+)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex 75927b2c81d..c3b6b8d1734 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -3,6 +3,7 @@\n #include \"commit-graph.h\"\n #include \"repository.h\"\n #include \"object-store.h\"\n+#include \"bloom.h\"\n \n int cmd__read_graph(int argc, const char **argv)\n {\n@@ -45,6 +46,18 @@ int cmd__read_graph(int argc, const char **argv)\n \t\tprintf(\" bloom_data\");\n \tprintf(\"\\n\");\n \n+\tprintf(\"options:\");\n+\tif (graph->bloom_filter_settings)\n+\t\tprintf(\" bloom(%d,%d,%d)\",\n+\t\t graph->bloom_filter_settings->hash_version,\n+\t\t graph->bloom_filter_settings->bits_per_entry,\n+\t\t graph->bloom_filter_settings->num_hashes);\n+\tif (graph->read_generation_data)\n+\t\tprintf(\" read_generation_data\");\n+\tif (graph->topo_levels)\n+\t\tprintf(\" topo_levels\");\n+\tprintf(\"\\n\");\n+\n \tUNLEAK(graph);\n \n \treturn 0;\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex cc3cebf6722..5ed6d2a21c1 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -48,6 +48,7 @@ graph_read_expect () {\n \theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata generation_data bloom_indexes bloom_data\n+\toptions: bloom(1,10,7)\n \tEOF\n \ttest-tool read-graph >actual &&\n \ttest_cmp expect actual\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex edb728f77c3..2b05026cf6d 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -104,6 +104,7 @@ graph_read_expect() {\n \theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata$OPTIONAL\n+\toptions:\n \tEOF\n \ttest-tool read-graph >output &&\n \ttest_cmp expect output\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 847b8097109..778fa418de2 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -34,6 +34,7 @@ graph_read_expect() {\n \theader: 43475048 1 $(test_oid oid_version) 4 $NUM_BASE\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata generation_data\n+\toptions:\n \tEOF\n \ttest-tool read-graph >output &&\n \ttest_cmp expect output\n@@ -508,6 +509,7 @@ test_expect_success 'setup repo for mixed generation commit-graph-chain' '\n \t\theader: 43475048 1 $(test_oid oid_version) 4 1\n \t\tnum_commits: $NUM_SECOND_LAYER_COMMITS\n \t\tchunks: oid_fanout oid_lookup commit_metadata\n+\t\toptions:\n \t\tEOF\n \t\ttest_cmp expect output &&\n \t\tgit commit-graph verify &&\n@@ -540,6 +542,7 @@ test_expect_success 'do not write generation data chunk if not present on existi\n \t\theader: 43475048 1 $(test_oid oid_version) 4 2\n \t\tnum_commits: $NUM_THIRD_LAYER_COMMITS\n \t\tchunks: oid_fanout oid_lookup commit_metadata\n+\t\toptions:\n \t\tEOF\n \t\ttest_cmp expect output &&\n \t\tgit commit-graph verify\n@@ -581,6 +584,7 @@ test_expect_success 'do not write generation data chunk if the topmost remaining\n \t\theader: 43475048 1 $(test_oid oid_version) 4 2\n \t\tnum_commits: $(($NUM_THIRD_LAYER_COMMITS + $NUM_FOURTH_LAYER_COMMITS))\n \t\tchunks: oid_fanout oid_lookup commit_metadata\n+\t\toptions:\n \t\tEOF\n \t\ttest_cmp expect output &&\n \t\tgit commit-graph verify\n@@ -620,6 +624,7 @@ test_expect_success 'write generation data chunk if topmost remaining layer has\n \t\theader: 43475048 1 $(test_oid oid_version) 5 1\n \t\tnum_commits: $(($NUM_SECOND_LAYER_COMMITS + $NUM_THIRD_LAYER_COMMITS + $NUM_FOURTH_LAYER_COMMITS + $NUM_FIFTH_LAYER_COMMITS))\n \t\tchunks: oid_fanout oid_lookup commit_metadata generation_data\n+\t\toptions:\n \t\tEOF\n \t\ttest_cmp expect output\n \t)\n-- \ngitgitgadget\n\n"},{"id":"449757","messageId":"cbcbf10e6994d469abc5463d08165e6a2ba73153.1646056423.git.gitgitgadget@gmail.com","threadId":"57472","inReplyTo":"pull.1163.v2.git.1646056423.gitgitgadget@gmail.com","subject":"[PATCH v2 2/4] commit-graph: fix ordering bug in generation numbers","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-28T13:53:41Z","receivedAt":"2022-02-28T13:53:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen computing the generation numbers for a commit-graph, we compute\nthe corrected commit dates and then check if their offsets from the\nactual dates is too large to fit in the 32-bit Generation Data chunk.\nHowever, there is a problem with this approach: if we have parsed the\ngeneration data from the previous commit-graph, then we continue the\nloop because the corrected commit date is already computed. This causes\nan under-count in the number of overflow values.\n\nIt is incorrect to add an increment to num_generation_data_overflows\nnext to this 'continue' statement, because we might start\ndouble-counting commits that are computed because of the depth-first\nsearch walk from a commit with an earlier OID.\n\nInstead, iterate over the full commit list at the end, checking the\noffsets to see how many grow beyond the maximum value.\n\nUpdate a test in t5318 to use a larger time value, which will help\ndemonstrate this bug in more cases. It still won't hit all potential\ncases until the next change, which reenables reading generation numbers.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 10 +++++++---\n t/t5318-commit-graph.sh | 4 ++--\n 2 files changed, 9 insertions(+), 5 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 265c010122e..a19bd96c2ee 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1556,12 +1556,16 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \t\t\t\tif (current->date && current->date > max_corrected_commit_date)\n \t\t\t\t\tmax_corrected_commit_date = current->date - 1;\n \t\t\t\tcommit_graph_data_at(current)->generation = max_corrected_commit_date + 1;\n-\n-\t\t\t\tif (commit_graph_data_at(current)->generation - current->date > GENERATION_NUMBER_V2_OFFSET_MAX)\n-\t\t\t\t\tctx->num_generation_data_overflows++;\n \t\t\t}\n \t\t}\n \t}\n+\n+\tfor (i = 0; i < ctx->commits.nr; i++) {\n+\t\tstruct commit *c = ctx->commits.list[i];\n+\t\ttimestamp_t offset = commit_graph_data_at(c)->generation - c->date;\n+\t\tif (offset > GENERATION_NUMBER_V2_OFFSET_MAX)\n+\t\t\tctx->num_generation_data_overflows++;\n+\t}\n \tstop_progress(&ctx->progress);\n }\n \ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 2b05026cf6d..f9bffe38013 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -467,10 +467,10 @@ test_expect_success 'warn on improper hash version' '\n \t)\n '\n \n-test_expect_success 'lower layers have overflow chunk' '\n+test_expect_success TIME_IS_64BIT,TIME_T_IS_64BIT 'lower layers have overflow chunk' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tUNIX_EPOCH_ZERO=\"@0 +0000\" &&\n-\tFUTURE_DATE=\"@2147483646 +0000\" &&\n+\tFUTURE_DATE=\"@4147483646 +0000\" &&\n \trm -f .git/objects/info/commit-graph &&\n \ttest_commit --date \"$FUTURE_DATE\" future-1 &&\n \ttest_commit --date \"$UNIX_EPOCH_ZERO\" old-1 &&\n-- \ngitgitgadget\n\n"},{"id":"449758","messageId":"5bc6a7660d897ca6c52eabba8fb9ecfb6304dabb.1646056423.git.gitgitgadget@gmail.com","threadId":"57472","inReplyTo":"pull.1163.v2.git.1646056423.gitgitgadget@gmail.com","subject":"[PATCH v2 3/4] commit-graph: start parsing generation v2 (again)","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-28T13:53:42Z","receivedAt":"2022-02-28T13:53:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'read_generation_data' member of 'struct commit_graph' was\nintroduced by 1fdc383c5 (commit-graph: use generation v2 only if entire\nchain does, 2021-01-16). The intention was to avoid using corrected\ncommit dates if not all layers of a commit-graph had that data stored.\nThe logic in validate_mixed_generation_chain() at that point incorrectly\ninitialized read_generation_data to 1 if and only if the tip\ncommit-graph contained the Corrected Commit Date chunk.\n\nThis was \"fixed\" in 448a39e65 (commit-graph: validate layers for\ngeneration data, 2021-02-02) to validate that read_generation_data was\neither non-zero for all layers, or it would set read_generation_data to\nzero for all layers.\n\nThe problem here is that read_generation_data is not initialized to be\nnon-zero anywhere!\n\nThis change initializes read_generation_data immediately after the chunk\nis parsed, so each layer will have its value present as soon as\npossible.\n\nThe read_generation_data member is used in fill_commit_graph_info() to\ndetermine if we should use the corrected commit date or the topological\nlevels stored in the Commit Data chunk. Due to this bug, all previous\nversions of Git were defaulting to topological levels in all cases!\n\nThis can be measured with some performance tests. Using the Linux kernel\nas a testbed, I generated a complete commit-graph containing corrected\ncommit dates and tested the 'new' version against the previous, 'old'\nversion.\n\nFirst, rev-list with --topo-order demonstrates a 26% improvement using\ncorrected commit dates:\n\nhyperfine \\\n\t-n \"old\" \"$OLD_GIT rev-list --topo-order -1000 v3.6\" \\\n\t-n \"new\" \"$NEW_GIT rev-list --topo-order -1000 v3.6\" \\\n\t--warmup=10\n\nBenchmark 1: old\n Time (mean ± σ): 57.1 ms ± 3.1 ms\n Range (min … max): 52.9 ms … 62.0 ms 55 runs\n\nBenchmark 2: new\n Time (mean ± σ): 45.5 ms ± 3.3 ms\n Range (min … max): 39.9 ms … 51.7 ms 59 runs\n\nSummary\n 'new' ran\n 1.26 ± 0.11 times faster than 'old'\n\nThese performance improvements are due to the algorithmic improvements\ngiven by walking fewer commits due to the higher cutoffs from corrected\ncommit dates.\n\nHowever, this comes at a cost. The additional I/O cost of parsing the\ncorrected commit dates is visible in case of merge-base commands that do\nnot reduce the overall number of walked commits.\n\nhyperfine \\\n -n \"old\" \"$OLD_GIT merge-base v4.8 v4.9\" \\\n -n \"new\" \"$NEW_GIT merge-base v4.8 v4.9\" \\\n --warmup=10\n\nBenchmark 1: old\n Time (mean ± σ): 110.4 ms ± 6.4 ms\n Range (min … max): 96.0 ms … 118.3 ms 25 runs\n\nBenchmark 2: new\n Time (mean ± σ): 150.7 ms ± 1.1 ms\n Range (min … max): 149.3 ms … 153.4 ms 19 runs\n\nSummary\n 'old' ran\n 1.36 ± 0.08 times faster than 'new'\n\nPerformance issues like this are what motivated 702110aac (commit-graph:\nuse config to specify generation type, 2021-02-25).\n\nIn the future, we could fix this performance problem by inserting the\ncorrected commit date offsets into the Commit Date chunk instead of\nhaving that data in an extra chunk.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 3 +++\n t/t4216-log-bloom.sh | 2 +-\n t/t5318-commit-graph.sh | 14 ++++++++++++--\n t/t5324-split-commit-graph.sh | 9 +++++++--\n 4 files changed, 23 insertions(+), 5 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a19bd96c2ee..8e52bb09552 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -407,6 +407,9 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t\t&graph->chunk_generation_data);\n \t\tpair_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW,\n \t\t\t&graph->chunk_generation_data_overflow);\n+\n+\t\tif (graph->chunk_generation_data)\n+\t\t\tgraph->read_generation_data = 1;\n \t}\n \n \tif (r->settings.commit_graph_read_changed_paths) {\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 5ed6d2a21c1..fa9d32facfb 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -48,7 +48,7 @@ graph_read_expect () {\n \theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata generation_data bloom_indexes bloom_data\n-\toptions: bloom(1,10,7)\n+\toptions: bloom(1,10,7) read_generation_data\n \tEOF\n \ttest-tool read-graph >actual &&\n \ttest_cmp expect actual\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex f9bffe38013..1afee1c2705 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -100,11 +100,21 @@ graph_read_expect() {\n \t\tOPTIONAL=\" $2\"\n \t\tNUM_CHUNKS=$((3 + $(echo \"$2\" | wc -w)))\n \tfi\n+\tGENERATION_VERSION=2\n+\tif test ! -z \"$3\"\n+\tthen\n+\t\tGENERATION_VERSION=$3\n+\tfi\n+\tOPTIONS=\n+\tif test $GENERATION_VERSION -gt 1\n+\tthen\n+\t\tOPTIONS=\" read_generation_data\"\n+\tfi\n \tcat >expect <<- EOF\n \theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata$OPTIONAL\n-\toptions:\n+\toptions:$OPTIONS\n \tEOF\n \ttest-tool read-graph >output &&\n \ttest_cmp expect output\n@@ -498,7 +508,7 @@ test_expect_success 'git commit-graph verify' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit rev-parse commits/8 | git -c commitGraph.generationVersion=1 commit-graph write --stdin-commits &&\n \tgit commit-graph verify >output &&\n-\tgraph_read_expect 9 extra_edges\n+\tgraph_read_expect 9 extra_edges 1\n '\n \n NUM_COMMITS=9\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 778fa418de2..669ddc645fa 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -30,11 +30,16 @@ graph_read_expect() {\n \tthen\n \t\tNUM_BASE=$2\n \tfi\n+\tOPTIONS=\n+\tif test -z \"$3\"\n+\tthen\n+\t\tOPTIONS=\" read_generation_data\"\n+\tfi\n \tcat >expect <<- EOF\n \theader: 43475048 1 $(test_oid oid_version) 4 $NUM_BASE\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata generation_data\n-\toptions:\n+\toptions:$OPTIONS\n \tEOF\n \ttest-tool read-graph >output &&\n \ttest_cmp expect output\n@@ -624,7 +629,7 @@ test_expect_success 'write generation data chunk if topmost remaining layer has\n \t\theader: 43475048 1 $(test_oid oid_version) 5 1\n \t\tnum_commits: $(($NUM_SECOND_LAYER_COMMITS + $NUM_THIRD_LAYER_COMMITS + $NUM_FOURTH_LAYER_COMMITS + $NUM_FIFTH_LAYER_COMMITS))\n \t\tchunks: oid_fanout oid_lookup commit_metadata generation_data\n-\t\toptions:\n+\t\toptions: read_generation_data\n \t\tEOF\n \t\ttest_cmp expect output\n \t)\n-- \ngitgitgadget\n\n"},{"id":"449759","messageId":"193217c71e0aaf3f56a02d9abec6753bd19aba71.1646056423.git.gitgitgadget@gmail.com","threadId":"57472","inReplyTo":"pull.1163.v2.git.1646056423.gitgitgadget@gmail.com","subject":"[PATCH v2 4/4] commit-graph: fix generation number v2 overflow values","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-02-28T13:53:43Z","receivedAt":"2022-02-28T13:54:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe Generation Data Chunk was implemented and tested in e8b63005c\n(commit-graph: implement generation data chunk, 2021-01-16), but the\ntest was carefully constructed to work on systems with 32-bit dates.\nSince the corrected commit date offsets still required more than 31\nbits, this triggered writing the generation_data_overflow chunk.\n\nHowever, upon closer look, the\nwrite_graph_chunk_generation_data_overflow() method writes the offsets\nto the chunk (as dictated by the format) but fill_commit_graph_info()\ntreats the value in the chunk as if it is the full corrected commit date\n(not an offset). For some reason, this does not cause an issue when\nusing the FUTURE_DATE specified in t5318-commit-graph.sh, but it does\nshow up as a failure in 'git commit-graph verify' if we increase that\nFUTURE_DATE to be above four billion.\n\nFix this error and update the test to require 64-bit dates so we can\nsafely use this large value in our test.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 2 +-\n t/t5318-commit-graph.sh | 17 +++++++++++++++--\n 2 files changed, 16 insertions(+), 3 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8e52bb09552..b86a6a634fe 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -806,7 +806,7 @@ static void fill_commit_graph_info(struct commit *item, struct commit_graph *g,\n \t\t\t\tdie(_(\"commit-graph requires overflow generation data but has none\"));\n \n \t\t\toffset_pos = offset ^ CORRECTED_COMMIT_DATE_OFFSET_OVERFLOW;\n-\t\t\tgraph_data->generation = get_be64(g->chunk_generation_data_overflow + 8 * offset_pos);\n+\t\t\tgraph_data->generation = item->date + get_be64(g->chunk_generation_data_overflow + 8 * offset_pos);\n \t\t} else\n \t\t\tgraph_data->generation = item->date + offset;\n \t} else\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 1afee1c2705..f4ffaad661d 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -815,6 +815,15 @@ test_expect_success 'corrupt commit-graph write (missing tree)' '\n \t)\n '\n \n+# The remaining tests check timestamps that flow over\n+# 32-bits. The graph_git_behavior checks can't take a\n+# prereq, so just stop here if we are on a 32-bit machine.\n+\n+if ! test_have_prereq TIME_IS_64BIT || ! test_have_prereq TIME_T_IS_64BIT\n+then\n+\ttest_done\n+fi\n+\n # We test the overflow-related code with the following repo history:\n #\n # 4:F - 5:N - 6:U\n@@ -832,10 +841,10 @@ test_expect_success 'corrupt commit-graph write (missing tree)' '\n # The largest offset observed is 2 ^ 31, just large enough to overflow.\n #\n \n-test_expect_success 'set up and verify repo with generation data overflow chunk' '\n+test_expect_success TIME_IS_64BIT,TIME_T_IS_64BIT 'set up and verify repo with generation data overflow chunk' '\n \tobjdir=\".git/objects\" &&\n \tUNIX_EPOCH_ZERO=\"@0 +0000\" &&\n-\tFUTURE_DATE=\"@2147483646 +0000\" &&\n+\tFUTURE_DATE=\"@4000000000 +0000\" &&\n \ttest_oid_cache <<-EOF &&\n \toid_version sha1:1\n \toid_version sha256:2\n@@ -867,4 +876,8 @@ test_expect_success 'set up and verify repo with generation data overflow chunk'\n \n graph_git_behavior 'generation data overflow chunk repo' repo left right\n \n+# Do not add tests at the end of this file, unless they require 64-bit\n+# timestamps, since this portion of the script is only executed when\n+# time data types have 64 bits.\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"449763","messageId":"YhzkdMxrIGlNutr6@ncase","threadId":"57472","inReplyTo":"a3436b92a32f7f6dd02ad61eb2337a4d088d5e9c.1645735117.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-02-28T15:18:20Z","receivedAt":"2022-02-28T15:18:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Feb 24, 2022 at 08:38:32PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The 'read_generation_data' member of 'struct commit_graph' was\n> introduced by 1fdc383c5 (commit-graph: use generation v2 only if entire\n> chain does, 2021-01-16). The intention was to avoid using corrected\n> commit dates if not all layers of a commit-graph had that data stored.\n> The logic in validate_mixed_generation_chain() at that point incorrectly\n> initialized read_generation_data to 1 if and only if the tip\n> commit-graph contained the Corrected Commit Date chunk.\n> \n> This was \"fixed\" in 448a39e65 (commit-graph: validate layers for\n> generation data, 2021-02-02) to validate that read_generation_data was\n> either non-zero for all layers, or it would set read_generation_data to\n> zero for all layers.\n> \n> The problem here is that read_generation_data is not initialized to be\n> non-zero anywhere!\n> \n> This change initializes read_generation_data immediately after the chunk\n> is parsed, so each layer will have its value present as soon as\n> possible.\n> \n> The read_generation_data member is used in fill_commit_graph_info() to\n> determine if we should use the corrected commit date or the topological\n> levels stored in the Commit Data chunk. Due to this bug, all previous\n> versions of Git were defaulting to topological levels in all cases!\n> \n> This can be measured with some performance tests. Using the Linux kernel\n> as a testbed, I generated a complete commit-graph containing corrected\n> commit dates and tested the 'new' version against the previous, 'old'\n> version.\n> \n> First, rev-list with --topo-order demonstrates a 26% improvement using\n> corrected commit dates:\n> \n> hyperfine \\\n> \t-n \"old\" \"$OLD_GIT rev-list --topo-order -1000 v3.6\" \\\n> \t-n \"new\" \"$NEW_GIT rev-list --topo-order -1000 v3.6\" \\\n> \t--warmup=10\n> \n> Benchmark 1: old\n> Time (mean ± σ): 57.1 ms ± 3.1 ms\n> Range (min … max): 52.9 ms … 62.0 ms 55 runs\n> \n> Benchmark 2: new\n> Time (mean ± σ): 45.5 ms ± 3.3 ms\n> Range (min … max): 39.9 ms … 51.7 ms 59 runs\n> \n> Summary\n> 'new' ran\n> 1.26 ± 0.11 times faster than 'old'\n> \n> These performance improvements are due to the algorithmic improvements\n> given by walking fewer commits due to the higher cutoffs from corrected\n> commit dates.\n> \n> However, this comes at a cost. The additional I/O cost of parsing the\n> corrected commit dates is visible in case of merge-base commands that do\n> not reduce the overall number of walked commits.\n> \n> hyperfine \\\n> -n \"old\" \"$OLD_GIT merge-base v4.8 v4.9\" \\\n> -n \"new\" \"$NEW_GIT merge-base v4.8 v4.9\" \\\n> --warmup=10\n> \n> Benchmark 1: old\n> Time (mean ± σ): 110.4 ms ± 6.4 ms\n> Range (min … max): 96.0 ms … 118.3 ms 25 runs\n> \n> Benchmark 2: new\n> Time (mean ± σ): 150.7 ms ± 1.1 ms\n> Range (min … max): 149.3 ms … 153.4 ms 19 runs\n> \n> Summary\n> 'old' ran\n> 1.36 ± 0.08 times faster than 'new'\n> \n> Performance issues like this are what motivated 702110aac (commit-graph:\n> use config to specify generation type, 2021-02-25).\n> \n> In the future, we could fix this performance problem by inserting the\n> corrected commit date offsets into the Commit Date chunk instead of\n> having that data in an extra chunk.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 3 +++\n> t/t4216-log-bloom.sh | 2 +-\n> t/t5318-commit-graph.sh | 14 ++++++++++++--\n> t/t5324-split-commit-graph.sh | 9 +++++++--\n> 4 files changed, 23 insertions(+), 5 deletions(-)\n> \n> diff --git a/commit-graph.c b/commit-graph.c\n> index a19bd96c2ee..8e52bb09552 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -407,6 +407,9 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n> \t\t\t&graph->chunk_generation_data);\n> \t\tpair_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW,\n> \t\t\t&graph->chunk_generation_data_overflow);\n> +\n> +\t\tif (graph->chunk_generation_data)\n> +\t\t\tgraph->read_generation_data = 1;\n> \t}\n> \n> \tif (r->settings.commit_graph_read_changed_paths) {\n\nI wanted to test your changes because they seem quite exciting in the\ncontext of my work as well, but this commit seems to uncover a bug with\nhow we handle overflows. I originally triggered the bug when trying to\ndo a mirror-fetch, but as it turns it seems to trigger now whenever the\ncommit-graph is being read:\n\n $ git commit-graph verify\n fatal: commit-graph requires overflow generation data but has none\n\n $ git commit-graph write --split\n Finding commits for commit graph among packed objects: 100% (10235119/10235119), done.\n fatal: commit-graph requires overflow generation data but has none\n\n $ git commit-graph write --split=replace\n Finding commits for commit graph among packed objects: 100% (10235119/10235119), done.\n fatal: commit-graph requires overflow generation data but has none\n\nI initially assumed this may be a bug with how we previously wrote the\ncommit-graph, but removing all chains still reliably triggers it:\n\n $ rm -f objects/info/commit-graphs/*\n $ git commit-graph write --split\n Finding commits for commit graph among packed objects: 100% (10235119/10235119), done.\n fatal: commit-graph requires overflow generation data but has none\n\nI haven't yet found the time to dig deeper into why this is happening.\nWhile the repository is publicly accessible at [1], unfortunately the\nbug seems to be triggered by a commit that's only kept alive by an\ninternal reference.\n\nPatrick\n\n[1]: https://gitlab.com/gitlab-com/www-gitlab-com.git\n"},{"id":"449764","messageId":"220228.86pmn73toq.gmgdl@evledraar.gmail.com","threadId":"57472","inReplyTo":"d19f5ee8-af92-805f-7ea2-8285862c1123@github.com","subject":"Re: [PATCH 5/7] commit-graph: document file format v2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-28T14:27:49Z","receivedAt":"2022-02-28T15:19:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Feb 28 2022, Derrick Stolee wrote:\n\n> On 2/25/2022 5:31 PM, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> On Thu, Feb 24 2022, Derrick Stolee via GitGitGadget wrote:\n>> \n> ...\n>>> Generation Data (ID: {'G', 'D', 'A', 'T' }) (N * 4 bytes) [Optional]\n>>> * This list of 4-byte values store corrected commit date offsets for the\n>>> @@ -103,6 +112,9 @@ CHUNK DATA:\n>>> * Generation Data chunk is present only when commit-graph file is written\n>>> by compatible versions of Git and in case of split commit-graph chains,\n>>> the topmost layer also has Generation Data chunk.\n>>> + * This chunk does not exist if the commit-graph file format version is 2,\n>>> + because the corrected commit date offset data is stored in the Commit\n>>> + Data chunk.\n>>> \n>>> Generation Data Overflow (ID: {'G', 'D', 'O', 'V' }) [Optional]\n>>> * This list of 8-byte values stores the corrected commit date offsets\n>> \n>> We talked a while ago now about how we do commit-graph format changes\n>> and this is partially echoing those earlier questions[1] from 2019.\n>> \n>> I fully understand why we're writing this amended CDAT chunk in a\n>> different layout. By not having the GDAT side-chunk to look up in the\n>> data is more local, that part of the file is more compact etc.\n>> \n>> What I don't understand is why getting those performance improvements\n>> requires the breaking version change & the writing of the incompatible\n>> version number.\n>> \n>> I.e. couldn't the differently formatted CDAT chunk be written instead to a new\n>> chunk name (say \"2DAT\") instead? Per [1] we'd pay a small fixed cost for\n>> a possibly empty chunk (I didn't re-do those numbers), but surely the\n>> performance improvements will be about the same for that miniscule\n>> overhead.\n>\n> CDAT is a required chunk. It is part of the v1 spec that CDAT exists\n> and is correct. All other Git clients will error out when reading a\n> \"v1\" graph without such a chunk, and in a way that is less helpful to\n> users. Instead of clearly indicating \"file version is too new\" it will\n> say \"commit-graph is missing the Commit Data chunk\" which is not\n> helpful.\n\nYes. That would be the worst of both worlds.\n\nI thought the reference to the 2019-era post made it clear (which is\nexplicit about this aspect), but I'm talking about writing one of:\n\n A. An empty chunk\n B. Keeping a \"stale\" chunk around (as we re-write the graph)\n C. Duplicate writes of new/old chunks.\n\nAnd not simply omitting the CDAT chunk. As you point out would give you\nall the drawbacks of a version number change, with none of the benefits.\n\nI haven't re-tested this now, but at the time doing any of (A..C) would\nwork smoothly for older clients, while giving newer ones improved data.\n\n>> It will give you something you can't have here, which is optional\n>> compatibility with older clients by writing both versions. That'll be a\n>> ~2x as large file on disk, but with the page cache & each client version\n>> skipping to the data it needs caching characteristics & data locality\n>> should work out to about the same thing.\n>\n> Writing both is the only way that this could work without incrementing\n> the graph version number, but I'd rather just update the number and\n> avoid wasting the effort to write that extra data.\n\n...\n\n> It seems you are hyper-focused on \"we don't _need_ to update the version\n> number\" and you are willing to recommend wasteful approaches in order to\n> support that stance.\n\nI'd say less hyper-focused, and more clarifying an IMO major unstated\ntrade-off of the proposed format change.\n\n> So: you're right. We don't _need_ to update the version number. But this\n> is the best choice among the options available.\n\n...\n\n>> Or maybe they won't. I just found it surprising when reviewing this to\n>> not find an answer to why that approach wasn't\n>> considered.\n>\n> The point is to create a new format that can be chosen when deployed\n> in an environment where older Git versions will not exist (such as\n> a Git server). The new version is not chosen by default and instead\n> is opt-in through the commitGraph.generationVersion config option.\n>\n> Perhaps in a year or two we would consider making this the new\n> default, but there is no rush to do so.\n\nLooking into this a bit more I think that in either case this is less of\na big deal after my 43d35618055 (commit-graph write: don't die if the\nexisting graph is corrupt, 2019-03-25), which came out of some of those\ndiscussions at the time of [1].\n\nI.e. now a client that only understands version N-1 will warn when\nloading it, wheras it's only if a pre-v2.22.0 client (which has that\ncommit) reads the repository that we'd hard die on it, correct?\n\nBut speaking of hyper-focus. I think that arguably applies to you in\nthis case when considering the trade-offs of these sorts of format\nchanges :)\n\nI.e. you're primarily considering cases of say a git server (presumably\nrunning on GitHub) or another such deployment where it's easy to have\nfull control over all of your versions \"in the wild\".\n\nAnd thus a three-phase rollout of something like a format change can be\ndone in a timely and predictable manner.\n\nBut git is used by *a lot* of people in a bunch of different\nscenarios. E.g.:\n\n * A shared (hopefully read-only) NFS mounted by remote \"unmanaged\" clients.\n * A tarred-up directory including a .git, which may be transferred to\n a machine with a pre-v2.22.0 version.\n\nOr even softer cases of failure, such as:\n\n * A cronjob causes an alert/incident somewhere because the server \n operator started writing a new version, but forgot about a set\n of machines that are still on the old version.\n\nI think that even if it's less conceptually clean it's worth considering\nbeing over backwards to be kinder to such use-cases, unless it's really\nrequired for other reasons to break such in-the-wild use-cases.\n\nOr in this case, if it's thought to be worth it to help reviewers decide\nby separating the performance improvement aspect from the changed\ninteraction between new graphs and older clients.\n\nAs a further nit on the proposed end-state here: Do I understand it\ncorrectly that commitGraph.generationVersion=[1|2] (i.e. on current\n\"master\") will always result in a file that's compatible with older\nversions, since the only thing \"v2\" there controls now is to write the\noptional GDAT and GDOV chunks?\n\nWhereas going from commitGraph.generationVersion=2 to\ncommitGraph.generationVersion=3 in this series will impact older clients\nas noted above, since we're bumping the version (of the file, to 2 if\nthe config is 3, which as Junio noted is a bit confusing).\n\nI think if you're set on going down the path of bumping the top-level\nversion that deserves to be made much clearer in the added\ndocumentation. Right now the only hint to that is a passing mention that\nfor v3:\n\n [it] will be incompatible with some old versions of Git\n\nWhich if we're opting for breaking format changes really should note\nsome of the caveats above, that pre-v2.22.0 hard-dies, and probably\ndescribe \"some old versions of Git\" a bit more clearly.\n\nIt actually means once this gets released \"the git version that was the\nlatest one you could download yesterday\". Which a reader of the docs\nprobably won't expect when starting to play with this in mixed-version\nenvironment.\n\n1. https://lore.kernel.org/git/87h8acivkh.fsf@evledraar.gmail.com/\n"},{"id":"449765","messageId":"220228.86lexv3tgb.gmgdl@evledraar.gmail.com","threadId":"57472","inReplyTo":"2f89275314b4a2a89a18d14e41602bbe2e1988dc.1646056423.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/4] test-read-graph: include extra post-parse info","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-28T15:22:04Z","receivedAt":"2022-02-28T15:24:26Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Feb 28 2022, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> It can be helpful to verify that the 'struct commit_graph' that results\n> from parsing a commit-graph is correctly structured. The existence of\n> different chunks is not enough to verify that all of the optional\n> features are correctly enabled.\n>\n> Update 'test-tool read-graph' to output an \"options:\" line that includes\n> information for different parts of the struct commit_graph.\n>\n> In particular, this change demonstrates that the read_generation_data\n> option is never being enabled, which will be fixed in a later change.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> t/helper/test-read-graph.c | 13 +++++++++++++\n> t/t4216-log-bloom.sh | 1 +\n> t/t5318-commit-graph.sh | 1 +\n> t/t5324-split-commit-graph.sh | 5 +++++\n> 4 files changed, 20 insertions(+)\n>\n> diff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\n> index 75927b2c81d..c3b6b8d1734 100644\n> --- a/t/helper/test-read-graph.c\n> +++ b/t/helper/test-read-graph.c\n> @@ -3,6 +3,7 @@\n> #include \"commit-graph.h\"\n> #include \"repository.h\"\n> #include \"object-store.h\"\n> +#include \"bloom.h\"\n> \n> int cmd__read_graph(int argc, const char **argv)\n> {\n> @@ -45,6 +46,18 @@ int cmd__read_graph(int argc, const char **argv)\n> \t\tprintf(\" bloom_data\");\n> \tprintf(\"\\n\");\n> \n> +\tprintf(\"options:\");\n> +\tif (graph->bloom_filter_settings)\n> +\t\tprintf(\" bloom(%d,%d,%d)\",\n\nI think this is probably unportable, as other code (including in\ncommit-graph.c) uses \"%\"PRIu32 when printing uint32_t.\n\nDoes this work on our Linux32 job? I was going to quickly check the PR\nCI, but it appears the run was skipped for some reason.\n\n> +\t\t graph->bloom_filter_settings->hash_version,\n> +\t\t graph->bloom_filter_settings->bits_per_entry,\n> +\t\t graph->bloom_filter_settings->num_hashes);\n> +\tif (graph->read_generation_data)\n> +\t\tprintf(\" read_generation_data\");\n> +\tif (graph->topo_levels)\n> +\t\tprintf(\" topo_levels\");\n> +\tprintf(\"\\n\");\n> +\n> \tUNLEAK(graph);\n> \n> \treturn 0;\n> diff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\n> index cc3cebf6722..5ed6d2a21c1 100755\n> --- a/t/t4216-log-bloom.sh\n> +++ b/t/t4216-log-bloom.sh\n> @@ -48,6 +48,7 @@ graph_read_expect () {\n> \theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n> \tnum_commits: $1\n> \tchunks: oid_fanout oid_lookup commit_metadata generation_data bloom_indexes bloom_data\n> +\toptions: bloom(1,10,7)\n> \tEOF\n> \ttest-tool read-graph >actual &&\n> \ttest_cmp expect actual\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index edb728f77c3..2b05026cf6d 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -104,6 +104,7 @@ graph_read_expect() {\n> \theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n> \tnum_commits: $1\n> \tchunks: oid_fanout oid_lookup commit_metadata$OPTIONAL\n> +\toptions:\n> \tEOF\n> \ttest-tool read-graph >output &&\n> \ttest_cmp expect output\n> diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\n> index 847b8097109..778fa418de2 100755\n> --- a/t/t5324-split-commit-graph.sh\n> +++ b/t/t5324-split-commit-graph.sh\n> @@ -34,6 +34,7 @@ graph_read_expect() {\n> \theader: 43475048 1 $(test_oid oid_version) 4 $NUM_BASE\n> \tnum_commits: $1\n> \tchunks: oid_fanout oid_lookup commit_metadata generation_data\n> +\toptions:\n> \tEOF\n> \ttest-tool read-graph >output &&\n> \ttest_cmp expect output\n> @@ -508,6 +509,7 @@ test_expect_success 'setup repo for mixed generation commit-graph-chain' '\n> \t\theader: 43475048 1 $(test_oid oid_version) 4 1\n> \t\tnum_commits: $NUM_SECOND_LAYER_COMMITS\n> \t\tchunks: oid_fanout oid_lookup commit_metadata\n> +\t\toptions:\n> \t\tEOF\n> \t\ttest_cmp expect output &&\n> \t\tgit commit-graph verify &&\n> @@ -540,6 +542,7 @@ test_expect_success 'do not write generation data chunk if not present on existi\n> \t\theader: 43475048 1 $(test_oid oid_version) 4 2\n> \t\tnum_commits: $NUM_THIRD_LAYER_COMMITS\n> \t\tchunks: oid_fanout oid_lookup commit_metadata\n> +\t\toptions:\n> \t\tEOF\n> \t\ttest_cmp expect output &&\n> \t\tgit commit-graph verify\n> @@ -581,6 +584,7 @@ test_expect_success 'do not write generation data chunk if the topmost remaining\n> \t\theader: 43475048 1 $(test_oid oid_version) 4 2\n> \t\tnum_commits: $(($NUM_THIRD_LAYER_COMMITS + $NUM_FOURTH_LAYER_COMMITS))\n> \t\tchunks: oid_fanout oid_lookup commit_metadata\n> +\t\toptions:\n> \t\tEOF\n> \t\ttest_cmp expect output &&\n> \t\tgit commit-graph verify\n> @@ -620,6 +624,7 @@ test_expect_success 'write generation data chunk if topmost remaining layer has\n> \t\theader: 43475048 1 $(test_oid oid_version) 5 1\n> \t\tnum_commits: $(($NUM_SECOND_LAYER_COMMITS + $NUM_THIRD_LAYER_COMMITS + $NUM_FOURTH_LAYER_COMMITS + $NUM_FIFTH_LAYER_COMMITS))\n> \t\tchunks: oid_fanout oid_lookup commit_metadata generation_data\n> +\t\toptions:\n> \t\tEOF\n> \t\ttest_cmp expect output\n> \t)\n\nI think the rest of this all looks good and obviously correct.\n"},{"id":"449768","messageId":"220228.86h78j3t67.gmgdl@evledraar.gmail.com","threadId":"57472","inReplyTo":"cbcbf10e6994d469abc5463d08165e6a2ba73153.1646056423.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/4] commit-graph: fix ordering bug in generation numbers","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-28T15:25:06Z","receivedAt":"2022-02-28T15:30:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Feb 28 2022, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n> [...]\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 265c010122e..a19bd96c2ee 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -1556,12 +1556,16 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n> \t\t\t\tif (current->date && current->date > max_corrected_commit_date)\n> \t\t\t\t\tmax_corrected_commit_date = current->date - 1;\n> \t\t\t\tcommit_graph_data_at(current)->generation = max_corrected_commit_date + 1;\n> -\n> -\t\t\t\tif (commit_graph_data_at(current)->generation - current->date > GENERATION_NUMBER_V2_OFFSET_MAX)\n> -\t\t\t\t\tctx->num_generation_data_overflows++;\n> \t\t\t}\n> \t\t}\n> \t}\n> +\n> +\tfor (i = 0; i < ctx->commits.nr; i++) {\n> +\t\tstruct commit *c = ctx->commits.list[i];\n> +\t\ttimestamp_t offset = commit_graph_data_at(c)->generation - c->date;\n> +\t\tif (offset > GENERATION_NUMBER_V2_OFFSET_MAX)\n> +\t\t\tctx->num_generation_data_overflows++;\n> +\t}\n> \tstop_progress(&ctx->progress);\n> }\n> \n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 2b05026cf6d..f9bffe38013 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -467,10 +467,10 @@ test_expect_success 'warn on improper hash version' '\n> \t)\n> '\n> \n> -test_expect_success 'lower layers have overflow chunk' '\n> +test_expect_success TIME_IS_64BIT,TIME_T_IS_64BIT 'lower layers have overflow chunk' '\n> \tcd \"$TRASH_DIRECTORY/full\" &&\n> \tUNIX_EPOCH_ZERO=\"@0 +0000\" &&\n> -\tFUTURE_DATE=\"@2147483646 +0000\" &&\n> +\tFUTURE_DATE=\"@4147483646 +0000\" &&\n> \trm -f .git/objects/info/commit-graph &&\n> \ttest_commit --date \"$FUTURE_DATE\" future-1 &&\n> \ttest_commit --date \"$UNIX_EPOCH_ZERO\" old-1 &&\n\nIsn't it worth splitting up this test instead, so that we can test cases\nwhere 32 bit timestamps overflow without these new prereqs.\n\nUnless I'm missing something that would just be a matter of splitting\nthis test into helper that takes that $FUTURE_DATE as an argument, then\nrunning it for both timestamps, with TIME_IS_64BIT,TIME_T_IS_64BIT on\nthe 64 bit one.\n\nOr maybe I don't get it, but it seems like we're throwing out some\ncarefully considered testing for 32 bit compatibility with the\nproverbial bath water here....\n\nAside from that I wonder how this interacts with both:\n\n #define CORRECTED_COMMIT_DATE_OFFSET_OVERFLOW (1ULL << 31)\n\nAnd this existing code, where offset is timestamp_t, but\nnum_generation_data_overflows is an \"int\":\n\n offset = CORRECTED_COMMIT_DATE_OFFSET_OVERFLOW | num_generation_data_overflows;\n\nThat proooobably does the right thing if int is say 32 bit, but\ntimestamp_t is 64 bit (does such an OS exist?), but maybe worth looking\nat with a second pair of eyes...\n"},{"id":"449769","messageId":"220228.86czj73sre.gmgdl@evledraar.gmail.com","threadId":"57472","inReplyTo":"5bc6a7660d897ca6c52eabba8fb9ecfb6304dabb.1646056423.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/4] commit-graph: start parsing generation v2 (again)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-28T15:30:39Z","receivedAt":"2022-02-28T15:39:23Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Feb 28 2022, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n> [...]\n> +\tGENERATION_VERSION=2\n> +\tif test ! -z \"$3\"\n\nTIL this works somewhere :)\n\nI thought it *might* be unportable behavior (but didn't check at\nfirst), but it's not! We have a few such cases already.\n\nBut IMO much less puzzling would be at least:\n\n if ! test -z \"$3\"\n\nOr in this case, more plainly:\n\n if test -n \"$3\"\n\n> +\tthen\n> +\t\tGENERATION_VERSION=$3\n> +\tfi\n> +\tOPTIONS=\n> +\tif test $GENERATION_VERSION -gt 1\n> +\tthen\n> +\t\tOPTIONS=\" read_generation_data\"\n> +\tfi\n\nOr actually, since we don't use $GENERATION_VERSION further down setting\nit to a default etc. here seems a bit odd. Perhaps something closer to:\n\n if test $# -eq 3 && test $3 -gt 1\n\nIt's also possible to be more clever as e.g.:\n\n test \"${3:-2}\" -gt 1\n\nBut that hardly seems worth it...\n\n> NUM_COMMITS=9\n> diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\n> index 778fa418de2..669ddc645fa 100755\n> --- a/t/t5324-split-commit-graph.sh\n> +++ b/t/t5324-split-commit-graph.sh\n> @@ -30,11 +30,16 @@ graph_read_expect() {\n> \tthen\n> \t\tNUM_BASE=$2\n> \tfi\n> +\tOPTIONS=\n> +\tif test -z \"$3\"\n> +\tthen\n> +\t\tOPTIONS=\" read_generation_data\"\n> +\tfi\n> \tcat >expect <<- EOF\n> \theader: 43475048 1 $(test_oid oid_version) 4 $NUM_BASE\n> \tnum_commits: $1\n> \tchunks: oid_fanout oid_lookup commit_metadata generation_data\n> -\toptions:\n> +\toptions:$OPTIONS\n> \tEOF\n> \ttest-tool read-graph >output &&\n> \ttest_cmp expect output\n\nNot a new issue, but it would be nice to have the mostly copy/pasted\ncode in a lib-commit-graph.sh or something, but probably too distracting\nfor this small series...\n"},{"id":"449770","messageId":"220228.868rtv3sfx.gmgdl@evledraar.gmail.com","threadId":"57472","inReplyTo":"193217c71e0aaf3f56a02d9abec6753bd19aba71.1646056423.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/4] commit-graph: fix generation number v2 overflow values","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-28T15:40:26Z","receivedAt":"2022-02-28T15:46:17Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Feb 28 2022, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> The Generation Data Chunk was implemented and tested in e8b63005c\n> (commit-graph: implement generation data chunk, 2021-01-16), but the\n> test was carefully constructed to work on systems with 32-bit dates.\n> Since the corrected commit date offsets still required more than 31\n> bits, this triggered writing the generation_data_overflow chunk.\n>\n> However, upon closer look, the\n> write_graph_chunk_generation_data_overflow() method writes the offsets\n> to the chunk (as dictated by the format) but fill_commit_graph_info()\n> treats the value in the chunk as if it is the full corrected commit date\n> (not an offset). For some reason, this does not cause an issue when\n> using the FUTURE_DATE specified in t5318-commit-graph.sh, but it does\n> show up as a failure in 'git commit-graph verify' if we increase that\n> FUTURE_DATE to be above four billion.\n>\n> Fix this error and update the test to require 64-bit dates so we can\n> safely use this large value in our test.\n\nHrm, so perhaps re my comment on 2/4 @2147483646 was never used? I'm not\nsure I understand this.\n\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 8e52bb09552..b86a6a634fe 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -806,7 +806,7 @@ static void fill_commit_graph_info(struct commit *item, struct commit_graph *g,\n> \t\t\t\tdie(_(\"commit-graph requires overflow generation data but has none\"));\n> \n> \t\t\toffset_pos = offset ^ CORRECTED_COMMIT_DATE_OFFSET_OVERFLOW;\n> -\t\t\tgraph_data->generation = get_be64(g->chunk_generation_data_overflow + 8 * offset_pos);\n> +\t\t\tgraph_data->generation = item->date + get_be64(g->chunk_generation_data_overflow + 8 * offset_pos);\n> \t\t} else\n> \t\t\tgraph_data->generation = item->date + offset;\n> \t} else\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 1afee1c2705..f4ffaad661d 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -815,6 +815,15 @@ test_expect_success 'corrupt commit-graph write (missing tree)' '\n> \t)\n> '\n\nThis goes back to my comment on 3/4 but:\n\n> +# The remaining tests check timestamps that flow over\n> +# 32-bits. The graph_git_behavior checks can't take a\n> +# prereq, so just stop here if we are on a 32-bit machine.\n> +\n> +if ! test_have_prereq TIME_IS_64BIT || ! test_have_prereq TIME_T_IS_64BIT\n> +then\n> +\ttest_done\n> +fi\n> +\n\nThis...(continued)...\n\n> # We test the overflow-related code with the following repo history:\n> #\n> # 4:F - 5:N - 6:U\n> @@ -832,10 +841,10 @@ test_expect_success 'corrupt commit-graph write (missing tree)' '\n> # The largest offset observed is 2 ^ 31, just large enough to overflow.\n> #\n> \n> -test_expect_success 'set up and verify repo with generation data overflow chunk' '\n> +test_expect_success TIME_IS_64BIT,TIME_T_IS_64BIT 'set up and verify repo with generation data overflow chunk' '\n> \tobjdir=\".git/objects\" &&\n> \tUNIX_EPOCH_ZERO=\"@0 +0000\" &&\n> -\tFUTURE_DATE=\"@2147483646 +0000\" &&\n> +\tFUTURE_DATE=\"@4000000000 +0000\" &&\n\nHrm, again this may be over my head, but @4147483646 instead of\n@2147483646 in the other test, but @4000000000 instead of the same here?\n\n\n> \ttest_oid_cache <<-EOF &&\n> \toid_version sha1:1\n> \toid_version sha256:2\n> @@ -867,4 +876,8 @@ test_expect_success 'set up and verify repo with generation data overflow chunk'\n> \n> graph_git_behavior 'generation data overflow chunk repo' repo left right\n> \n> +# Do not add tests at the end of this file, unless they require 64-bit\n> +# timestamps, since this portion of the script is only executed when\n> +# time data types have 64 bits.\n> +\n> test_done\n\n...and this would really be much nicer if we split this test up into its\nown file, which would be obviously named to note the\nissue. tXXXX-commit-graph-64bit-timestamp.sh or something.\n\nAs shown in my recent 0a2bfccb9c8 (t0051: use \"skip_all\" under !MINGW in\nsingle-test file, 2022-02-04) you'll also get much nicer output from\n\"prove\" in that case.\n \n"},{"id":"449776","messageId":"e29ec01f-3039-6992-8ade-800ad32fcf34@github.com","threadId":"57472","inReplyTo":"YhzkdMxrIGlNutr6@ncase","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-28T16:23:38Z","receivedAt":"2022-02-28T16:23:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/28/2022 10:18 AM, Patrick Steinhardt wrote:\n> On Thu, Feb 24, 2022 at 08:38:32PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n...\n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index a19bd96c2ee..8e52bb09552 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> @@ -407,6 +407,9 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n>> \t\t\t&graph->chunk_generation_data);\n>> \t\tpair_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW,\n>> \t\t\t&graph->chunk_generation_data_overflow);\n>> +\n>> +\t\tif (graph->chunk_generation_data)\n>> +\t\t\tgraph->read_generation_data = 1;\n>> \t}\n>> \n>> \tif (r->settings.commit_graph_read_changed_paths) {\n> \n> I wanted to test your changes because they seem quite exciting in the\n> context of my work as well, but this commit seems to uncover a bug with\n> how we handle overflows. I originally triggered the bug when trying to\n> do a mirror-fetch, but as it turns it seems to trigger now whenever the\n> commit-graph is being read:\n> \n> $ git commit-graph verify\n> fatal: commit-graph requires overflow generation data but has none\n> \n> $ git commit-graph write --split\n> Finding commits for commit graph among packed objects: 100% (10235119/10235119), done.\n> fatal: commit-graph requires overflow generation data but has none\n> \n> $ git commit-graph write --split=replace\n> Finding commits for commit graph among packed objects: 100% (10235119/10235119), done.\n> fatal: commit-graph requires overflow generation data but has none\n> \n> I initially assumed this may be a bug with how we previously wrote the\n> commit-graph, but removing all chains still reliably triggers it:\n> \n> $ rm -f objects/info/commit-graphs/*\n> $ git commit-graph write --split\n> Finding commits for commit graph among packed objects: 100% (10235119/10235119), done.\n> fatal: commit-graph requires overflow generation data but has none\n> \n> I haven't yet found the time to dig deeper into why this is happening.\n> While the repository is publicly accessible at [1], unfortunately the\n> bug seems to be triggered by a commit that's only kept alive by an\n> internal reference.\n> \n> Patrick\n> \n> [1]: https://gitlab.com/gitlab-com/www-gitlab-com.git\n\nThanks for including this information. Just to be clear: did you\ninclude patch 4 in your tests, or not? Patch 4 includes a fix\nrelated to overflow values, so it would be helpful to know if you\nfound a _different_ bug or if it is the same one.\n\nThanks,\n-Stolee\n"},{"id":"449779","messageId":"e74d72bc-b6f5-2e1c-63d1-d3a580f3dc11@github.com","threadId":"57472","inReplyTo":"220228.86pmn73toq.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 5/7] commit-graph: document file format v2","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-28T16:39:43Z","receivedAt":"2022-02-28T16:39:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/28/2022 9:27 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Feb 28 2022, Derrick Stolee wrote:\n> \n>> On 2/25/2022 5:31 PM, Ævar Arnfjörð Bjarmason wrote:\n\n>>> Or maybe they won't. I just found it surprising when reviewing this to\n>>> not find an answer to why that approach wasn't\n>>> considered.\n>>\n>> The point is to create a new format that can be chosen when deployed\n>> in an environment where older Git versions will not exist (such as\n>> a Git server). The new version is not chosen by default and instead\n>> is opt-in through the commitGraph.generationVersion config option.\n>>\n>> Perhaps in a year or two we would consider making this the new\n>> default, but there is no rush to do so.\n> \n> Looking into this a bit more I think that in either case this is less of\n> a big deal after my 43d35618055 (commit-graph write: don't die if the\n> existing graph is corrupt, 2019-03-25), which came out of some of those\n> discussions at the time of [1].\n> \n> I.e. now a client that only understands version N-1 will warn when\n> loading it, wheras it's only if a pre-v2.22.0 client (which has that\n> commit) reads the repository that we'd hard die on it, correct?\n> \n> But speaking of hyper-focus. I think that arguably applies to you in\n> this case when considering the trade-offs of these sorts of format\n> changes :)\n> \n> I.e. you're primarily considering cases of say a git server (presumably\n> running on GitHub) or another such deployment where it's easy to have\n> full control over all of your versions \"in the wild\".\n\nI'm thinking of servers, yes, but also 99% of clients who only upgrade\n(or _maybe_ downgrade to a recent, previous version occasionally).\n \n> And thus a three-phase rollout of something like a format change can be\n> done in a timely and predictable manner.\n> \n> But git is used by *a lot* of people in a bunch of different\n> scenarios. E.g.:\n> \n> * A shared (hopefully read-only) NFS mounted by remote \"unmanaged\" clients.\n> * A tarred-up directory including a .git, which may be transferred to\n> a machine with a pre-v2.22.0 version.\n> \n> Or even softer cases of failure, such as:\n> \n> * A cronjob causes an alert/incident somewhere because the server \n> operator started writing a new version, but forgot about a set\n> of machines that are still on the old version.\n\nIt is important to continue supporting these cases, and this change does\nnot cause any issues for them. However, this handful of corner cases\nshould not block progress in the main cases.\n\n> I think that even if it's less conceptually clean it's worth considering\n> being over backwards to be kinder to such use-cases, unless it's really\n> required for other reasons to break such in-the-wild use-cases.\n> \n> Or in this case, if it's thought to be worth it to help reviewers decide\n> by separating the performance improvement aspect from the changed\n> interaction between new graphs and older clients.\n> \n> As a further nit on the proposed end-state here: Do I understand it\n> correctly that commitGraph.generationVersion=[1|2] (i.e. on current\n> \"master\") will always result in a file that's compatible with older\n> versions, since the only thing \"v2\" there controls now is to write the\n> optional GDAT and GDOV chunks?\n> \n> Whereas going from commitGraph.generationVersion=2 to\n> commitGraph.generationVersion=3 in this series will impact older clients\n> as noted above, since we're bumping the version (of the file, to 2 if\n> the config is 3, which as Junio noted is a bit confusing).\n> \n> I think if you're set on going down the path of bumping the top-level\n> version that deserves to be made much clearer in the added\n> documentation. Right now the only hint to that is a passing mention that\n> for v3:\n> \n> [it] will be incompatible with some old versions of Git\n> \n> Which if we're opting for breaking format changes really should note\n> some of the caveats above, that pre-v2.22.0 hard-dies, and probably\n> describe \"some old versions of Git\" a bit more clearly.\n> \n> It actually means once this gets released \"the git version that was the\n> latest one you could download yesterday\". Which a reader of the docs\n> probably won't expect when starting to play with this in mixed-version\n> environment.\n> \n> 1. https://lore.kernel.org/git/87h8acivkh.fsf@evledraar.gmail.com/\n\nThis documentation could be altered to be specific about versions,\nbut such a specific change makes assumptions of the version that will\ninclude it. As of now, the generation number v2 fixes will _probably_\nget in for 2.36 and the format change would have enough time to cook\nfor 2.37, so I'll update the docs to refer to that version explicitly.\n\nThe pre-2.22.0 change might be helpful to mention, but it could also be\nnoise to the reader. We can revisit this when these patches are\nsubmitted again in another thread. There's also concern about third-\nparty tools like libgit2. I'd rather draw the line as \"tread carefully\nhere\" than \"here is so much information that a reader might think it\nis all they need to know\".\n\nThanks,\n-Stolee\n"},{"id":"449781","messageId":"b65f405e-665b-ae32-aa4e-7dbdc874485d@github.com","threadId":"57472","inReplyTo":"220228.86czj73sre.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v2 3/4] commit-graph: start parsing generation v2 (again)","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-28T16:43:59Z","receivedAt":"2022-02-28T16:44:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"\n\nOn 2/28/2022 10:30 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Feb 28 2022, Derrick Stolee via GitGitGadget wrote:\n> \n>> From: Derrick Stolee \n>> [...]\n>> +\tGENERATION_VERSION=2\n>> +\tif test ! -z \"$3\"\n> \n> TIL this works somewhere :)\n> \n> I thought it *might* be unportable behavior (but didn't check at\n> first), but it's not! We have a few such cases already.\n> \n> But IMO much less puzzling would be at least:\n> \n> if ! test -z \"$3\"\n> \n> Or in this case, more plainly:\n> \n> if test -n \"$3\"\n\nSure, that makes sense.\n\n>> +\tthen\n>> +\t\tGENERATION_VERSION=$3\n>> +\tfi\n>> +\tOPTIONS=\n>> +\tif test $GENERATION_VERSION -gt 1\n>> +\tthen\n>> +\t\tOPTIONS=\" read_generation_data\"\n>> +\tfi\n> \n> Or actually, since we don't use $GENERATION_VERSION further down setting\n> it to a default etc. here seems a bit odd. Perhaps something closer to:\n> \n> if test $# -eq 3 && test $3 -gt 1\n> \n> It's also possible to be more clever as e.g.:\n> \n> test \"${3:-2}\" -gt 1\n> \n> But that hardly seems worth it...\n\nI prefer to use a variable so the code is self-documenting.\n\n>> NUM_COMMITS=9\n>> diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\n>> index 778fa418de2..669ddc645fa 100755\n>> --- a/t/t5324-split-commit-graph.sh\n>> +++ b/t/t5324-split-commit-graph.sh\n>> @@ -30,11 +30,16 @@ graph_read_expect() {\n>> \tthen\n>> \t\tNUM_BASE=$2\n>> \tfi\n>> +\tOPTIONS=\n>> +\tif test -z \"$3\"\n>> +\tthen\n>> +\t\tOPTIONS=\" read_generation_data\"\n>> +\tfi\n>> \tcat >expect <<- EOF\n>> \theader: 43475048 1 $(test_oid oid_version) 4 $NUM_BASE\n>> \tnum_commits: $1\n>> \tchunks: oid_fanout oid_lookup commit_metadata generation_data\n>> -\toptions:\n>> +\toptions:$OPTIONS\n>> \tEOF\n>> \ttest-tool read-graph >output &&\n>> \ttest_cmp expect output\n> \n> Not a new issue, but it would be nice to have the mostly copy/pasted\n> code in a lib-commit-graph.sh or something, but probably too distracting\n> for this small series...\n\nThese cases are different enough in the needs of the test files\nthat they cannot be shared without significant complication.\n\nThanks,\n-Stolee\n"},{"id":"449783","messageId":"Yhz/hsDDGNVjt64R@xps","threadId":"57472","inReplyTo":"e29ec01f-3039-6992-8ade-800ad32fcf34@github.com","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-02-28T16:59:50Z","receivedAt":"2022-02-28T16:56:10Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Feb 28, 2022 at 11:23:38AM -0500, Derrick Stolee wrote:\n> On 2/28/2022 10:18 AM, Patrick Steinhardt wrote:\n> > On Thu, Feb 24, 2022 at 08:38:32PM +0000, Derrick Stolee via GitGitGadget wrote:\n> >> From: Derrick Stolee \n> ...\n> >> diff --git a/commit-graph.c b/commit-graph.c\n> >> index a19bd96c2ee..8e52bb09552 100644\n> >> --- a/commit-graph.c\n> >> +++ b/commit-graph.c\n> >> @@ -407,6 +407,9 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n> >> \t\t\t&graph->chunk_generation_data);\n> >> \t\tpair_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW,\n> >> \t\t\t&graph->chunk_generation_data_overflow);\n> >> +\n> >> +\t\tif (graph->chunk_generation_data)\n> >> +\t\t\tgraph->read_generation_data = 1;\n> >> \t}\n> >> \n> >> \tif (r->settings.commit_graph_read_changed_paths) {\n> > \n> > I wanted to test your changes because they seem quite exciting in the\n> > context of my work as well, but this commit seems to uncover a bug with\n> > how we handle overflows. I originally triggered the bug when trying to\n> > do a mirror-fetch, but as it turns it seems to trigger now whenever the\n> > commit-graph is being read:\n> > \n> > $ git commit-graph verify\n> > fatal: commit-graph requires overflow generation data but has none\n> > \n> > $ git commit-graph write --split\n> > Finding commits for commit graph among packed objects: 100% (10235119/10235119), done.\n> > fatal: commit-graph requires overflow generation data but has none\n> > \n> > $ git commit-graph write --split=replace\n> > Finding commits for commit graph among packed objects: 100% (10235119/10235119), done.\n> > fatal: commit-graph requires overflow generation data but has none\n> > \n> > I initially assumed this may be a bug with how we previously wrote the\n> > commit-graph, but removing all chains still reliably triggers it:\n> > \n> > $ rm -f objects/info/commit-graphs/*\n> > $ git commit-graph write --split\n> > Finding commits for commit graph among packed objects: 100% (10235119/10235119), done.\n> > fatal: commit-graph requires overflow generation data but has none\n> > \n> > I haven't yet found the time to dig deeper into why this is happening.\n> > While the repository is publicly accessible at [1], unfortunately the\n> > bug seems to be triggered by a commit that's only kept alive by an\n> > internal reference.\n> > \n> > Patrick\n> > \n> > [1]: https://gitlab.com/gitlab-com/www-gitlab-com.git\n> \n> Thanks for including this information. Just to be clear: did you\n> include patch 4 in your tests, or not? Patch 4 includes a fix\n> related to overflow values, so it would be helpful to know if you\n> found a _different_ bug or if it is the same one.\n> \n> Thanks,\n> -Stolee\n\nI initially only applied the first three patches, but after having hit\nthe fatal error I also applied the rest of this series to have a look at\nwhether it is indeed fixed already by one of your later patches. The\nerror remains the same though.\n\nPatrick\n"},{"id":"449797","messageId":"dbb59fe7-4918-50ef-33a6-79eb430445e8@github.com","threadId":"57472","inReplyTo":"Yhz/hsDDGNVjt64R@xps","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-02-28T18:44:01Z","receivedAt":"2022-02-28T18:45:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/28/2022 11:59 AM, Patrick Steinhardt wrote:\n> On Mon, Feb 28, 2022 at 11:23:38AM -0500, Derrick Stolee wrote:\n>> On 2/28/2022 10:18 AM, Patrick Steinhardt wrote:\n>>> I haven't yet found the time to dig deeper into why this is happening.\n>>> While the repository is publicly accessible at [1], unfortunately the\n>>> bug seems to be triggered by a commit that's only kept alive by an\n>>> internal reference.\n>>>\n>>> Patrick\n>>>\n>>> [1]: https://gitlab.com/gitlab-com/www-gitlab-com.git\n>>\n>> Thanks for including this information. Just to be clear: did you\n>> include patch 4 in your tests, or not? Patch 4 includes a fix\n>> related to overflow values, so it would be helpful to know if you\n>> found a _different_ bug or if it is the same one.\n>>\n>> Thanks,\n>> -Stolee\n> \n> I initially only applied the first three patches, but after having hit\n> the fatal error I also applied the rest of this series to have a look at\n> whether it is indeed fixed already by one of your later patches. The\n> error remains the same though.\n\nThanks for this extra context. Is this a commit-graph that you wrote\nwith the first three patches and then you get an error when reading it?\n\nDo you get the same error when deleting that file and rewriting it with\nall patches included?\n\nThanks,\n-Stolee\n"},{"id":"449812","messageId":"220228.86ilsy3a8b.gmgdl@evledraar.gmail.com","threadId":"57472","inReplyTo":"e74d72bc-b6f5-2e1c-63d1-d3a580f3dc11@github.com","subject":"Re: [PATCH 5/7] commit-graph: document file format v2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-02-28T21:14:16Z","receivedAt":"2022-02-28T22:19:38Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Feb 28 2022, Derrick Stolee wrote:\n\n> On 2/28/2022 9:27 AM, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> On Mon, Feb 28 2022, Derrick Stolee wrote:\n>> \n>>> On 2/25/2022 5:31 PM, Ævar Arnfjörð Bjarmason wrote:\n>\n>>>> Or maybe they won't. I just found it surprising when reviewing this to\n>>>> not find an answer to why that approach wasn't\n>>>> considered.\n>>>\n>>> The point is to create a new format that can be chosen when deployed\n>>> in an environment where older Git versions will not exist (such as\n>>> a Git server). The new version is not chosen by default and instead\n>>> is opt-in through the commitGraph.generationVersion config option.\n>>>\n>>> Perhaps in a year or two we would consider making this the new\n>>> default, but there is no rush to do so.\n>> \n>> Looking into this a bit more I think that in either case this is less of\n>> a big deal after my 43d35618055 (commit-graph write: don't die if the\n>> existing graph is corrupt, 2019-03-25), which came out of some of those\n>> discussions at the time of [1].\n>> \n>> I.e. now a client that only understands version N-1 will warn when\n>> loading it, wheras it's only if a pre-v2.22.0 client (which has that\n>> commit) reads the repository that we'd hard die on it, correct?\n>> \n>> But speaking of hyper-focus. I think that arguably applies to you in\n>> this case when considering the trade-offs of these sorts of format\n>> changes :)\n>> \n>> I.e. you're primarily considering cases of say a git server (presumably\n>> running on GitHub) or another such deployment where it's easy to have\n>> full control over all of your versions \"in the wild\".\n>\n> I'm thinking of servers, yes, but also 99% of clients who only upgrade\n> (or _maybe_ downgrade to a recent, previous version occasionally).\n\n*nod*\n\n>> And thus a three-phase rollout of something like a format change can be\n>> done in a timely and predictable manner.\n>> \n>> But git is used by *a lot* of people in a bunch of different\n>> scenarios. E.g.:\n>> \n>> * A shared (hopefully read-only) NFS mounted by remote \"unmanaged\" clients.\n>> * A tarred-up directory including a .git, which may be transferred to\n>> a machine with a pre-v2.22.0 version.\n>> \n>> Or even softer cases of failure, such as:\n>> \n>> * A cronjob causes an alert/incident somewhere because the server \n>> operator started writing a new version, but forgot about a set\n>> of machines that are still on the old version.\n>\n> It is important to continue supporting these cases, and this change does\n> not cause any issues for them.\n\nThe issues in those cases will range from warnings on older versions\nwhen loading the graph to errors if it's pre-v2.22.0, with the\nperformance benefits v3 placing them out of range of v2-only clients.\n\nI think arguable that's OK/worth it, but it's \"not [any] issues\", no?\n\n> However, this handful of corner cases should not block progress in the\n> main cases.\n\nWhat progress would be blocked?\n\nI'm only talking about whether we choose to consider a \"new graph\" to be an:\n\n \n \n \n\nv.s.:\n\n \n \n\nI.e. the \"progress\" this series is about is in getting the data locality\nwith smaller data with the new content.\n\nBut that's also possible to get with a very low amount of fixed-overhead.\n\nPer the referenced E-Mail an \"empty\" commit-graph file was ~1k bytes in\n2019, I haven't re-checked. In terms of wasted space it's miniscule &\n<1/4 of one FS page on Linux.\n\nI'm not just trying to rehash the same points, I *think* the version\nbump is just an aesthetic choice & we're not getting any performance\ndifference out of that.\n\nBut I'm not sure from the \"block progress\" etc., so maybe I'm still\nmissing something...\n\n>> I think that even if it's less conceptually clean it's worth considering\n>> being over backwards to be kinder to such use-cases, unless it's really\n>> required for other reasons to break such in-the-wild use-cases.\n>> \n>> Or in this case, if it's thought to be worth it to help reviewers decide\n>> by separating the performance improvement aspect from the changed\n>> interaction between new graphs and older clients.\n>> \n>> As a further nit on the proposed end-state here: Do I understand it\n>> correctly that commitGraph.generationVersion=[1|2] (i.e. on current\n>> \"master\") will always result in a file that's compatible with older\n>> versions, since the only thing \"v2\" there controls now is to write the\n>> optional GDAT and GDOV chunks?\n>> \n>> Whereas going from commitGraph.generationVersion=2 to\n>> commitGraph.generationVersion=3 in this series will impact older clients\n>> as noted above, since we're bumping the version (of the file, to 2 if\n>> the config is 3, which as Junio noted is a bit confusing).\n>> \n>> I think if you're set on going down the path of bumping the top-level\n>> version that deserves to be made much clearer in the added\n>> documentation. Right now the only hint to that is a passing mention that\n>> for v3:\n>> \n>> [it] will be incompatible with some old versions of Git\n>> \n>> Which if we're opting for breaking format changes really should note\n>> some of the caveats above, that pre-v2.22.0 hard-dies, and probably\n>> describe \"some old versions of Git\" a bit more clearly.\n>> \n>> It actually means once this gets released \"the git version that was the\n>> latest one you could download yesterday\". Which a reader of the docs\n>> probably won't expect when starting to play with this in mixed-version\n>> environment.\n>> \n>> 1. https://lore.kernel.org/git/87h8acivkh.fsf@evledraar.gmail.com/\n>\n> This documentation could be altered to be specific about versions,\n> but such a specific change makes assumptions of the version that will\n> include it. As of now, the generation number v2 fixes will _probably_\n> get in for 2.36 and the format change would have enough time to cook\n> for 2.37, so I'll update the docs to refer to that version explicitly.\n\n...\n\n> The pre-2.22.0 change might be helpful to mention, but it could also be\n> noise to the reader. We can revisit this when these patches are\n> submitted again in another thread. There's also concern about third-\n> party tools like libgit2. I'd rather draw the line as \"tread carefully\n> here\" than \"here is so much information that a reader might think it\n> is all they need to know\".\n\nIn terms of concern about libgit2 or any other implementation (which I\nhaven't looked at) isn't \"tread carefully\" to do it with new chunks if\npossible, which we've done before with BIDX/BDAT, v.s. a version bump we\nhaven't done?\n\nI'd think it wouldn't be an issue either way for any reader of the\nformat, and libgit2 is more specialized & won't have someone on RHEL6 or\nwhatever trying to inspect a random repo.\n\nIt just seems like a win-win to have a performance improvement with\nsmooth backwards compatibility v.s. without, if that's possible.\n"},{"id":"449901","messageId":"Yh3rZX6cJpkHmRZc@ncase","threadId":"57472","inReplyTo":"dbb59fe7-4918-50ef-33a6-79eb430445e8@github.com","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-03-01T09:46:13Z","receivedAt":"2022-03-01T09:46:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Feb 28, 2022 at 01:44:01PM -0500, Derrick Stolee wrote:\n> On 2/28/2022 11:59 AM, Patrick Steinhardt wrote:\n> > On Mon, Feb 28, 2022 at 11:23:38AM -0500, Derrick Stolee wrote:\n> >> On 2/28/2022 10:18 AM, Patrick Steinhardt wrote:\n> >>> I haven't yet found the time to dig deeper into why this is happening.\n> >>> While the repository is publicly accessible at [1], unfortunately the\n> >>> bug seems to be triggered by a commit that's only kept alive by an\n> >>> internal reference.\n> >>>\n> >>> Patrick\n> >>>\n> >>> [1]: https://gitlab.com/gitlab-com/www-gitlab-com.git\n> >>\n> >> Thanks for including this information. Just to be clear: did you\n> >> include patch 4 in your tests, or not? Patch 4 includes a fix\n> >> related to overflow values, so it would be helpful to know if you\n> >> found a _different_ bug or if it is the same one.\n> >>\n> >> Thanks,\n> >> -Stolee\n> > \n> > I initially only applied the first three patches, but after having hit\n> > the fatal error I also applied the rest of this series to have a look at\n> > whether it is indeed fixed already by one of your later patches. The\n> > error remains the same though.\n> \n> Thanks for this extra context. Is this a commit-graph that you wrote\n> with the first three patches and then you get an error when reading it?\n> \n> Do you get the same error when deleting that file and rewriting it with\n> all patches included?\n> \n> Thanks,\n> -Stolee\n\nYes, I do. I've applied all four patches from v2 on top of 715d08a9e5\n(The eighth batch, 2022-02-25) and still get the same results:\n\n $ find objects/info/commit-graphs/\n objects/info/commit-graphs/\n objects/info/commit-graphs/graph-607e641165f3e83a82d5b14af4e611bf2a688f35.graph\n objects/info/commit-graphs/commit-graph-chain\n objects/info/commit-graphs/graph-5f357c7573c0075d42d82b28e660bc3eac01bfe8.graph\n objects/info/commit-graphs/graph-e0c12ead1b61c7c30720ae372e8a9f98d95dfb2d.graph\n objects/info/commit-graphs/graph-c96723b133c2d81106a01ecd7a8773bb2ef6c2e1.graph\n\n $ git commit-graph verify\n fatal: commit-graph requires overflow generation data but has none\n\n $ git commit-graph write\n Finding commits for commit graph among packed objects: 100% (10235119/10235119), done.\n Expanding reachable commits in commit graph: 2197197, done.\n Finding extra edges in commit graph: 100% (2197197/2197197), done.\n fatal: commit-graph requires overflow generation data but has none\n\n $ rm -rf objects/info/commit-graphs/\n\n $ git commit-graph write\n Finding commits for commit graph among packed objects: 100% (10235119/10235119), done.\n Expanding reachable commits in commit graph: 2197197, done.\n Finding extra edges in commit graph: 100% (2197197/2197197), done.\n fatal: commit-graph requires overflow generation data but has none)\n\nSo even generating them completely anew doesn't seem to generate the\noverflow generation data.\n\nPatrick\n"},{"id":"449919","messageId":"Yh325v3RBDMxjFnD@ncase","threadId":"57472","inReplyTo":"Yh3rZX6cJpkHmRZc@ncase","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-03-01T10:35:18Z","receivedAt":"2022-03-01T10:35:28Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Mar 01, 2022 at 10:46:14AM +0100, Patrick Steinhardt wrote:\n> On Mon, Feb 28, 2022 at 01:44:01PM -0500, Derrick Stolee wrote:\n> > On 2/28/2022 11:59 AM, Patrick Steinhardt wrote:\n> > > On Mon, Feb 28, 2022 at 11:23:38AM -0500, Derrick Stolee wrote:\n> > >> On 2/28/2022 10:18 AM, Patrick Steinhardt wrote:\n> > >>> I haven't yet found the time to dig deeper into why this is happening.\n> > >>> While the repository is publicly accessible at [1], unfortunately the\n> > >>> bug seems to be triggered by a commit that's only kept alive by an\n> > >>> internal reference.\n> > >>>\n> > >>> Patrick\n> > >>>\n> > >>> [1]: https://gitlab.com/gitlab-com/www-gitlab-com.git\n> > >>\n> > >> Thanks for including this information. Just to be clear: did you\n> > >> include patch 4 in your tests, or not? Patch 4 includes a fix\n> > >> related to overflow values, so it would be helpful to know if you\n> > >> found a _different_ bug or if it is the same one.\n> > >>\n> > >> Thanks,\n> > >> -Stolee\n> > > \n> > > I initially only applied the first three patches, but after having hit\n> > > the fatal error I also applied the rest of this series to have a look at\n> > > whether it is indeed fixed already by one of your later patches. The\n> > > error remains the same though.\n> > \n> > Thanks for this extra context. Is this a commit-graph that you wrote\n> > with the first three patches and then you get an error when reading it?\n> > \n> > Do you get the same error when deleting that file and rewriting it with\n> > all patches included?\n> > \n> > Thanks,\n> > -Stolee\n> \n> Yes, I do. I've applied all four patches from v2 on top of 715d08a9e5\n> (The eighth batch, 2022-02-25) and still get the same results:\n> \n> $ find objects/info/commit-graphs/\n> objects/info/commit-graphs/\n> objects/info/commit-graphs/graph-607e641165f3e83a82d5b14af4e611bf2a688f35.graph\n> objects/info/commit-graphs/commit-graph-chain\n> objects/info/commit-graphs/graph-5f357c7573c0075d42d82b28e660bc3eac01bfe8.graph\n> objects/info/commit-graphs/graph-e0c12ead1b61c7c30720ae372e8a9f98d95dfb2d.graph\n> objects/info/commit-graphs/graph-c96723b133c2d81106a01ecd7a8773bb2ef6c2e1.graph\n> \n> $ git commit-graph verify\n> fatal: commit-graph requires overflow generation data but has none\n> \n> $ git commit-graph write\n> Finding commits for commit graph among packed objects: 100% (10235119/10235119), done.\n> Expanding reachable commits in commit graph: 2197197, done.\n> Finding extra edges in commit graph: 100% (2197197/2197197), done.\n> fatal: commit-graph requires overflow generation data but has none\n> \n> $ rm -rf objects/info/commit-graphs/\n> \n> $ git commit-graph write\n> Finding commits for commit graph among packed objects: 100% (10235119/10235119), done.\n> Expanding reachable commits in commit graph: 2197197, done.\n> Finding extra edges in commit graph: 100% (2197197/2197197), done.\n> fatal: commit-graph requires overflow generation data but has none)\n> \n> So even generating them completely anew doesn't seem to generate the\n> overflow generation data.\n> \n> Patrick\n\nI stand corrected. I forgot that the repository at hand was connected to\nanother one via `objects/info/alternates`. If I prune commit-graphs from\nthat alternate, too, then it works alright with your patches.\n\nThis makes me wonder how such a bugfix should be handled though. As this\nseries is right now, users will be faced with repository corruption as\nsoon as they upgrade their Git version to one that contains this patch\nseries. This corruption needs manual action: they have to go into the\nrepository, delete the commit-graphs and then optionally create new\nones.\n\nThis is not a good user experience, and it's worse on the server-side\nwhere we now have a timeframe where all commit-graphs are potentially\ncorrupt. This effectively leads to us being unable to serve those repos\nat all until we have rewritten the commit-graphs because all commands\nwhich make use of the commit-graph will now die:\n\n $ git log\n fatal: commit-graph requires overflow generation data but has none\n\nSo the question is whether this is a change that needs to be rolled out\nover multiple releases. First we'd get in the bug fix such that we write\ncorrect commit-graphs, and after this fix has been released we can also\nrelease the fix that starts to actually parse the generation. This\nensures there's a grace period during which we can hopefully correct the\ndata on-disk such that users are not faced with failures.\n\nThe better alternative would probably be to just gracefully handle\ncommit-graphs which are corrupted in such a way. Can we maybe just\ncontinue to not parse generations in case we find that the commit-graph\ndoesn't have overflow generation data?\n\nThis is more of a general issue though: commit-graphs are an auxiliary\ncache that is not required for proper operation at all. If we fail to\nparse it, then Git shouldn't die but instead fail gracefully just ignore\nit. Furthermore, if we notice that graphs are corrupt when we try to\nwrite new ones, we may just delete the corrupt versions automatically\nand generate completely new ones.\n\nPatrick\n"},{"id":"449924","messageId":"f50e74f0-9ffa-f4f2-4663-269801495ed3@github.com","threadId":"57472","inReplyTo":"Yh325v3RBDMxjFnD@ncase","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-01T14:06:44Z","receivedAt":"2022-03-01T14:06:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/1/2022 5:35 AM, Patrick Steinhardt wrote:\n> On Tue, Mar 01, 2022 at 10:46:14AM +0100, Patrick Steinhardt wrote:\n>> On Mon, Feb 28, 2022 at 01:44:01PM -0500, Derrick Stolee wrote:\n>>> On 2/28/2022 11:59 AM, Patrick Steinhardt wrote:\n>>>> On Mon, Feb 28, 2022 at 11:23:38AM -0500, Derrick Stolee wrote:\n>>>>> On 2/28/2022 10:18 AM, Patrick Steinhardt wrote:\n>>>>>> I haven't yet found the time to dig deeper into why this is happening.\n>>>>>> While the repository is publicly accessible at [1], unfortunately the\n>>>>>> bug seems to be triggered by a commit that's only kept alive by an\n>>>>>> internal reference.\n>>>>>>\n>>>>>> Patrick\n>>>>>>\n>>>>>> [1]: https://gitlab.com/gitlab-com/www-gitlab-com.git\n>>>>>\n>>>>> Thanks for including this information. Just to be clear: did you\n>>>>> include patch 4 in your tests, or not? Patch 4 includes a fix\n>>>>> related to overflow values, so it would be helpful to know if you\n>>>>> found a _different_ bug or if it is the same one.\n>>>>>\n>>>>> Thanks,\n>>>>> -Stolee\n>>>>\n>>>> I initially only applied the first three patches, but after having hit\n>>>> the fatal error I also applied the rest of this series to have a look at\n>>>> whether it is indeed fixed already by one of your later patches. The\n>>>> error remains the same though.\n>>>\n>>> Thanks for this extra context. Is this a commit-graph that you wrote\n>>> with the first three patches and then you get an error when reading it?\n>>>\n>>> Do you get the same error when deleting that file and rewriting it with\n>>> all patches included?\n>>>\n>>> Thanks,\n>>> -Stolee\n>>\n>> Yes, I do. I've applied all four patches from v2 on top of 715d08a9e5\n>> (The eighth batch, 2022-02-25) and still get the same results:\n>>\n>> $ find objects/info/commit-graphs/\n>> objects/info/commit-graphs/\n>> objects/info/commit-graphs/graph-607e641165f3e83a82d5b14af4e611bf2a688f35.graph\n>> objects/info/commit-graphs/commit-graph-chain\n>> objects/info/commit-graphs/graph-5f357c7573c0075d42d82b28e660bc3eac01bfe8.graph\n>> objects/info/commit-graphs/graph-e0c12ead1b61c7c30720ae372e8a9f98d95dfb2d.graph\n>> objects/info/commit-graphs/graph-c96723b133c2d81106a01ecd7a8773bb2ef6c2e1.graph\n>>\n>> $ git commit-graph verify\n>> fatal: commit-graph requires overflow generation data but has none\n>>\n>> $ git commit-graph write\n>> Finding commits for commit graph among packed objects: 100% (10235119/10235119), done.\n>> Expanding reachable commits in commit graph: 2197197, done.\n>> Finding extra edges in commit graph: 100% (2197197/2197197), done.\n>> fatal: commit-graph requires overflow generation data but has none\n>>\n>> $ rm -rf objects/info/commit-graphs/\n>>\n>> $ git commit-graph write\n>> Finding commits for commit graph among packed objects: 100% (10235119/10235119), done.\n>> Expanding reachable commits in commit graph: 2197197, done.\n>> Finding extra edges in commit graph: 100% (2197197/2197197), done.\n>> fatal: commit-graph requires overflow generation data but has none)\n>>\n>> So even generating them completely anew doesn't seem to generate the\n>> overflow generation data.\n>>\n>> Patrick\n> \n> I stand corrected. I forgot that the repository at hand was connected to\n> another one via `objects/info/alternates`. If I prune commit-graphs from\n> that alternate, too, then it works alright with your patches.\n\nOK, thanks. That clarifies the situation.\n\nI ordered the patches such that the fix in patch 4 could be immediately\ntestable, which is not the case without patch 3. However, it does leave\nthis temporary state where information can be incorrect if only a subset\nof the series is applied.\n\n> This makes me wonder how such a bugfix should be handled though. As this\n> series is right now, users will be faced with repository corruption as\n> soon as they upgrade their Git version to one that contains this patch\n> series. This corruption needs manual action: they have to go into the\n> repository, delete the commit-graphs and then optionally create new\n> ones.\n> \n> This is not a good user experience, and it's worse on the server-side\n> where we now have a timeframe where all commit-graphs are potentially\n> corrupt. This effectively leads to us being unable to serve those repos\n> at all until we have rewritten the commit-graphs because all commands\n> which make use of the commit-graph will now die:\n> \n> $ git log\n> fatal: commit-graph requires overflow generation data but has none\n> \n> So the question is whether this is a change that needs to be rolled out\n> over multiple releases. First we'd get in the bug fix such that we write\n> correct commit-graphs, and after this fix has been released we can also\n> release the fix that starts to actually parse the generation. This\n> ensures there's a grace period during which we can hopefully correct the\n> data on-disk such that users are not faced with failures.\n\nYou are right that we need to be careful here, but I also think that\nprevious versions of Git always wrote the correct data. Here is my\nthought process:\n\n1. To get this bug, we need to have parsed the corrected commit date\n from an existing commit-graph in order to under-count the number\n of overflow values.\n\n2. Before this series, Git versions were not parsing the corrected\n commit date, so they recompute the corrected commit date every\n time the commit-graph is written, getting the proper count of\n overflow values.\n\nFor these reasons, data written by previous versions of Git are\ncorrect and can be trusted without a staged release.\n\nDoes this make sense? Or, do you experience a different result when\nyou build commit-graphs with a released Git version and then when\nwriting on top with all patches applied?\n\n> The better alternative would probably be to just gracefully handle\n> commit-graphs which are corrupted in such a way. Can we maybe just\n> continue to not parse generations in case we find that the commit-graph\n> doesn't have overflow generation data?\n>\n> This is more of a general issue though: commit-graphs are an auxiliary\n> cache that is not required for proper operation at all. If we fail to\n> parse it, then Git shouldn't die but instead fail gracefully just ignore\n> it. Furthermore, if we notice that graphs are corrupt when we try to\n> write new ones, we may just delete the corrupt versions automatically\n> and generate completely new ones.\n\nYou are right that we can be better about failures here and report\nand error instead of a die(). Especially in this case, we could just\nrevert to topological levels instead of throwing out the commit-graph\nentirely.\n\nThis seems like something for another series, so we can be sure to\naudit all cases of fatal errors when parsing the commit-graph so we\ncatch all of them and do the \"best\" thing in each case.\n\nThanks,\n-Stolee\n"},{"id":"449925","messageId":"9b52fdd3-64fc-34b1-d4ee-660b4fb73f39@github.com","threadId":"57472","inReplyTo":"220228.86ilsy3a8b.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 5/7] commit-graph: document file format v2","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-01T14:19:06Z","receivedAt":"2022-03-01T14:19:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/28/2022 4:14 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Feb 28 2022, Derrick Stolee wrote:\n> \n>> On 2/28/2022 9:27 AM, Ævar Arnfjörð Bjarmason wrote:\n>>>\n>>> On Mon, Feb 28 2022, Derrick Stolee wrote:\n>>>\n>>>> On 2/25/2022 5:31 PM, Ævar Arnfjörð Bjarmason wrote:\n>>\n>>>>> Or maybe they won't. I just found it surprising when reviewing this to\n>>>>> not find an answer to why that approach wasn't\n>>>>> considered.\n>>>>\n>>>> The point is to create a new format that can be chosen when deployed\n>>>> in an environment where older Git versions will not exist (such as\n>>>> a Git server). The new version is not chosen by default and instead\n>>>> is opt-in through the commitGraph.generationVersion config option.\n>>>>\n>>>> Perhaps in a year or two we would consider making this the new\n>>>> default, but there is no rush to do so.\n>>>\n>>> Looking into this a bit more I think that in either case this is less of\n>>> a big deal after my 43d35618055 (commit-graph write: don't die if the\n>>> existing graph is corrupt, 2019-03-25), which came out of some of those\n>>> discussions at the time of [1].\n>>>\n>>> I.e. now a client that only understands version N-1 will warn when\n>>> loading it, wheras it's only if a pre-v2.22.0 client (which has that\n>>> commit) reads the repository that we'd hard die on it, correct?\n>>>\n>>> But speaking of hyper-focus. I think that arguably applies to you in\n>>> this case when considering the trade-offs of these sorts of format\n>>> changes :)\n>>>\n>>> I.e. you're primarily considering cases of say a git server (presumably\n>>> running on GitHub) or another such deployment where it's easy to have\n>>> full control over all of your versions \"in the wild\".\n>>\n>> I'm thinking of servers, yes, but also 99% of clients who only upgrade\n>> (or _maybe_ downgrade to a recent, previous version occasionally).\n> \n> *nod*\n> \n>>> And thus a three-phase rollout of something like a format change can be\n>>> done in a timely and predictable manner.\n>>>\n>>> But git is used by *a lot* of people in a bunch of different\n>>> scenarios. E.g.:\n>>>\n>>> * A shared (hopefully read-only) NFS mounted by remote \"unmanaged\" clients.\n>>> * A tarred-up directory including a .git, which may be transferred to\n>>> a machine with a pre-v2.22.0 version.\n>>>\n>>> Or even softer cases of failure, such as:\n>>>\n>>> * A cronjob causes an alert/incident somewhere because the server \n>>> operator started writing a new version, but forgot about a set\n>>> of machines that are still on the old version.\n>>\n>> It is important to continue supporting these cases, and this change does\n>> not cause any issues for them.\n> \n> The issues in those cases will range from warnings on older versions\n> when loading the graph to errors if it's pre-v2.22.0, with the\n> performance benefits v3 placing them out of range of v2-only clients.\n> \n> I think arguable that's OK/worth it, but it's \"not [any] issues\", no?\n\nWhat I mean is that this change does not enable the new graph version\nby default, so these users do not have any issues unless someone opts\nin to the feature while in this mixed scenario.\n\n>> However, this handful of corner cases should not block progress in the\n>> main cases.\n> \n> What progress would be blocked?\n> \n> I'm only talking about whether we choose to consider a \"new graph\" to be an:\n> \n> \n> \n> \n> \n> v.s.:\n> \n> \n> \n> \n> I.e. the \"progress\" this series is about is in getting the data locality\n> with smaller data with the new content.\n> \n> But that's also possible to get with a very low amount of fixed-overhead.\n> \n> Per the referenced E-Mail an \"empty\" commit-graph file was ~1k bytes in\n> 2019, I haven't re-checked. In terms of wasted space it's miniscule &\n> <1/4 of one FS page on Linux.\n\nIf you're talking \"empty\" data, then you need to have an empty Commit\nData chunk _and_ and empty OID Lookup chunk in order to not have\nbreakage. So you'd need duplicate versions of these chunks for the\nnew \"Commit Data 2\" chunk. Then we need special-casing for all of this\nduring parsing that is unnecessary complexity.\n\nFinally, the end result becomes \"older versions get slower without\nany warning\" instead of \"older versions get a message about not\nunderstanding the commit-graph file\".\n \n> I'm not just trying to rehash the same points, I *think* the version\n> bump is just an aesthetic choice & we're not getting any performance\n> difference out of that.\n> \n> But I'm not sure from the \"block progress\" etc., so maybe I'm still\n> missing something...\n\nThe fact that we have a Generation Data chunk instead of already\nbumping the file format version number is already a concession to\nthis concern about backwards compatibility.\n\nWith the point above about empty Commit Data Chunks, the only way\nto properly conserve backwards compatibility is to have a full\nCommit Data Chunk as well as a second copy that contains the new\noffsets instead of topological levels. This is wasteful.\n\n>>> I think that even if it's less conceptually clean it's worth considering\n>>> being over backwards to be kinder to such use-cases, unless it's really\n>>> required for other reasons to break such in-the-wild use-cases.\n>>>\n>>> Or in this case, if it's thought to be worth it to help reviewers decide\n>>> by separating the performance improvement aspect from the changed\n>>> interaction between new graphs and older clients.\n>>>\n>>> As a further nit on the proposed end-state here: Do I understand it\n>>> correctly that commitGraph.generationVersion=[1|2] (i.e. on current\n>>> \"master\") will always result in a file that's compatible with older\n>>> versions, since the only thing \"v2\" there controls now is to write the\n>>> optional GDAT and GDOV chunks?\n>>>\n>>> Whereas going from commitGraph.generationVersion=2 to\n>>> commitGraph.generationVersion=3 in this series will impact older clients\n>>> as noted above, since we're bumping the version (of the file, to 2 if\n>>> the config is 3, which as Junio noted is a bit confusing).\n>>>\n>>> I think if you're set on going down the path of bumping the top-level\n>>> version that deserves to be made much clearer in the added\n>>> documentation. Right now the only hint to that is a passing mention that\n>>> for v3:\n>>>\n>>> [it] will be incompatible with some old versions of Git\n>>>\n>>> Which if we're opting for breaking format changes really should note\n>>> some of the caveats above, that pre-v2.22.0 hard-dies, and probably\n>>> describe \"some old versions of Git\" a bit more clearly.\n>>>\n>>> It actually means once this gets released \"the git version that was the\n>>> latest one you could download yesterday\". Which a reader of the docs\n>>> probably won't expect when starting to play with this in mixed-version\n>>> environment.\n>>>\n>>> 1. https://lore.kernel.org/git/87h8acivkh.fsf@evledraar.gmail.com/\n>>\n>> This documentation could be altered to be specific about versions,\n>> but such a specific change makes assumptions of the version that will\n>> include it. As of now, the generation number v2 fixes will _probably_\n>> get in for 2.36 and the format change would have enough time to cook\n>> for 2.37, so I'll update the docs to refer to that version explicitly.\n> \n> ...\n> \n>> The pre-2.22.0 change might be helpful to mention, but it could also be\n>> noise to the reader. We can revisit this when these patches are\n>> submitted again in another thread. There's also concern about third-\n>> party tools like libgit2. I'd rather draw the line as \"tread carefully\n>> here\" than \"here is so much information that a reader might think it\n>> is all they need to know\".\n> \n> In terms of concern about libgit2 or any other implementation (which I\n> haven't looked at) isn't \"tread carefully\" to do it with new chunks if\n> possible, which we've done before with BIDX/BDAT, v.s. a version bump we\n> haven't done?\n\nNew chunks adding new information is part of the design. Changing\nthe location of existing data is new here.\n\n> I'd think it wouldn't be an issue either way for any reader of the\n> format, and libgit2 is more specialized & won't have someone on RHEL6 or\n> whatever trying to inspect a random repo.\n> \n> It just seems like a win-win to have a performance improvement with\n> smooth backwards compatibility v.s. without, if that's possible.\n\nYou are right that it is _possible_, but I don't think that the\nside-effects are worth it. Those being:\n\n* \"Empty CDAT Chunk\": Silently slowing down older clients.\n* \"Duplicate CDAT Chunk\": Wasted data.\n\nFinally, I want to reiterate that by making this opt-in, users make\nthe call about whether or not they are in a scenario where this\ncompatibility issue is appropriate for them. This includes waiting\nto see if third-party tools like libgit2 are updated to understand\nthis version.\n\nThanks,\n-Stolee\n"},{"id":"449927","messageId":"Yh4zehdSnHLW1HuK@ncase","threadId":"57472","inReplyTo":"f50e74f0-9ffa-f4f2-4663-269801495ed3@github.com","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-03-01T14:53:46Z","receivedAt":"2022-03-01T14:54:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Mar 01, 2022 at 09:06:44AM -0500, Derrick Stolee wrote:\n> On 3/1/2022 5:35 AM, Patrick Steinhardt wrote:\n> > On Tue, Mar 01, 2022 at 10:46:14AM +0100, Patrick Steinhardt wrote:\n> >> On Mon, Feb 28, 2022 at 01:44:01PM -0500, Derrick Stolee wrote:\n> >>> On 2/28/2022 11:59 AM, Patrick Steinhardt wrote:\n> >>>> On Mon, Feb 28, 2022 at 11:23:38AM -0500, Derrick Stolee wrote:\n> >>>>> On 2/28/2022 10:18 AM, Patrick Steinhardt wrote:\n> >>>>>> I haven't yet found the time to dig deeper into why this is happening.\n> >>>>>> While the repository is publicly accessible at [1], unfortunately the\n> >>>>>> bug seems to be triggered by a commit that's only kept alive by an\n> >>>>>> internal reference.\n> >>>>>>\n> >>>>>> Patrick\n> >>>>>>\n> >>>>>> [1]: https://gitlab.com/gitlab-com/www-gitlab-com.git\n> >>>>>\n> >>>>> Thanks for including this information. Just to be clear: did you\n> >>>>> include patch 4 in your tests, or not? Patch 4 includes a fix\n> >>>>> related to overflow values, so it would be helpful to know if you\n> >>>>> found a _different_ bug or if it is the same one.\n> >>>>>\n> >>>>> Thanks,\n> >>>>> -Stolee\n> >>>>\n> >>>> I initially only applied the first three patches, but after having hit\n> >>>> the fatal error I also applied the rest of this series to have a look at\n> >>>> whether it is indeed fixed already by one of your later patches. The\n> >>>> error remains the same though.\n> >>>\n> >>> Thanks for this extra context. Is this a commit-graph that you wrote\n> >>> with the first three patches and then you get an error when reading it?\n> >>>\n> >>> Do you get the same error when deleting that file and rewriting it with\n> >>> all patches included?\n> >>>\n> >>> Thanks,\n> >>> -Stolee\n> >>\n> >> Yes, I do. I've applied all four patches from v2 on top of 715d08a9e5\n> >> (The eighth batch, 2022-02-25) and still get the same results:\n> >>\n> >> $ find objects/info/commit-graphs/\n> >> objects/info/commit-graphs/\n> >> objects/info/commit-graphs/graph-607e641165f3e83a82d5b14af4e611bf2a688f35.graph\n> >> objects/info/commit-graphs/commit-graph-chain\n> >> objects/info/commit-graphs/graph-5f357c7573c0075d42d82b28e660bc3eac01bfe8.graph\n> >> objects/info/commit-graphs/graph-e0c12ead1b61c7c30720ae372e8a9f98d95dfb2d.graph\n> >> objects/info/commit-graphs/graph-c96723b133c2d81106a01ecd7a8773bb2ef6c2e1.graph\n> >>\n> >> $ git commit-graph verify\n> >> fatal: commit-graph requires overflow generation data but has none\n> >>\n> >> $ git commit-graph write\n> >> Finding commits for commit graph among packed objects: 100% (10235119/10235119), done.\n> >> Expanding reachable commits in commit graph: 2197197, done.\n> >> Finding extra edges in commit graph: 100% (2197197/2197197), done.\n> >> fatal: commit-graph requires overflow generation data but has none\n> >>\n> >> $ rm -rf objects/info/commit-graphs/\n> >>\n> >> $ git commit-graph write\n> >> Finding commits for commit graph among packed objects: 100% (10235119/10235119), done.\n> >> Expanding reachable commits in commit graph: 2197197, done.\n> >> Finding extra edges in commit graph: 100% (2197197/2197197), done.\n> >> fatal: commit-graph requires overflow generation data but has none)\n> >>\n> >> So even generating them completely anew doesn't seem to generate the\n> >> overflow generation data.\n> >>\n> >> Patrick\n> > \n> > I stand corrected. I forgot that the repository at hand was connected to\n> > another one via `objects/info/alternates`. If I prune commit-graphs from\n> > that alternate, too, then it works alright with your patches.\n> \n> OK, thanks. That clarifies the situation.\n> \n> I ordered the patches such that the fix in patch 4 could be immediately\n> testable, which is not the case without patch 3. However, it does leave\n> this temporary state where information can be incorrect if only a subset\n> of the series is applied.\n> \n> > This makes me wonder how such a bugfix should be handled though. As this\n> > series is right now, users will be faced with repository corruption as\n> > soon as they upgrade their Git version to one that contains this patch\n> > series. This corruption needs manual action: they have to go into the\n> > repository, delete the commit-graphs and then optionally create new\n> > ones.\n> > \n> > This is not a good user experience, and it's worse on the server-side\n> > where we now have a timeframe where all commit-graphs are potentially\n> > corrupt. This effectively leads to us being unable to serve those repos\n> > at all until we have rewritten the commit-graphs because all commands\n> > which make use of the commit-graph will now die:\n> > \n> > $ git log\n> > fatal: commit-graph requires overflow generation data but has none\n> > \n> > So the question is whether this is a change that needs to be rolled out\n> > over multiple releases. First we'd get in the bug fix such that we write\n> > correct commit-graphs, and after this fix has been released we can also\n> > release the fix that starts to actually parse the generation. This\n> > ensures there's a grace period during which we can hopefully correct the\n> > data on-disk such that users are not faced with failures.\n> \n> You are right that we need to be careful here, but I also think that\n> previous versions of Git always wrote the correct data. Here is my\n> thought process:\n> \n> 1. To get this bug, we need to have parsed the corrected commit date\n> from an existing commit-graph in order to under-count the number\n> of overflow values.\n> \n> 2. Before this series, Git versions were not parsing the corrected\n> commit date, so they recompute the corrected commit date every\n> time the commit-graph is written, getting the proper count of\n> overflow values.\n> \n> For these reasons, data written by previous versions of Git are\n> correct and can be trusted without a staged release.\n> \n> Does this make sense? Or, do you experience a different result when\n> you build commit-graphs with a released Git version and then when\n> writing on top with all patches applied?\n\nJust to verify my understanding: you claim that the bug I was hitting\nshouldn't be encountered in the wild when the release , but\nonly if one were to write a commit-graph with the intermediate stafe\nuntil patch 3/4 of your patch series?\n\nHum. I have re-verified, and this indeed seems to play out. So I must've\naccidentally ran all my testing with the state generated without the\nfinal patch which fixes the corruption. I do see lots of the following\nwarnings, but overall I can verify and write the commit-graph just fine:\n\n commit-graph generation for commit c80a42de8803e2d77818d0c82f88e748d7f9425f is 1623362063 < 1623362139\n\nThanks for your patience, and sorry for the noise :)\n\n> > The better alternative would probably be to just gracefully handle\n> > commit-graphs which are corrupted in such a way. Can we maybe just\n> > continue to not parse generations in case we find that the commit-graph\n> > doesn't have overflow generation data?\n> >\n> > This is more of a general issue though: commit-graphs are an auxiliary\n> > cache that is not required for proper operation at all. If we fail to\n> > parse it, then Git shouldn't die but instead fail gracefully just ignore\n> > it. Furthermore, if we notice that graphs are corrupt when we try to\n> > write new ones, we may just delete the corrupt versions automatically\n> > and generate completely new ones.\n> \n> You are right that we can be better about failures here and report\n> and error instead of a die(). Especially in this case, we could just\n> revert to topological levels instead of throwing out the commit-graph\n> entirely.\n> \n> This seems like something for another series, so we can be sure to\n> audit all cases of fatal errors when parsing the commit-graph so we\n> catch all of them and do the \"best\" thing in each case.\n\nI agree.\n\nPatrick\n"},{"id":"449931","messageId":"1b9912f7-87be-2520-bb53-9e23529ad233@github.com","threadId":"57472","inReplyTo":"Yh4zehdSnHLW1HuK@ncase","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-01T15:25:46Z","receivedAt":"2022-03-01T15:26:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/1/2022 9:53 AM, Patrick Steinhardt wrote:\n> On Tue, Mar 01, 2022 at 09:06:44AM -0500, Derrick Stolee wrote:\n>> On 3/1/2022 5:35 AM, Patrick Steinhardt wrote:\n>>> On Tue, Mar 01, 2022 at 10:46:14AM +0100, Patrick Steinhardt wrote:\n>>>> On Mon, Feb 28, 2022 at 01:44:01PM -0500, Derrick Stolee wrote:\n>>>>> On 2/28/2022 11:59 AM, Patrick Steinhardt wrote:\n>>>>>> On Mon, Feb 28, 2022 at 11:23:38AM -0500, Derrick Stolee wrote:\n>>>>>>> On 2/28/2022 10:18 AM, Patrick Steinhardt wrote:\n>>>>>>>> [1]: https://gitlab.com/gitlab-com/www-gitlab-com.git\n...\n>>> So the question is whether this is a change that needs to be rolled out\n>>> over multiple releases. First we'd get in the bug fix such that we write\n>>> correct commit-graphs, and after this fix has been released we can also\n>>> release the fix that starts to actually parse the generation. This\n>>> ensures there's a grace period during which we can hopefully correct the\n>>> data on-disk such that users are not faced with failures.\n>>\n>> You are right that we need to be careful here, but I also think that\n>> previous versions of Git always wrote the correct data. Here is my\n>> thought process:\n>>\n>> 1. To get this bug, we need to have parsed the corrected commit date\n>> from an existing commit-graph in order to under-count the number\n>> of overflow values.\n>>\n>> 2. Before this series, Git versions were not parsing the corrected\n>> commit date, so they recompute the corrected commit date every\n>> time the commit-graph is written, getting the proper count of\n>> overflow values.\n>>\n>> For these reasons, data written by previous versions of Git are\n>> correct and can be trusted without a staged release.\n>>\n>> Does this make sense? Or, do you experience a different result when\n>> you build commit-graphs with a released Git version and then when\n>> writing on top with all patches applied?\n> \n> Just to verify my understanding: you claim that the bug I was hitting\n> shouldn't be encountered in the wild when the release , but\n> only if one were to write a commit-graph with the intermediate stafe\n> until patch 3/4 of your patch series?\n\nThat is my claim. And my testing of the repo at [1] has demonstrated\nthat it works correctly in these cases.\n \n> Hum. I have re-verified, and this indeed seems to play out. So I must've\n> accidentally ran all my testing with the state generated without the\n> final patch which fixes the corruption. I do see lots of the following\n> warnings, but overall I can verify and write the commit-graph just fine:\n> \n> commit-graph generation for commit c80a42de8803e2d77818d0c82f88e748d7f9425f is 1623362063 < 1623362139\n\nBut I'm not able to generate these warnings from either version. I\ntried generating different levels of a split commit-graph, but\ncould not reproduce it. If you have reproduction steps using current\n'master' (or any released Git version) and the four patches here,\nthen I would love to get a full understanding of your errors.\n\nThanks,\n-Stolee\n"},{"id":"449932","messageId":"220301.86y21tznw3.gmgdl@evledraar.gmail.com","threadId":"57472","inReplyTo":"9b52fdd3-64fc-34b1-d4ee-660b4fb73f39@github.com","subject":"Re: [PATCH 5/7] commit-graph: document file format v2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-01T14:29:10Z","receivedAt":"2022-03-01T15:35:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Mar 01 2022, Derrick Stolee wrote:\n\n> On 2/28/2022 4:14 PM, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> On Mon, Feb 28 2022, Derrick Stolee wrote:\n>> \n>>> On 2/28/2022 9:27 AM, Ævar Arnfjörð Bjarmason wrote:\n>>>>\n>>>> On Mon, Feb 28 2022, Derrick Stolee wrote:\n>>>>\n>>>>> On 2/25/2022 5:31 PM, Ævar Arnfjörð Bjarmason wrote:\n>>>\n>>>>>> Or maybe they won't. I just found it surprising when reviewing this to\n>>>>>> not find an answer to why that approach wasn't\n>>>>>> considered.\n>>>>>\n>>>>> The point is to create a new format that can be chosen when deployed\n>>>>> in an environment where older Git versions will not exist (such as\n>>>>> a Git server). The new version is not chosen by default and instead\n>>>>> is opt-in through the commitGraph.generationVersion config option.\n>>>>>\n>>>>> Perhaps in a year or two we would consider making this the new\n>>>>> default, but there is no rush to do so.\n>>>>\n>>>> Looking into this a bit more I think that in either case this is less of\n>>>> a big deal after my 43d35618055 (commit-graph write: don't die if the\n>>>> existing graph is corrupt, 2019-03-25), which came out of some of those\n>>>> discussions at the time of [1].\n>>>>\n>>>> I.e. now a client that only understands version N-1 will warn when\n>>>> loading it, wheras it's only if a pre-v2.22.0 client (which has that\n>>>> commit) reads the repository that we'd hard die on it, correct?\n>>>>\n>>>> But speaking of hyper-focus. I think that arguably applies to you in\n>>>> this case when considering the trade-offs of these sorts of format\n>>>> changes :)\n>>>>\n>>>> I.e. you're primarily considering cases of say a git server (presumably\n>>>> running on GitHub) or another such deployment where it's easy to have\n>>>> full control over all of your versions \"in the wild\".\n>>>\n>>> I'm thinking of servers, yes, but also 99% of clients who only upgrade\n>>> (or _maybe_ downgrade to a recent, previous version occasionally).\n>> \n>> *nod*\n>> \n>>>> And thus a three-phase rollout of something like a format change can be\n>>>> done in a timely and predictable manner.\n>>>>\n>>>> But git is used by *a lot* of people in a bunch of different\n>>>> scenarios. E.g.:\n>>>>\n>>>> * A shared (hopefully read-only) NFS mounted by remote \"unmanaged\" clients.\n>>>> * A tarred-up directory including a .git, which may be transferred to\n>>>> a machine with a pre-v2.22.0 version.\n>>>>\n>>>> Or even softer cases of failure, such as:\n>>>>\n>>>> * A cronjob causes an alert/incident somewhere because the server \n>>>> operator started writing a new version, but forgot about a set\n>>>> of machines that are still on the old version.\n>>>\n>>> It is important to continue supporting these cases, and this change does\n>>> not cause any issues for them.\n>> \n>> The issues in those cases will range from warnings on older versions\n>> when loading the graph to errors if it's pre-v2.22.0, with the\n>> performance benefits v3 placing them out of range of v2-only clients.\n>> \n>> I think arguable that's OK/worth it, but it's \"not [any] issues\", no?\n>\n> What I mean is that this change does not enable the new graph version\n> by default, so these users do not have any issues unless someone opts\n> in to the feature while in this mixed scenario.\n\nIndeed. FWIW I wasn't confused about that bit. I'm just commenting on\n/how/ we do version upgrades, and if we can save users unnecessary\nhassle relatively easily.\n\nBut I also think the writing is on the wall that you'll want to\n(understandably) bump the default sooner than later, or if not for this\ndata for other future chunks.\n\n>>> However, this handful of corner cases should not block progress in the\n>>> main cases.\n>> \n>> What progress would be blocked?\n>> \n>> I'm only talking about whether we choose to consider a \"new graph\" to be an:\n>> \n>> \n>> \n>> \n>> \n>> v.s.:\n>> \n>> \n>> \n>> \n>> I.e. the \"progress\" this series is about is in getting the data locality\n>> with smaller data with the new content.\n>> \n>> But that's also possible to get with a very low amount of fixed-overhead.\n>> \n>> Per the referenced E-Mail an \"empty\" commit-graph file was ~1k bytes in\n>> 2019, I haven't re-checked. In terms of wasted space it's miniscule &\n>> <1/4 of one FS page on Linux.\n>\n> If you're talking \"empty\" data, then you need to have an empty Commit\n> Data chunk _and_ and empty OID Lookup chunk in order to not have\n> breakage. So you'd need duplicate versions of these chunks for the\n> new \"Commit Data 2\" chunk. Then we need special-casing for all of this\n> during parsing that is unnecessary complexity.\n\nWhy does it need to be special-cased? Don't we just call pair_chunk() on\nthe new chunk name, and if it doesn't exist fall back on the old\nchunk. We'll then note what format we're parsing, just as this series\ndoes.\n\n> Finally, the end result becomes \"older versions get slower without\n> any warning\" instead of \"older versions get a message about not\n> understanding the commit-graph file\".\n\nSure, IF you want to write such an empty chunk. The point is that you\nnow have the option.\n\nAnd this is the same edge case we already dealt with for\nGDAT/GDOV. I.e. older readers who didn't understand it would be slower.\n\nWe can still have a feature to make older clients intentionally\nbreak/warn, it seems to me that if you'd want such a thing you'd want it\naside from the specific mechanism of this proposed upgrade.\n\nOr you could dual-write the data for older clients, which I think\nprobably isn't worth the hassle.\n\nI.e. if you're worried about silent slowdown older clients happily\nignoring the BIDX and BDAT chunks are silently slower.\n\n>> I'm not just trying to rehash the same points, I *think* the version\n>> bump is just an aesthetic choice & we're not getting any performance\n>> difference out of that.\n>> \n>> But I'm not sure from the \"block progress\" etc., so maybe I'm still\n>> missing something...\n>\n> The fact that we have a Generation Data chunk instead of already\n> bumping the file format version number is already a concession to\n> this concern about backwards compatibility.\n\nSure, but not taking that version bumping route in the past shouldn't\nbias us towards doing it now, should it?\n\n> With the point above about empty Commit Data Chunks, the only way\n> to properly conserve backwards compatibility is to have a full\n> Commit Data Chunk as well as a second copy that contains the new\n> offsets instead of topological levels. This is wasteful.\n\nEmpty chunks would be a handful of bytes, and not produce those\nerrors/warnings, and AFAICT without any downsides.\n\nBut that's me assuming that the overlap between people for whom the\ncommit-graph is critical for performance and those using wildly\ndifferent versions is pretty much zero.\n\nThe reason I mentioned it at all initially in\nhttps://lore.kernel.org/git/220228.86pmn73toq.gmgdl@evledraar.gmail.com/\nwas in reference to trying to understand the context of the performance\ngains, i.e. whether they'd be equivalent with a new chunk or dual-write\ndata.\n\n>>>> I think that even if it's less conceptually clean it's worth considering\n>>>> being over backwards to be kinder to such use-cases, unless it's really\n>>>> required for other reasons to break such in-the-wild use-cases.\n>>>>\n>>>> Or in this case, if it's thought to be worth it to help reviewers decide\n>>>> by separating the performance improvement aspect from the changed\n>>>> interaction between new graphs and older clients.\n>>>>\n>>>> As a further nit on the proposed end-state here: Do I understand it\n>>>> correctly that commitGraph.generationVersion=[1|2] (i.e. on current\n>>>> \"master\") will always result in a file that's compatible with older\n>>>> versions, since the only thing \"v2\" there controls now is to write the\n>>>> optional GDAT and GDOV chunks?\n>>>>\n>>>> Whereas going from commitGraph.generationVersion=2 to\n>>>> commitGraph.generationVersion=3 in this series will impact older clients\n>>>> as noted above, since we're bumping the version (of the file, to 2 if\n>>>> the config is 3, which as Junio noted is a bit confusing).\n>>>>\n>>>> I think if you're set on going down the path of bumping the top-level\n>>>> version that deserves to be made much clearer in the added\n>>>> documentation. Right now the only hint to that is a passing mention that\n>>>> for v3:\n>>>>\n>>>> [it] will be incompatible with some old versions of Git\n>>>>\n>>>> Which if we're opting for breaking format changes really should note\n>>>> some of the caveats above, that pre-v2.22.0 hard-dies, and probably\n>>>> describe \"some old versions of Git\" a bit more clearly.\n>>>>\n>>>> It actually means once this gets released \"the git version that was the\n>>>> latest one you could download yesterday\". Which a reader of the docs\n>>>> probably won't expect when starting to play with this in mixed-version\n>>>> environment.\n>>>>\n>>>> 1. https://lore.kernel.org/git/87h8acivkh.fsf@evledraar.gmail.com/\n>>>\n>>> This documentation could be altered to be specific about versions,\n>>> but such a specific change makes assumptions of the version that will\n>>> include it. As of now, the generation number v2 fixes will _probably_\n>>> get in for 2.36 and the format change would have enough time to cook\n>>> for 2.37, so I'll update the docs to refer to that version explicitly.\n>> \n>> ...\n>> \n>>> The pre-2.22.0 change might be helpful to mention, but it could also be\n>>> noise to the reader. We can revisit this when these patches are\n>>> submitted again in another thread. There's also concern about third-\n>>> party tools like libgit2. I'd rather draw the line as \"tread carefully\n>>> here\" than \"here is so much information that a reader might think it\n>>> is all they need to know\".\n>> \n>> In terms of concern about libgit2 or any other implementation (which I\n>> haven't looked at) isn't \"tread carefully\" to do it with new chunks if\n>> possible, which we've done before with BIDX/BDAT, v.s. a version bump we\n>> haven't done?\n>\n> New chunks adding new information is part of the design. Changing\n> the location of existing data is new here.\n\nWe've never bumped the top-level version number, and hard dying on \"git\nstatus\" or whatever was also part of the initial design :)\n\nI think it's legitimate to ask/argue that these version number bumps are\nsomething we should be reserving for truly incompatible format bumps,\nv.s. \"new indexes\".\n\nI.e. this case is similar to us having a SQL database with N tables, and\nwe'd like to add a new table or index.\n\nWe could have a central \"schema_version\", or we could just add a new\ntable in a backwards-compatiable way. Older clients read older\ndata/tables, which is possibly empty.\n\nThe commit-graph is essentially such a key-value store when it comes to\ntop-level chunks, and we're already making decisions on what data to\nload/use based on chunk existence.\n\n>> I'd think it wouldn't be an issue either way for any reader of the\n>> format, and libgit2 is more specialized & won't have someone on RHEL6 or\n>> whatever trying to inspect a random repo.\n>> \n>> It just seems like a win-win to have a performance improvement with\n>> smooth backwards compatibility v.s. without, if that's possible.\n>\n> You are right that it is _possible_, but I don't think that the\n> side-effects are worth it. Those being:\n>\n> * \"Empty CDAT Chunk\": Silently slowing down older clients.\n> * \"Duplicate CDAT Chunk\": Wasted data.\n>\n> Finally, I want to reiterate that by making this opt-in, users make\n> the call about whether or not they are in a scenario where this\n> compatibility issue is appropriate for them. This includes waiting\n> to see if third-party tools like libgit2 are updated to understand\n> this version.\n\nI think we're probably not getting any further here & this back & forth,\nit's certainly been interesting, and thanks a lot for your patience and\ntime.\n\nI'll try to look at some of this once the \"prep\" patches land.\n\nIf you'll end up doing this via the version route I'm not going to\nstrongly object to it or anything, I was just trying to review this to\nsee if I understood the trade-offs & constraints involved. In particular\nwith reference to those changes in 2019 that I did to make\nformat/corruption/version transitions non-fatal.\n\nThanks!\n"},{"id":"449934","messageId":"8860432e-9440-2f6b-3647-550f908c893d@github.com","threadId":"57472","inReplyTo":"220301.86y21tznw3.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH 5/7] commit-graph: document file format v2","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-01T15:59:51Z","receivedAt":"2022-03-01T15:59:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/1/2022 9:29 AM, Ævar Arnfjörð Bjarmason wrote:\n\nI agree that this discussion has mostly run its course and I'll do my\nbest to summarize it in the commit messages of a future patch series.\n\nI just wanted to focus on two things in the most-recent reply:\n\n> On Tue, Mar 01 2022, Derrick Stolee wrote:\n> \n>> On 2/28/2022 4:14 PM, Ævar Arnfjörð Bjarmason wrote:\n>>>\n>>> On Mon, Feb 28 2022, Derrick Stolee wrote:\n>>> I think arguable that's OK/worth it, but it's \"not [any] issues\", no?\n>>\n>> What I mean is that this change does not enable the new graph version\n>> by default, so these users do not have any issues unless someone opts\n>> in to the feature while in this mixed scenario.\n> \n> Indeed. FWIW I wasn't confused about that bit. I'm just commenting on\n> /how/ we do version upgrades, and if we can save users unnecessary\n> hassle relatively easily.\n> \n> But I also think the writing is on the wall that you'll want to\n> (understandably) bump the default sooner than later, or if not for this\n> data for other future chunks.\n\nI think somewhere I said we wouldn't want to update this default for\nat least a year after it ships, but I'm also happy to never update it\nand let experts opt-in when they want.\n\n>> Finally, the end result becomes \"older versions get slower without\n>> any warning\" instead of \"older versions get a message about not\n>> understanding the commit-graph file\".\n> \n> Sure, IF you want to write such an empty chunk. The point is that you\n> now have the option.\n> \n> And this is the same edge case we already dealt with for\n> GDAT/GDOV. I.e. older readers who didn't understand it would be slower.\n> \n> We can still have a feature to make older clients intentionally\n> break/warn, it seems to me that if you'd want such a thing you'd want it\n> aside from the specific mechanism of this proposed upgrade.\n> \n> Or you could dual-write the data for older clients, which I think\n> probably isn't worth the hassle.\n> \n> I.e. if you're worried about silent slowdown older clients happily\n> ignoring the BIDX and BDAT chunks are silently slower.\n\nOlder clients ignoring BIDX and BDAT chunks means they are silently\nslower than newer clients, but they are still as fast as they were\nyesterday. The empty chunk approach will make those older Git versions\nslower than they were yesterday.\n\nThanks,\n-Stolee\n"},{"id":"449942","messageId":"220301.86tuchzi1i.gmgdl@evledraar.gmail.com","threadId":"57472","inReplyTo":"pull.1163.v2.git.1646056423.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/4] Commit-graph: Generation Number v2 Fixes","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-03-01T17:23:43Z","receivedAt":"2022-03-01T17:42:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Feb 28 2022, Derrick Stolee via GitGitGadget wrote:\n\n> In particular, Git has been ignoring corrected commit dates since shortly\n> after they were introduced. This is due to a bug I introduced when trying to\n> make split commit-graphs safer with mixed generation number versions. I also\n> noticed an issue with the offset overflows that I only noticed after writing\n> generation number v3 using a smaller offset size, actually triggering the\n> bug in the test suite.\n\nI think sans existing small issues/fixes I noted this looks good.\n\nJust a bit on the overall direction/design. And don't worry, not the\nverison v.s. chunk all over again (except notes about how the two\nversions eventually interact) :)\n\nI.e. the post-state here looks good, but I wondered about the direction\nof:\n\n * We have a commitGraph.readChangedPaths for *reading* BIDX/BDAT, on by default\n * We have a commitgraph.generationVersion which pre this series is 1, post-2.\n * >= 2 means look at the GDAT/GDOV chunk, and when splitting/rewriting etc.\n carry them forward.\n\nSo, I wonder:\n\nA. Do we really need these \"yes I'll read this thing in the file\" settings at all?\n\n Isn't it sufficient to have core.commitGraph=false as an escape hatch, do we\n really need to be able to selectively ignore individual parts of the file on-disk?\n\nB. For a \"selective ignore\" the commitGraph.readChangedPaths=BOOL makes sense, but\n given the follow-up series it seems odd to end up with a commitgraph.generationVersion=3\n which bumps the top-level version of the commit-graph.\n\nI.e. commitgraph.generationVersion=2 *is* optional since the GDAT/GCOV\nchunks can be ignored, but commitgraph.generationVersion=3 is *not* since\nit'll also bump the format version to v2 (not v3!).\n\nSo yeah, like I said I'm being quiet about the top-level version\nv.s. chunks here blah blah :)\n\nBut for the end-state you want (as I understand it) wouldn't this make\nmore sense:\n\n1. Just say we have write settings + \"core.commitGraph=false\" escape\n hatch, no \"selective read\".\n\n2. Make commitgraph.generationVersion=2 an alias for a more obviously named\n commitGraph.readGenerationData=true (optional). Maybe deprecate\n \"commitGraph.readGenerationData\" (say \"error: just don't write it then\")\n\n3. Never have a commitgraph.generationVersion=3 setting, but instead\n add say a core.writeCommitGraphVersion=2.\n\n We'd thus not be conflating \"commitgraph.generationVersion\" which *is*\n optional and is on both the read *and* write side, with your WIP\n commitgraph.generationVersion=3 which also changes how writes happen,\n but is not at all optional for reads. You'll either support it or get a\n warning() when loading the graph.\n\nI think this should all be OK, and I don't think it conflicts with your\nstated preferences about the top-level version v.s. optional/redundant\nchunks.\n\nIt makes things simpler since we'd always read data we find, with no\n\"maybe ignore it if it's there\" settings.\n\nAnd it avoids user confusion of e.g. getting an error about not\nunderstanding a v2 graph after setting a commitgraph.generationVersion=3\nsetting (since one is 0-indexed...), and having \"versions\" in the same\nconfig setting being first optionalon read/write, to very much not\noptional.\n\n"},{"id":"449993","messageId":"pull.1163.v3.git.1646164112.gitgitgadget@gmail.com","threadId":"57472","inReplyTo":"pull.1163.v2.git.1646056423.gitgitgadget@gmail.com","subject":"[PATCH v3 0/5] Commit-graph: Generation Number v2 Fixes","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T19:48:27Z","receivedAt":"2022-03-01T19:49:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This patch series fixes some bugs in generation number v2. They were\ndiscovered while building generation number v3, but that implementation will\nbe delayed until these fixes are merged.\n\nIn particular, Git has been ignoring corrected commit dates since shortly\nafter they were introduced. This is due to a bug I introduced when trying to\nmake split commit-graphs safer with mixed generation number versions. I also\nnoticed an issue with the offset overflows that I only noticed after writing\ngeneration number v3 using a smaller offset size, actually triggering the\nbug in the test suite.\n\n\nUpdates in v3\n=============\n\n * Used portable printf macros for uint32.\n * Added a new patch that creates lib-commit-graph.sh in preparation for new\n 64-bit test script.\n * While copying this information, replaced 'test ! -z ...' with 'test -n\n ...'\n * Instead of editing existing overflow tests, created a new test script\n focused on 64-bit tests.\n\n\nUpdates in v2\n=============\n\n * Dropped generation v3 patches, saving them for later.\n * Updated a commit message to more clearly describe the problem with the\n old code.\n * Used an || instead of two if statements in test script.\n\nThanks, -Stolee\n\nDerrick Stolee (5):\n test-read-graph: include extra post-parse info\n t5318: extract helpers to lib-commit-graph.sh\n commit-graph: fix ordering bug in generation numbers\n commit-graph: start parsing generation v2 (again)\n commit-graph: fix generation number v2 overflow values\n\n commit-graph.c | 15 +++++--\n t/helper/test-read-graph.c | 13 ++++++\n t/lib-commit-graph.sh | 58 ++++++++++++++++++++++++++\n t/t4216-log-bloom.sh | 1 +\n t/t5318-commit-graph.sh | 55 +++----------------------\n t/t5324-split-commit-graph.sh | 10 +++++\n t/t5328-commit-graph-64bit-time.sh | 66 ++++++++++++++++++++++++++++++\n 7 files changed, 164 insertions(+), 54 deletions(-)\n create mode 100755 t/lib-commit-graph.sh\n create mode 100755 t/t5328-commit-graph-64bit-time.sh\n\n\nbase-commit: dab1b7905d0b295f1acef9785bb2b9cbb0fdec84\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1163%2Fderrickstolee%2Fgen-v3-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1163/derrickstolee/gen-v3-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1163\n\nRange-diff vs v2:\n\n 1: 2f89275314b ! 1: 4eca8028c97 test-read-graph: include extra post-parse info\n @@ t/helper/test-read-graph.c: int cmd__read_graph(int argc, const char **argv)\n \n +\tprintf(\"options:\");\n +\tif (graph->bloom_filter_settings)\n -+\t\tprintf(\" bloom(%d,%d,%d)\",\n ++\t\tprintf(\" bloom(%\"PRIu32\",%\"PRIu32\",%\"PRIu32\")\",\n +\t\t graph->bloom_filter_settings->hash_version,\n +\t\t graph->bloom_filter_settings->bits_per_entry,\n +\t\t graph->bloom_filter_settings->num_hashes);\n -: ----------- > 2: a90cad8efa5 t5318: extract helpers to lib-commit-graph.sh\n 2: cbcbf10e699 ! 3: 562341b76b3 commit-graph: fix ordering bug in generation numbers\n @@ Commit message\n Instead, iterate over the full commit list at the end, checking the\n offsets to see how many grow beyond the maximum value.\n \n - Update a test in t5318 to use a larger time value, which will help\n - demonstrate this bug in more cases. It still won't hit all potential\n - cases until the next change, which reenables reading generation numbers.\n + Create a new t5328-commit-graph-64-bit-time.sh test script to handle\n + special cases of testing 64-bit timestampes. This helps demonstrate this\n + bug in more cases. It still won't hit all potential cases until the next\n + change, which reenables reading generation numbers. Use the skip_all\n + trick from 0a2bfccb9c8 (t0051: use \"skip_all\" under !MINGW in\n + single-test file, 2022-02-04) to make the output clean when run on a\n + 32-bit system.\n \n + Hepled-by: Ævar Arnfjörð Bjarmason \n Signed-off-by: Derrick Stolee \n \n ## commit-graph.c ##\n @@ t/t5318-commit-graph.sh: test_expect_success 'warn on improper hash version' '\n \trm -f .git/objects/info/commit-graph &&\n \ttest_commit --date \"$FUTURE_DATE\" future-1 &&\n \ttest_commit --date \"$UNIX_EPOCH_ZERO\" old-1 &&\n +\n + ## t/t5328-commit-graph-64bit-time.sh (new) ##\n +@@\n ++#!/bin/sh\n ++\n ++test_description='commit graph with 64-bit timestamps'\n ++. ./test-lib.sh\n ++\n ++if ! test_have_prereq TIME_IS_64BIT || ! test_have_prereq TIME_T_IS_64BIT\n ++then\n ++\tskip_all='skipping 64-bit timestamp tests'\n ++\ttest_done\n ++fi\n ++\n ++. \"$TEST_DIRECTORY\"/lib-commit-graph.sh\n ++\n ++UNIX_EPOCH_ZERO=\"@0 +0000\"\n ++FUTURE_DATE=\"@4147483646 +0000\"\n ++\n ++GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=0\n ++\n ++test_expect_success 'lower layers have overflow chunk' '\n ++\trm -f .git/objects/info/commit-graph &&\n ++\ttest_commit --date \"$FUTURE_DATE\" future-1 &&\n ++\ttest_commit --date \"$UNIX_EPOCH_ZERO\" old-1 &&\n ++\tgit commit-graph write --reachable &&\n ++\ttest_commit --date \"$FUTURE_DATE\" future-2 &&\n ++\ttest_commit --date \"$UNIX_EPOCH_ZERO\" old-2 &&\n ++\tgit commit-graph write --reachable --split=no-merge &&\n ++\ttest_commit extra &&\n ++\tgit commit-graph write --reachable --split=no-merge &&\n ++\tgit commit-graph write --reachable &&\n ++\tgraph_read_expect 5 \"generation_data generation_data_overflow\" &&\n ++\tmv .git/objects/info/commit-graph commit-graph-upgraded &&\n ++\tgit commit-graph write --reachable &&\n ++\tgraph_read_expect 5 \"generation_data generation_data_overflow\" &&\n ++\ttest_cmp .git/objects/info/commit-graph commit-graph-upgraded\n ++'\n ++\n ++graph_git_behavior 'overflow' '' HEAD~2 HEAD\n ++\n ++test_done\n 3: 5bc6a7660d8 ! 4: 041d96bf1d7 commit-graph: start parsing generation v2 (again)\n @@ commit-graph.c: struct commit_graph *parse_commit_graph(struct repository *r,\n \n \tif (r->settings.commit_graph_read_changed_paths) {\n \n - ## t/t4216-log-bloom.sh ##\n -@@ t/t4216-log-bloom.sh: graph_read_expect () {\n - \theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n - \tnum_commits: $1\n - \tchunks: oid_fanout oid_lookup commit_metadata generation_data bloom_indexes bloom_data\n --\toptions: bloom(1,10,7)\n -+\toptions: bloom(1,10,7) read_generation_data\n - \tEOF\n - \ttest-tool read-graph >actual &&\n - \ttest_cmp expect actual\n -\n - ## t/t5318-commit-graph.sh ##\n -@@ t/t5318-commit-graph.sh: graph_read_expect() {\n + ## t/lib-commit-graph.sh ##\n +@@ t/lib-commit-graph.sh: graph_read_expect() {\n \t\tOPTIONAL=\" $2\"\n \t\tNUM_CHUNKS=$((3 + $(echo \"$2\" | wc -w)))\n \tfi\n +\tGENERATION_VERSION=2\n -+\tif test ! -z \"$3\"\n ++\tif test -n \"$3\"\n +\tthen\n +\t\tGENERATION_VERSION=$3\n +\tfi\n @@ t/t5318-commit-graph.sh: graph_read_expect() {\n \tEOF\n \ttest-tool read-graph >output &&\n \ttest_cmp expect output\n +\n + ## t/t4216-log-bloom.sh ##\n +@@ t/t4216-log-bloom.sh: graph_read_expect () {\n + \theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n + \tnum_commits: $1\n + \tchunks: oid_fanout oid_lookup commit_metadata generation_data bloom_indexes bloom_data\n +-\toptions: bloom(1,10,7)\n ++\toptions: bloom(1,10,7) read_generation_data\n + \tEOF\n + \ttest-tool read-graph >actual &&\n + \ttest_cmp expect actual\n +\n + ## t/t5318-commit-graph.sh ##\n @@ t/t5318-commit-graph.sh: test_expect_success 'git commit-graph verify' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit rev-parse commits/8 | git -c commitGraph.generationVersion=1 commit-graph write --stdin-commits &&\n 4: 193217c71e0 ! 5: e957baa9d77 commit-graph: fix generation number v2 overflow values\n @@ Commit message\n show up as a failure in 'git commit-graph verify' if we increase that\n FUTURE_DATE to be above four billion.\n \n - Fix this error and update the test to require 64-bit dates so we can\n - safely use this large value in our test.\n + Fix this error and create a 64-bit timestamp version of the test so we\n + can test these larger values.\n \n Signed-off-by: Derrick Stolee \n \n @@ commit-graph.c: static void fill_commit_graph_info(struct commit *item, struct c\n \t\t\tgraph_data->generation = item->date + offset;\n \t} else\n \n - ## t/t5318-commit-graph.sh ##\n -@@ t/t5318-commit-graph.sh: test_expect_success 'corrupt commit-graph write (missing tree)' '\n - \t)\n - '\n + ## t/t5328-commit-graph-64bit-time.sh ##\n +@@ t/t5328-commit-graph-64bit-time.sh: test_expect_success 'lower layers have overflow chunk' '\n \n -+# The remaining tests check timestamps that flow over\n -+# 32-bits. The graph_git_behavior checks can't take a\n -+# prereq, so just stop here if we are on a 32-bit machine.\n -+\n -+if ! test_have_prereq TIME_IS_64BIT || ! test_have_prereq TIME_T_IS_64BIT\n -+then\n -+\ttest_done\n -+fi\n -+\n - # We test the overflow-related code with the following repo history:\n - #\n - # 4:F - 5:N - 6:U\n -@@ t/t5318-commit-graph.sh: test_expect_success 'corrupt commit-graph write (missing tree)' '\n - # The largest offset observed is 2 ^ 31, just large enough to overflow.\n - #\n - \n --test_expect_success 'set up and verify repo with generation data overflow chunk' '\n -+test_expect_success TIME_IS_64BIT,TIME_T_IS_64BIT 'set up and verify repo with generation data overflow chunk' '\n - \tobjdir=\".git/objects\" &&\n - \tUNIX_EPOCH_ZERO=\"@0 +0000\" &&\n --\tFUTURE_DATE=\"@2147483646 +0000\" &&\n -+\tFUTURE_DATE=\"@4000000000 +0000\" &&\n - \ttest_oid_cache <<-EOF &&\n - \toid_version sha1:1\n - \toid_version sha256:2\n -@@ t/t5318-commit-graph.sh: test_expect_success 'set up and verify repo with generation data overflow chunk'\n + graph_git_behavior 'overflow' '' HEAD~2 HEAD\n \n - graph_git_behavior 'generation data overflow chunk repo' repo left right\n - \n -+# Do not add tests at the end of this file, unless they require 64-bit\n -+# timestamps, since this portion of the script is only executed when\n -+# time data types have 64 bits.\n ++test_expect_success 'set up and verify repo with generation data overflow chunk' '\n ++\tmkdir repo &&\n ++\tcd repo &&\n ++\tgit init &&\n ++\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 1 &&\n ++\ttest_commit 2 &&\n ++\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 3 &&\n ++\tgit commit-graph write --reachable &&\n ++\tgraph_read_expect 3 generation_data &&\n ++\ttest_commit --date \"$FUTURE_DATE\" 4 &&\n ++\ttest_commit 5 &&\n ++\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 6 &&\n ++\tgit branch left &&\n ++\tgit reset --hard 3 &&\n ++\ttest_commit 7 &&\n ++\ttest_commit --date \"$FUTURE_DATE\" 8 &&\n ++\ttest_commit 9 &&\n ++\tgit branch right &&\n ++\tgit reset --hard 3 &&\n ++\ttest_merge M left right &&\n ++\tgit commit-graph write --reachable &&\n ++\tgraph_read_expect 10 \"generation_data generation_data_overflow\" &&\n ++\tgit commit-graph verify\n ++'\n ++\n ++graph_git_behavior 'overflow 2' repo left right\n +\n test_done\n\n-- \ngitgitgadget\n"},{"id":"449994","messageId":"562341b76b30ff3cdc01cea4cda550a7b8165bdd.1646164112.git.gitgitgadget@gmail.com","threadId":"57472","inReplyTo":"pull.1163.v3.git.1646164112.gitgitgadget@gmail.com","subject":"[PATCH v3 3/5] commit-graph: fix ordering bug in generation numbers","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T19:48:30Z","receivedAt":"2022-03-01T19:49:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen computing the generation numbers for a commit-graph, we compute\nthe corrected commit dates and then check if their offsets from the\nactual dates is too large to fit in the 32-bit Generation Data chunk.\nHowever, there is a problem with this approach: if we have parsed the\ngeneration data from the previous commit-graph, then we continue the\nloop because the corrected commit date is already computed. This causes\nan under-count in the number of overflow values.\n\nIt is incorrect to add an increment to num_generation_data_overflows\nnext to this 'continue' statement, because we might start\ndouble-counting commits that are computed because of the depth-first\nsearch walk from a commit with an earlier OID.\n\nInstead, iterate over the full commit list at the end, checking the\noffsets to see how many grow beyond the maximum value.\n\nCreate a new t5328-commit-graph-64-bit-time.sh test script to handle\nspecial cases of testing 64-bit timestampes. This helps demonstrate this\nbug in more cases. It still won't hit all potential cases until the next\nchange, which reenables reading generation numbers. Use the skip_all\ntrick from 0a2bfccb9c8 (t0051: use \"skip_all\" under !MINGW in\nsingle-test file, 2022-02-04) to make the output clean when run on a\n32-bit system.\n\nHepled-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 10 +++++---\n t/t5318-commit-graph.sh | 4 +--\n t/t5328-commit-graph-64bit-time.sh | 39 ++++++++++++++++++++++++++++++\n 3 files changed, 48 insertions(+), 5 deletions(-)\n create mode 100755 t/t5328-commit-graph-64bit-time.sh\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 265c010122e..a19bd96c2ee 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1556,12 +1556,16 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \t\t\t\tif (current->date && current->date > max_corrected_commit_date)\n \t\t\t\t\tmax_corrected_commit_date = current->date - 1;\n \t\t\t\tcommit_graph_data_at(current)->generation = max_corrected_commit_date + 1;\n-\n-\t\t\t\tif (commit_graph_data_at(current)->generation - current->date > GENERATION_NUMBER_V2_OFFSET_MAX)\n-\t\t\t\t\tctx->num_generation_data_overflows++;\n \t\t\t}\n \t\t}\n \t}\n+\n+\tfor (i = 0; i < ctx->commits.nr; i++) {\n+\t\tstruct commit *c = ctx->commits.list[i];\n+\t\ttimestamp_t offset = commit_graph_data_at(c)->generation - c->date;\n+\t\tif (offset > GENERATION_NUMBER_V2_OFFSET_MAX)\n+\t\t\tctx->num_generation_data_overflows++;\n+\t}\n \tstop_progress(&ctx->progress);\n }\n \ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 9e2b5884dae..0ed7e9de8e6 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -425,10 +425,10 @@ test_expect_success 'warn on improper hash version' '\n \t)\n '\n \n-test_expect_success 'lower layers have overflow chunk' '\n+test_expect_success TIME_IS_64BIT,TIME_T_IS_64BIT 'lower layers have overflow chunk' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tUNIX_EPOCH_ZERO=\"@0 +0000\" &&\n-\tFUTURE_DATE=\"@2147483646 +0000\" &&\n+\tFUTURE_DATE=\"@4147483646 +0000\" &&\n \trm -f .git/objects/info/commit-graph &&\n \ttest_commit --date \"$FUTURE_DATE\" future-1 &&\n \ttest_commit --date \"$UNIX_EPOCH_ZERO\" old-1 &&\ndiff --git a/t/t5328-commit-graph-64bit-time.sh b/t/t5328-commit-graph-64bit-time.sh\nnew file mode 100755\nindex 00000000000..28114bcaf47\n--- /dev/null\n+++ b/t/t5328-commit-graph-64bit-time.sh\n@@ -0,0 +1,39 @@\n+#!/bin/sh\n+\n+test_description='commit graph with 64-bit timestamps'\n+. ./test-lib.sh\n+\n+if ! test_have_prereq TIME_IS_64BIT || ! test_have_prereq TIME_T_IS_64BIT\n+then\n+\tskip_all='skipping 64-bit timestamp tests'\n+\ttest_done\n+fi\n+\n+. \"$TEST_DIRECTORY\"/lib-commit-graph.sh\n+\n+UNIX_EPOCH_ZERO=\"@0 +0000\"\n+FUTURE_DATE=\"@4147483646 +0000\"\n+\n+GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=0\n+\n+test_expect_success 'lower layers have overflow chunk' '\n+\trm -f .git/objects/info/commit-graph &&\n+\ttest_commit --date \"$FUTURE_DATE\" future-1 &&\n+\ttest_commit --date \"$UNIX_EPOCH_ZERO\" old-1 &&\n+\tgit commit-graph write --reachable &&\n+\ttest_commit --date \"$FUTURE_DATE\" future-2 &&\n+\ttest_commit --date \"$UNIX_EPOCH_ZERO\" old-2 &&\n+\tgit commit-graph write --reachable --split=no-merge &&\n+\ttest_commit extra &&\n+\tgit commit-graph write --reachable --split=no-merge &&\n+\tgit commit-graph write --reachable &&\n+\tgraph_read_expect 5 \"generation_data generation_data_overflow\" &&\n+\tmv .git/objects/info/commit-graph commit-graph-upgraded &&\n+\tgit commit-graph write --reachable &&\n+\tgraph_read_expect 5 \"generation_data generation_data_overflow\" &&\n+\ttest_cmp .git/objects/info/commit-graph commit-graph-upgraded\n+'\n+\n+graph_git_behavior 'overflow' '' HEAD~2 HEAD\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"449995","messageId":"e957baa9d77873239ab0e0056138fd490f029f5c.1646164112.git.gitgitgadget@gmail.com","threadId":"57472","inReplyTo":"pull.1163.v3.git.1646164112.gitgitgadget@gmail.com","subject":"[PATCH v3 5/5] commit-graph: fix generation number v2 overflow values","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T19:48:32Z","receivedAt":"2022-03-01T19:50:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe Generation Data Chunk was implemented and tested in e8b63005c\n(commit-graph: implement generation data chunk, 2021-01-16), but the\ntest was carefully constructed to work on systems with 32-bit dates.\nSince the corrected commit date offsets still required more than 31\nbits, this triggered writing the generation_data_overflow chunk.\n\nHowever, upon closer look, the\nwrite_graph_chunk_generation_data_overflow() method writes the offsets\nto the chunk (as dictated by the format) but fill_commit_graph_info()\ntreats the value in the chunk as if it is the full corrected commit date\n(not an offset). For some reason, this does not cause an issue when\nusing the FUTURE_DATE specified in t5318-commit-graph.sh, but it does\nshow up as a failure in 'git commit-graph verify' if we increase that\nFUTURE_DATE to be above four billion.\n\nFix this error and create a 64-bit timestamp version of the test so we\ncan test these larger values.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 2 +-\n t/t5328-commit-graph-64bit-time.sh | 27 +++++++++++++++++++++++++++\n 2 files changed, 28 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8e52bb09552..b86a6a634fe 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -806,7 +806,7 @@ static void fill_commit_graph_info(struct commit *item, struct commit_graph *g,\n \t\t\t\tdie(_(\"commit-graph requires overflow generation data but has none\"));\n \n \t\t\toffset_pos = offset ^ CORRECTED_COMMIT_DATE_OFFSET_OVERFLOW;\n-\t\t\tgraph_data->generation = get_be64(g->chunk_generation_data_overflow + 8 * offset_pos);\n+\t\t\tgraph_data->generation = item->date + get_be64(g->chunk_generation_data_overflow + 8 * offset_pos);\n \t\t} else\n \t\t\tgraph_data->generation = item->date + offset;\n \t} else\ndiff --git a/t/t5328-commit-graph-64bit-time.sh b/t/t5328-commit-graph-64bit-time.sh\nindex 28114bcaf47..093f0c067af 100755\n--- a/t/t5328-commit-graph-64bit-time.sh\n+++ b/t/t5328-commit-graph-64bit-time.sh\n@@ -36,4 +36,31 @@ test_expect_success 'lower layers have overflow chunk' '\n \n graph_git_behavior 'overflow' '' HEAD~2 HEAD\n \n+test_expect_success 'set up and verify repo with generation data overflow chunk' '\n+\tmkdir repo &&\n+\tcd repo &&\n+\tgit init &&\n+\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 1 &&\n+\ttest_commit 2 &&\n+\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 3 &&\n+\tgit commit-graph write --reachable &&\n+\tgraph_read_expect 3 generation_data &&\n+\ttest_commit --date \"$FUTURE_DATE\" 4 &&\n+\ttest_commit 5 &&\n+\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 6 &&\n+\tgit branch left &&\n+\tgit reset --hard 3 &&\n+\ttest_commit 7 &&\n+\ttest_commit --date \"$FUTURE_DATE\" 8 &&\n+\ttest_commit 9 &&\n+\tgit branch right &&\n+\tgit reset --hard 3 &&\n+\ttest_merge M left right &&\n+\tgit commit-graph write --reachable &&\n+\tgraph_read_expect 10 \"generation_data generation_data_overflow\" &&\n+\tgit commit-graph verify\n+'\n+\n+graph_git_behavior 'overflow 2' repo left right\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"449996","messageId":"041d96bf1d7fecbae946ad71cbe2dc0b3492fd65.1646164112.git.gitgitgadget@gmail.com","threadId":"57472","inReplyTo":"pull.1163.v3.git.1646164112.gitgitgadget@gmail.com","subject":"[PATCH v3 4/5] commit-graph: start parsing generation v2 (again)","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T19:48:31Z","receivedAt":"2022-03-01T19:50:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'read_generation_data' member of 'struct commit_graph' was\nintroduced by 1fdc383c5 (commit-graph: use generation v2 only if entire\nchain does, 2021-01-16). The intention was to avoid using corrected\ncommit dates if not all layers of a commit-graph had that data stored.\nThe logic in validate_mixed_generation_chain() at that point incorrectly\ninitialized read_generation_data to 1 if and only if the tip\ncommit-graph contained the Corrected Commit Date chunk.\n\nThis was \"fixed\" in 448a39e65 (commit-graph: validate layers for\ngeneration data, 2021-02-02) to validate that read_generation_data was\neither non-zero for all layers, or it would set read_generation_data to\nzero for all layers.\n\nThe problem here is that read_generation_data is not initialized to be\nnon-zero anywhere!\n\nThis change initializes read_generation_data immediately after the chunk\nis parsed, so each layer will have its value present as soon as\npossible.\n\nThe read_generation_data member is used in fill_commit_graph_info() to\ndetermine if we should use the corrected commit date or the topological\nlevels stored in the Commit Data chunk. Due to this bug, all previous\nversions of Git were defaulting to topological levels in all cases!\n\nThis can be measured with some performance tests. Using the Linux kernel\nas a testbed, I generated a complete commit-graph containing corrected\ncommit dates and tested the 'new' version against the previous, 'old'\nversion.\n\nFirst, rev-list with --topo-order demonstrates a 26% improvement using\ncorrected commit dates:\n\nhyperfine \\\n\t-n \"old\" \"$OLD_GIT rev-list --topo-order -1000 v3.6\" \\\n\t-n \"new\" \"$NEW_GIT rev-list --topo-order -1000 v3.6\" \\\n\t--warmup=10\n\nBenchmark 1: old\n Time (mean ± σ): 57.1 ms ± 3.1 ms\n Range (min … max): 52.9 ms … 62.0 ms 55 runs\n\nBenchmark 2: new\n Time (mean ± σ): 45.5 ms ± 3.3 ms\n Range (min … max): 39.9 ms … 51.7 ms 59 runs\n\nSummary\n 'new' ran\n 1.26 ± 0.11 times faster than 'old'\n\nThese performance improvements are due to the algorithmic improvements\ngiven by walking fewer commits due to the higher cutoffs from corrected\ncommit dates.\n\nHowever, this comes at a cost. The additional I/O cost of parsing the\ncorrected commit dates is visible in case of merge-base commands that do\nnot reduce the overall number of walked commits.\n\nhyperfine \\\n -n \"old\" \"$OLD_GIT merge-base v4.8 v4.9\" \\\n -n \"new\" \"$NEW_GIT merge-base v4.8 v4.9\" \\\n --warmup=10\n\nBenchmark 1: old\n Time (mean ± σ): 110.4 ms ± 6.4 ms\n Range (min … max): 96.0 ms … 118.3 ms 25 runs\n\nBenchmark 2: new\n Time (mean ± σ): 150.7 ms ± 1.1 ms\n Range (min … max): 149.3 ms … 153.4 ms 19 runs\n\nSummary\n 'old' ran\n 1.36 ± 0.08 times faster than 'new'\n\nPerformance issues like this are what motivated 702110aac (commit-graph:\nuse config to specify generation type, 2021-02-25).\n\nIn the future, we could fix this performance problem by inserting the\ncorrected commit date offsets into the Commit Date chunk instead of\nhaving that data in an extra chunk.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 3 +++\n t/lib-commit-graph.sh | 12 +++++++++++-\n t/t4216-log-bloom.sh | 2 +-\n t/t5318-commit-graph.sh | 2 +-\n t/t5324-split-commit-graph.sh | 9 +++++++--\n 5 files changed, 23 insertions(+), 5 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a19bd96c2ee..8e52bb09552 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -407,6 +407,9 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t\t&graph->chunk_generation_data);\n \t\tpair_chunk(cf, GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW,\n \t\t\t&graph->chunk_generation_data_overflow);\n+\n+\t\tif (graph->chunk_generation_data)\n+\t\t\tgraph->read_generation_data = 1;\n \t}\n \n \tif (r->settings.commit_graph_read_changed_paths) {\ndiff --git a/t/lib-commit-graph.sh b/t/lib-commit-graph.sh\nindex 07e12b9d2fe..5d79e1a4e96 100755\n--- a/t/lib-commit-graph.sh\n+++ b/t/lib-commit-graph.sh\n@@ -37,11 +37,21 @@ graph_read_expect() {\n \t\tOPTIONAL=\" $2\"\n \t\tNUM_CHUNKS=$((3 + $(echo \"$2\" | wc -w)))\n \tfi\n+\tGENERATION_VERSION=2\n+\tif test -n \"$3\"\n+\tthen\n+\t\tGENERATION_VERSION=$3\n+\tfi\n+\tOPTIONS=\n+\tif test $GENERATION_VERSION -gt 1\n+\tthen\n+\t\tOPTIONS=\" read_generation_data\"\n+\tfi\n \tcat >expect <<- EOF\n \theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata$OPTIONAL\n-\toptions:\n+\toptions:$OPTIONS\n \tEOF\n \ttest-tool read-graph >output &&\n \ttest_cmp expect output\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 5ed6d2a21c1..fa9d32facfb 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -48,7 +48,7 @@ graph_read_expect () {\n \theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata generation_data bloom_indexes bloom_data\n-\toptions: bloom(1,10,7)\n+\toptions: bloom(1,10,7) read_generation_data\n \tEOF\n \ttest-tool read-graph >actual &&\n \ttest_cmp expect actual\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 0ed7e9de8e6..fbf0d64578c 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -456,7 +456,7 @@ test_expect_success 'git commit-graph verify' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit rev-parse commits/8 | git -c commitGraph.generationVersion=1 commit-graph write --stdin-commits &&\n \tgit commit-graph verify >output &&\n-\tgraph_read_expect 9 extra_edges\n+\tgraph_read_expect 9 extra_edges 1\n '\n \n NUM_COMMITS=9\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 778fa418de2..669ddc645fa 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -30,11 +30,16 @@ graph_read_expect() {\n \tthen\n \t\tNUM_BASE=$2\n \tfi\n+\tOPTIONS=\n+\tif test -z \"$3\"\n+\tthen\n+\t\tOPTIONS=\" read_generation_data\"\n+\tfi\n \tcat >expect <<- EOF\n \theader: 43475048 1 $(test_oid oid_version) 4 $NUM_BASE\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata generation_data\n-\toptions:\n+\toptions:$OPTIONS\n \tEOF\n \ttest-tool read-graph >output &&\n \ttest_cmp expect output\n@@ -624,7 +629,7 @@ test_expect_success 'write generation data chunk if topmost remaining layer has\n \t\theader: 43475048 1 $(test_oid oid_version) 5 1\n \t\tnum_commits: $(($NUM_SECOND_LAYER_COMMITS + $NUM_THIRD_LAYER_COMMITS + $NUM_FOURTH_LAYER_COMMITS + $NUM_FIFTH_LAYER_COMMITS))\n \t\tchunks: oid_fanout oid_lookup commit_metadata generation_data\n-\t\toptions:\n+\t\toptions: read_generation_data\n \t\tEOF\n \t\ttest_cmp expect output\n \t)\n-- \ngitgitgadget\n\n"},{"id":"449997","messageId":"4eca8028c971e32fdf34720de4674daf7004a294.1646164112.git.gitgitgadget@gmail.com","threadId":"57472","inReplyTo":"pull.1163.v3.git.1646164112.gitgitgadget@gmail.com","subject":"[PATCH v3 1/5] test-read-graph: include extra post-parse info","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T19:48:28Z","receivedAt":"2022-03-01T19:50:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIt can be helpful to verify that the 'struct commit_graph' that results\nfrom parsing a commit-graph is correctly structured. The existence of\ndifferent chunks is not enough to verify that all of the optional\nfeatures are correctly enabled.\n\nUpdate 'test-tool read-graph' to output an \"options:\" line that includes\ninformation for different parts of the struct commit_graph.\n\nIn particular, this change demonstrates that the read_generation_data\noption is never being enabled, which will be fixed in a later change.\n\nSigned-off-by: Derrick Stolee \n---\n t/helper/test-read-graph.c | 13 +++++++++++++\n t/t4216-log-bloom.sh | 1 +\n t/t5318-commit-graph.sh | 1 +\n t/t5324-split-commit-graph.sh | 5 +++++\n 4 files changed, 20 insertions(+)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex 75927b2c81d..98b73bb8f25 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -3,6 +3,7 @@\n #include \"commit-graph.h\"\n #include \"repository.h\"\n #include \"object-store.h\"\n+#include \"bloom.h\"\n \n int cmd__read_graph(int argc, const char **argv)\n {\n@@ -45,6 +46,18 @@ int cmd__read_graph(int argc, const char **argv)\n \t\tprintf(\" bloom_data\");\n \tprintf(\"\\n\");\n \n+\tprintf(\"options:\");\n+\tif (graph->bloom_filter_settings)\n+\t\tprintf(\" bloom(%\"PRIu32\",%\"PRIu32\",%\"PRIu32\")\",\n+\t\t graph->bloom_filter_settings->hash_version,\n+\t\t graph->bloom_filter_settings->bits_per_entry,\n+\t\t graph->bloom_filter_settings->num_hashes);\n+\tif (graph->read_generation_data)\n+\t\tprintf(\" read_generation_data\");\n+\tif (graph->topo_levels)\n+\t\tprintf(\" topo_levels\");\n+\tprintf(\"\\n\");\n+\n \tUNLEAK(graph);\n \n \treturn 0;\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex cc3cebf6722..5ed6d2a21c1 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -48,6 +48,7 @@ graph_read_expect () {\n \theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata generation_data bloom_indexes bloom_data\n+\toptions: bloom(1,10,7)\n \tEOF\n \ttest-tool read-graph >actual &&\n \ttest_cmp expect actual\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex edb728f77c3..2b05026cf6d 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -104,6 +104,7 @@ graph_read_expect() {\n \theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata$OPTIONAL\n+\toptions:\n \tEOF\n \ttest-tool read-graph >output &&\n \ttest_cmp expect output\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 847b8097109..778fa418de2 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -34,6 +34,7 @@ graph_read_expect() {\n \theader: 43475048 1 $(test_oid oid_version) 4 $NUM_BASE\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata generation_data\n+\toptions:\n \tEOF\n \ttest-tool read-graph >output &&\n \ttest_cmp expect output\n@@ -508,6 +509,7 @@ test_expect_success 'setup repo for mixed generation commit-graph-chain' '\n \t\theader: 43475048 1 $(test_oid oid_version) 4 1\n \t\tnum_commits: $NUM_SECOND_LAYER_COMMITS\n \t\tchunks: oid_fanout oid_lookup commit_metadata\n+\t\toptions:\n \t\tEOF\n \t\ttest_cmp expect output &&\n \t\tgit commit-graph verify &&\n@@ -540,6 +542,7 @@ test_expect_success 'do not write generation data chunk if not present on existi\n \t\theader: 43475048 1 $(test_oid oid_version) 4 2\n \t\tnum_commits: $NUM_THIRD_LAYER_COMMITS\n \t\tchunks: oid_fanout oid_lookup commit_metadata\n+\t\toptions:\n \t\tEOF\n \t\ttest_cmp expect output &&\n \t\tgit commit-graph verify\n@@ -581,6 +584,7 @@ test_expect_success 'do not write generation data chunk if the topmost remaining\n \t\theader: 43475048 1 $(test_oid oid_version) 4 2\n \t\tnum_commits: $(($NUM_THIRD_LAYER_COMMITS + $NUM_FOURTH_LAYER_COMMITS))\n \t\tchunks: oid_fanout oid_lookup commit_metadata\n+\t\toptions:\n \t\tEOF\n \t\ttest_cmp expect output &&\n \t\tgit commit-graph verify\n@@ -620,6 +624,7 @@ test_expect_success 'write generation data chunk if topmost remaining layer has\n \t\theader: 43475048 1 $(test_oid oid_version) 5 1\n \t\tnum_commits: $(($NUM_SECOND_LAYER_COMMITS + $NUM_THIRD_LAYER_COMMITS + $NUM_FOURTH_LAYER_COMMITS + $NUM_FIFTH_LAYER_COMMITS))\n \t\tchunks: oid_fanout oid_lookup commit_metadata generation_data\n+\t\toptions:\n \t\tEOF\n \t\ttest_cmp expect output\n \t)\n-- \ngitgitgadget\n\n"},{"id":"449998","messageId":"a90cad8efa5c2f316b6f054153e32e4e94cee025.1646164112.git.gitgitgadget@gmail.com","threadId":"57472","inReplyTo":"pull.1163.v3.git.1646164112.gitgitgadget@gmail.com","subject":"[PATCH v3 2/5] t5318: extract helpers to lib-commit-graph.sh","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2022-03-01T19:48:29Z","receivedAt":"2022-03-01T19:50:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe graph_git_behavior helper is useful for testing that certain Git\ncommands behave the same when using the commit-graph and when not using\nthe commit-graph. Extract it to a new lib-commit-graph.sh file for use\nin new test scripts that will split out from t5318.\n\nWhile doing this extraction, also extract graph_read_expect and the\nlogic for priming the test_oid_cache.\n\nSigned-off-by: Derrick Stolee \n---\n t/lib-commit-graph.sh | 48 +++++++++++++++++++++++++++++++++++++++\n t/t5318-commit-graph.sh | 50 ++---------------------------------------\n 2 files changed, 50 insertions(+), 48 deletions(-)\n create mode 100755 t/lib-commit-graph.sh\n\ndiff --git a/t/lib-commit-graph.sh b/t/lib-commit-graph.sh\nnew file mode 100755\nindex 00000000000..07e12b9d2fe\n--- /dev/null\n+++ b/t/lib-commit-graph.sh\n@@ -0,0 +1,48 @@\n+#!/bin/sh\n+\n+# Helper functions for testing commit-graphs.\n+\n+# Initialize OID cache with oid_version\n+test_oid_cache <<-EOF\n+oid_version sha1:1\n+oid_version sha256:2\n+EOF\n+\n+graph_git_two_modes() {\n+\tgit -c core.commitGraph=true $1 >output &&\n+\tgit -c core.commitGraph=false $1 >expect &&\n+\ttest_cmp expect output\n+}\n+\n+graph_git_behavior() {\n+\tMSG=$1\n+\tDIR=$2\n+\tBRANCH=$3\n+\tCOMPARE=$4\n+\ttest_expect_success \"check normal git operations: $MSG\" '\n+\t\tcd \"$TRASH_DIRECTORY/$DIR\" &&\n+\t\tgraph_git_two_modes \"log --oneline $BRANCH\" &&\n+\t\tgraph_git_two_modes \"log --topo-order $BRANCH\" &&\n+\t\tgraph_git_two_modes \"log --graph $COMPARE..$BRANCH\" &&\n+\t\tgraph_git_two_modes \"branch -vv\" &&\n+\t\tgraph_git_two_modes \"merge-base -a $BRANCH $COMPARE\"\n+\t'\n+}\n+\n+graph_read_expect() {\n+\tOPTIONAL=\"\"\n+\tNUM_CHUNKS=3\n+\tif test -n \"$2\"\n+\tthen\n+\t\tOPTIONAL=\" $2\"\n+\t\tNUM_CHUNKS=$((3 + $(echo \"$2\" | wc -w)))\n+\tfi\n+\tcat >expect <<- EOF\n+\theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n+\tnum_commits: $1\n+\tchunks: oid_fanout oid_lookup commit_metadata$OPTIONAL\n+\toptions:\n+\tEOF\n+\ttest-tool read-graph >output &&\n+\ttest_cmp expect output\n+}\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 2b05026cf6d..9e2b5884dae 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -29,12 +29,7 @@ test_expect_success 'setup full repo' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit init &&\n \tgit config core.commitGraph true &&\n-\tobjdir=\".git/objects\" &&\n-\n-\ttest_oid_cache <<-EOF\n-\toid_version sha1:1\n-\toid_version sha256:2\n-\tEOF\n+\tobjdir=\".git/objects\"\n '\n \n test_expect_success POSIXPERM 'tweak umask for modebit tests' '\n@@ -69,47 +64,10 @@ test_expect_success 'create commits and repack' '\n \tgit repack\n '\n \n-graph_git_two_modes() {\n-\tgit -c core.commitGraph=true $1 >output &&\n-\tgit -c core.commitGraph=false $1 >expect &&\n-\ttest_cmp expect output\n-}\n-\n-graph_git_behavior() {\n-\tMSG=$1\n-\tDIR=$2\n-\tBRANCH=$3\n-\tCOMPARE=$4\n-\ttest_expect_success \"check normal git operations: $MSG\" '\n-\t\tcd \"$TRASH_DIRECTORY/$DIR\" &&\n-\t\tgraph_git_two_modes \"log --oneline $BRANCH\" &&\n-\t\tgraph_git_two_modes \"log --topo-order $BRANCH\" &&\n-\t\tgraph_git_two_modes \"log --graph $COMPARE..$BRANCH\" &&\n-\t\tgraph_git_two_modes \"branch -vv\" &&\n-\t\tgraph_git_two_modes \"merge-base -a $BRANCH $COMPARE\"\n-\t'\n-}\n+. \"$TEST_DIRECTORY\"/lib-commit-graph.sh\n \n graph_git_behavior 'no graph' full commits/3 commits/1\n \n-graph_read_expect() {\n-\tOPTIONAL=\"\"\n-\tNUM_CHUNKS=3\n-\tif test ! -z \"$2\"\n-\tthen\n-\t\tOPTIONAL=\" $2\"\n-\t\tNUM_CHUNKS=$((3 + $(echo \"$2\" | wc -w)))\n-\tfi\n-\tcat >expect <<- EOF\n-\theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n-\tnum_commits: $1\n-\tchunks: oid_fanout oid_lookup commit_metadata$OPTIONAL\n-\toptions:\n-\tEOF\n-\ttest-tool read-graph >output &&\n-\ttest_cmp expect output\n-}\n-\n test_expect_success 'exit with correct error on bad input to --stdin-commits' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \t# invalid, non-hex OID\n@@ -826,10 +784,6 @@ test_expect_success 'set up and verify repo with generation data overflow chunk'\n \tobjdir=\".git/objects\" &&\n \tUNIX_EPOCH_ZERO=\"@0 +0000\" &&\n \tFUTURE_DATE=\"@2147483646 +0000\" &&\n-\ttest_oid_cache <<-EOF &&\n-\toid_version sha1:1\n-\toid_version sha256:2\n-\tEOF\n \tcd \"$TRASH_DIRECTORY\" &&\n \tmkdir repo &&\n \tcd repo &&\n-- \ngitgitgadget\n\n"},{"id":"450003","messageId":"xmqqmti9ph26.fsf@gitster.g","threadId":"57472","inReplyTo":"562341b76b30ff3cdc01cea4cda550a7b8165bdd.1646164112.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 3/5] commit-graph: fix ordering bug in generation numbers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-01T20:13:21Z","receivedAt":"2022-03-01T20:13:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> When computing the generation numbers for a commit-graph, we compute\n> the corrected commit dates and then check if their offsets from the\n> actual dates is too large to fit in the 32-bit Generation Data chunk.\n> However, there is a problem with this approach: if we have parsed the\n> generation data from the previous commit-graph, then we continue the\n> loop because the corrected commit date is already computed. This causes\n> an under-count in the number of overflow values.\n>\n> It is incorrect to add an increment to num_generation_data_overflows\n> next to this 'continue' statement, because we might start\n> double-counting commits that are computed because of the depth-first\n> search walk from a commit with an earlier OID.\n>\n> Instead, iterate over the full commit list at the end, checking the\n> offsets to see how many grow beyond the maximum value.\n\nOK.\n\n> Create a new t5328-commit-graph-64-bit-time.sh test script to handle\n> special cases of testing 64-bit timestampes. This helps demonstrate this\n> bug in more cases. It still won't hit all potential cases until the next\n> change, which reenables reading generation numbers. Use the skip_all\n> trick from 0a2bfccb9c8 (t0051: use \"skip_all\" under !MINGW in\n> single-test file, 2022-02-04) to make the output clean when run on a\n> 32-bit system.\n>\n> Hepled-by: Ævar Arnfjörð Bjarmason \n\nI can typofix this one locally if needed.\n"},{"id":"450015","messageId":"xmqqbkyppg9r.fsf@gitster.g","threadId":"57472","inReplyTo":"xmqqmti9ph26.fsf@gitster.g","subject":"Re: [PATCH v3 3/5] commit-graph: fix ordering bug in generation numbers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-01T20:30:24Z","receivedAt":"2022-03-01T20:31:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> Create a new t5328-commit-graph-64-bit-time.sh test script to handle\n>> special cases of testing 64-bit timestampes. This helps demonstrate this\n>> bug in more cases. It still won't hit all potential cases until the next\n>> change, which reenables reading generation numbers. Use the skip_all\n>> trick from 0a2bfccb9c8 (t0051: use \"skip_all\" under !MINGW in\n>> single-test file, 2022-02-04) to make the output clean when run on a\n>> 32-bit system.\n>>\n>> Hepled-by: Ævar Arnfjörð Bjarmason \n>\n> I can typofix this one locally if needed.\n\nWhat I meant was s/timestampes/timestamps/ and s/Hepled/Helped/.\n"},{"id":"450102","messageId":"Yh93vOkt2DkrGPh2@ncase","threadId":"57472","inReplyTo":"1b9912f7-87be-2520-bb53-9e23529ad233@github.com","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-03-02T13:57:16Z","receivedAt":"2022-03-02T13:57:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Mar 01, 2022 at 10:25:46AM -0500, Derrick Stolee wrote:\n> On 3/1/2022 9:53 AM, Patrick Steinhardt wrote:\n> > On Tue, Mar 01, 2022 at 09:06:44AM -0500, Derrick Stolee wrote:\n> >> On 3/1/2022 5:35 AM, Patrick Steinhardt wrote:\n> >>> On Tue, Mar 01, 2022 at 10:46:14AM +0100, Patrick Steinhardt wrote:\n> >>>> On Mon, Feb 28, 2022 at 01:44:01PM -0500, Derrick Stolee wrote:\n> >>>>> On 2/28/2022 11:59 AM, Patrick Steinhardt wrote:\n> >>>>>> On Mon, Feb 28, 2022 at 11:23:38AM -0500, Derrick Stolee wrote:\n> >>>>>>> On 2/28/2022 10:18 AM, Patrick Steinhardt wrote:\n> >>>>>>>> [1]: https://gitlab.com/gitlab-com/www-gitlab-com.git\n> ...\n> >>> So the question is whether this is a change that needs to be rolled out\n> >>> over multiple releases. First we'd get in the bug fix such that we write\n> >>> correct commit-graphs, and after this fix has been released we can also\n> >>> release the fix that starts to actually parse the generation. This\n> >>> ensures there's a grace period during which we can hopefully correct the\n> >>> data on-disk such that users are not faced with failures.\n> >>\n> >> You are right that we need to be careful here, but I also think that\n> >> previous versions of Git always wrote the correct data. Here is my\n> >> thought process:\n> >>\n> >> 1. To get this bug, we need to have parsed the corrected commit date\n> >> from an existing commit-graph in order to under-count the number\n> >> of overflow values.\n> >>\n> >> 2. Before this series, Git versions were not parsing the corrected\n> >> commit date, so they recompute the corrected commit date every\n> >> time the commit-graph is written, getting the proper count of\n> >> overflow values.\n> >>\n> >> For these reasons, data written by previous versions of Git are\n> >> correct and can be trusted without a staged release.\n> >>\n> >> Does this make sense? Or, do you experience a different result when\n> >> you build commit-graphs with a released Git version and then when\n> >> writing on top with all patches applied?\n> > \n> > Just to verify my understanding: you claim that the bug I was hitting\n> > shouldn't be encountered in the wild when the release , but\n> > only if one were to write a commit-graph with the intermediate stafe\n> > until patch 3/4 of your patch series?\n> \n> That is my claim. And my testing of the repo at [1] has demonstrated\n> that it works correctly in these cases.\n> \n> > Hum. I have re-verified, and this indeed seems to play out. So I must've\n> > accidentally ran all my testing with the state generated without the\n> > final patch which fixes the corruption. I do see lots of the following\n> > warnings, but overall I can verify and write the commit-graph just fine:\n> > \n> > commit-graph generation for commit c80a42de8803e2d77818d0c82f88e748d7f9425f is 1623362063 < 1623362139\n> \n> But I'm not able to generate these warnings from either version. I\n> tried generating different levels of a split commit-graph, but\n> could not reproduce it. If you have reproduction steps using current\n> 'master' (or any released Git version) and the four patches here,\n> then I would love to get a full understanding of your errors.\n> \n> Thanks,\n> -Stolee\n\nI haven't yet been able to reproduce it with publicly available data,\nbut with the internal references I'm able to evoke the warnings\nreliably. It only works when I have two repositories connected via\nalternates, when generating the commit-graph in the linked-to repo\nfirst, and then generating the commit-graph in the linking repo.\n\nThe following recipe allows me to reproduce, but rely on private data:\n\n $ git --version\n git version 2.35.1\n\n # The pool repository is the one we're linked to from the fork.\n $ cd \"$pool\"\n $ rm -rf objects/info/commit-graph objects/info/commit-graph\n $ git commit-graph write --split\n\n $ cd \"$fork\"\n $ rm -rf objects/info/commit-graph objects/info/commit-graph\n $ git commit-graph write --split\n\n $ git commit-graph verify --no-progress\n $ echo $?\n 0\n\n # This is 715d08a9e51251ad8290b181b6ac3b9e1f9719d7 with your full v2\n # applied on top.\n $ ~/Development/git/bin-wrappers/git --version\n git version 2.35.1.358.g7ede1bea24\n\n $ ~/Development/git/bin-wrappers/git commit-graph verify --no-progress\n commit-graph generation for commit 06a91bac00ed11128becd48d5ae77eacd8f24c97 is 1623273624 < 1623273710\n commit-graph generation for commit 0ae91029f27238e8f8e109c6bb3907f864dda14f is 1622151146 < 1622151220\n commit-graph generation for commit 0d4582a33d8c8e3eb01adbf564f5e1deeb3b56a2 is 1631045222 < 1631045225\n commit-graph generation for commit 0daf8976439d7e0bb9710c5ee63b570580e0dc03 is 1620347739 < 1620347789\n commit-graph generation for commit 0e0ee8ffb3fa22cee7d28e21cbd6df26454932cf is 1623783297 < 1623783380\n commit-graph generation for commit 0f08ab3de6ec115ea8a956a1996cb9759e640e74 is 1621543278 < 1621543339\n commit-graph generation for commit 133ed0319b5a66ae0c2be76e5a887b880452b111 is 1620949864 < 1620949915\n commit-graph generation for commit 1341b3e6c63343ae94a8a473fa057126ddd4669a is 1637344364 < 1637344384\n commit-graph generation for commit 15bdfc501c2c9f23e9353bf6e6a5facd9c32a07a is 1623348103 < 1623348133\n ...\n $ echo $?\n 1\n\nWhen generating commit-graphs with your patches applied the `verify`\nstep works alright.\n\nI've also by accident stumbled over the original error again:\n\n fatal: commit-graph requires overflow generation data but has none\n\nThis time it's definitely not caused by generating commit-graphs with an\nin-between state of your patch series because the data comes straight\nfrom production with no changes to the commit-graphs performed by\nmyself. There we're running Git v2.33.1 with a couple of backported\npatches (see [1]). While those patches cause us to make more use of the\ncommit-graph, none modify the way we generate them.\n\nOf note is that the commit-graph contains references to commits which\ndon't exist in the ODB anymore.\n\nPatrick\n\n[1]: https://gitlab.com/gitlab-org/gitlab-git/-/commits/pks-v2.33.1.gl3\n"},{"id":"450105","messageId":"f21f854d-3fe6-a0f9-1a7f-4396457e70fd@github.com","threadId":"57472","inReplyTo":"xmqqbkyppg9r.fsf@gitster.g","subject":"Re: [PATCH v3 3/5] commit-graph: fix ordering bug in generation numbers","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-02T14:13:24Z","receivedAt":"2022-03-02T14:13:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/1/2022 3:30 PM, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n>>> Create a new t5328-commit-graph-64-bit-time.sh test script to handle\n>>> special cases of testing 64-bit timestampes. This helps demonstrate this\n>>> bug in more cases. It still won't hit all potential cases until the next\n>>> change, which reenables reading generation numbers. Use the skip_all\n>>> trick from 0a2bfccb9c8 (t0051: use \"skip_all\" under !MINGW in\n>>> single-test file, 2022-02-04) to make the output clean when run on a\n>>> 32-bit system.\n>>>\n>>> Hepled-by: Ævar Arnfjörð Bjarmason \n>>\n>> I can typofix this one locally if needed.\n> \n> What I meant was s/timestampes/timestamps/ and s/Hepled/Helped/.\n\nThank you!\n-Stolee\n"},{"id":"450109","messageId":"db5eb248-1b54-9f24-8f8a-28c19a1eee6a@github.com","threadId":"57472","inReplyTo":"Yh93vOkt2DkrGPh2@ncase","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-02T14:57:17Z","receivedAt":"2022-03-02T14:57:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/2/2022 8:57 AM, Patrick Steinhardt wrote:\n> On Tue, Mar 01, 2022 at 10:25:46AM -0500, Derrick Stolee wrote:\n>> On 3/1/2022 9:53 AM, Patrick Steinhardt wrote:\n\n>>> Hum. I have re-verified, and this indeed seems to play out. So I must've\n>>> accidentally ran all my testing with the state generated without the\n>>> final patch which fixes the corruption. I do see lots of the following\n>>> warnings, but overall I can verify and write the commit-graph just fine:\n>>>\n>>> commit-graph generation for commit c80a42de8803e2d77818d0c82f88e748d7f9425f is 1623362063 < 1623362139\n>>\n>> But I'm not able to generate these warnings from either version. I\n>> tried generating different levels of a split commit-graph, but\n>> could not reproduce it. If you have reproduction steps using current\n>> 'master' (or any released Git version) and the four patches here,\n>> then I would love to get a full understanding of your errors.\n>>\n>> Thanks,\n>> -Stolee\n> \n> I haven't yet been able to reproduce it with publicly available data,\n> but with the internal references I'm able to evoke the warnings\n> reliably. It only works when I have two repositories connected via\n> alternates, when generating the commit-graph in the linked-to repo\n> first, and then generating the commit-graph in the linking repo.\n> \n> The following recipe allows me to reproduce, but rely on private data:\n> \n> $ git --version\n> git version 2.35.1\n> \n> # The pool repository is the one we're linked to from the fork.\n> $ cd \"$pool\"\n> $ rm -rf objects/info/commit-graph objects/info/commit-graph\n> $ git commit-graph write --split\n> \n> $ cd \"$fork\"\n> $ rm -rf objects/info/commit-graph objects/info/commit-graph\n> $ git commit-graph write --split\n> \n> $ git commit-graph verify --no-progress\n> $ echo $?\n> 0\n> \n> # This is 715d08a9e51251ad8290b181b6ac3b9e1f9719d7 with your full v2\n> # applied on top.\n> $ ~/Development/git/bin-wrappers/git --version\n> git version 2.35.1.358.g7ede1bea24\n> \n> $ ~/Development/git/bin-wrappers/git commit-graph verify --no-progress\n> commit-graph generation for commit 06a91bac00ed11128becd48d5ae77eacd8f24c97 is 1623273624 < 1623273710\n> commit-graph generation for commit 0ae91029f27238e8f8e109c6bb3907f864dda14f is 1622151146 < 1622151220\n> commit-graph generation for commit 0d4582a33d8c8e3eb01adbf564f5e1deeb3b56a2 is 1631045222 < 1631045225\n> commit-graph generation for commit 0daf8976439d7e0bb9710c5ee63b570580e0dc03 is 1620347739 < 1620347789\n> commit-graph generation for commit 0e0ee8ffb3fa22cee7d28e21cbd6df26454932cf is 1623783297 < 1623783380\n> commit-graph generation for commit 0f08ab3de6ec115ea8a956a1996cb9759e640e74 is 1621543278 < 1621543339\n> commit-graph generation for commit 133ed0319b5a66ae0c2be76e5a887b880452b111 is 1620949864 < 1620949915\n> commit-graph generation for commit 1341b3e6c63343ae94a8a473fa057126ddd4669a is 1637344364 < 1637344384\n> commit-graph generation for commit 15bdfc501c2c9f23e9353bf6e6a5facd9c32a07a is 1623348103 < 1623348133\n> ...\n> $ echo $?\n> 1\n> \n> When generating commit-graphs with your patches applied the `verify`\n> step works alright.\n> \n> I've also by accident stumbled over the original error again:\n> \n> fatal: commit-graph requires overflow generation data but has none\n> \n> This time it's definitely not caused by generating commit-graphs with an\n> in-between state of your patch series because the data comes straight\n> from production with no changes to the commit-graphs performed by\n> myself. There we're running Git v2.33.1 with a couple of backported\n> patches (see [1]). While those patches cause us to make more use of the\n> commit-graph, none modify the way we generate them.\n> \n> Of note is that the commit-graph contains references to commits which\n> don't exist in the ODB anymore.\n> \n> Patrick\n> \n> [1]: https://gitlab.com/gitlab-org/gitlab-git/-/commits/pks-v2.33.1.gl3\n\nThank you for your diligence here, Patrick. I really appreciate the\nwork you're putting in to verify the situation.\n\nSince our repro relies on private information, but is consistent, I\nwonder if we should take the patch below, which starts to ignore the\nolder generation number v2 data and only writes freshly-computed\nnumbers.\n\nThanks,\n-Stolee\n\n--- 8< ---\n\nFrom c53d8bd52bbcab3862e8a826ee75692edc7e4173 Mon Sep 17 00:00:00 2001\nFrom: Derrick Stolee \nDate: Wed, 2 Mar 2022 09:45:13 -0500\nSubject: [PATCH v3 5/4] commit-graph: declare bankruptcy on GDAT chunks\n\nThe Generation Data (GDAT) and Generation Data Overflow (GDOV) chunks\nstore corrected commit date offsets, used for generation number v2.\nRecent changes have demonstrated that previous versions of Git were\nincorrectly parsing data from these chunks, but might have also been\nwriting them incorrectly.\n\nI asserted [1] that the previous fixes were sufficient because the known\nreasons for incorrectly writing generation number v2 data relied on\nparsing the information incorrectly out of a commit-graph file, but the\nprevious versions of Git were not reading the generation number v2 data.\n\nHowever, Patrick demonstrated [2] a case where in split commit-graphs\nacross an alternate boundary (and possibly some other special\nconditions) it was possible to have a commit-graph that was generated by\na previous version of Git have incorrect generation number v2 data which\nresults in errors like the following:\n\n commit-graph generation for commit is 1623273624 < 1623273710\n\n[1] https://lore.kernel.org/git/f50e74f0-9ffa-f4f2-4663-269801495ed3@github.com/\n[2] https://lore.kernel.org/git/Yh93vOkt2DkrGPh2@ncase/\n\nClearly, there is something else going on. The situation is not\ncompletely understood, but the errors do not reproduce if the\ncommit-graphs are all generated by a Git version including these recent\nfixes.\n\nIf we cannot trust the existing data in the GDAT and GDOV chunks, then\nwe can alter the format to change the chunk IDs for these chunks. This\ncauses the new version of Git to silently ignore the older chunks (and\ndisabling generation number v2 in the process) while writing new\ncommit-graph files with correct data in the GDA2 and GDO2 chunks.\n\nUpdate commit-graph-format.txt including a historical note about these\ndeprecated chunks.\n\nReported-by: Patrick Steinhardt \nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph-format.txt | 12 ++++++++++--\n commit-graph.c | 4 ++--\n 2 files changed, 12 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex 87971c27dd7..484b185ba98 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -93,7 +93,7 @@ CHUNK DATA:\n 2 bits of the lowest byte, storing the 33rd and 34th bit of the\n commit time.\n \n- Generation Data (ID: {'G', 'D', 'A', 'T' }) (N * 4 bytes) [Optional]\n+ Generation Data (ID: {'G', 'D', 'A', '2' }) (N * 4 bytes) [Optional]\n * This list of 4-byte values store corrected commit date offsets for the\n commits, arranged in the same order as commit data chunk.\n * If the corrected commit date offset cannot be stored within 31 bits,\n@@ -104,7 +104,7 @@ CHUNK DATA:\n by compatible versions of Git and in case of split commit-graph chains,\n the topmost layer also has Generation Data chunk.\n \n- Generation Data Overflow (ID: {'G', 'D', 'O', 'V' }) [Optional]\n+ Generation Data Overflow (ID: {'G', 'D', 'O', '2' }) [Optional]\n * This list of 8-byte values stores the corrected commit date offsets\n for commits with corrected commit date offsets that cannot be\n stored within 31 bits.\n@@ -156,3 +156,11 @@ CHUNK DATA:\n TRAILER:\n \n \tH-byte HASH-checksum of all of the above.\n+\n+== Historical Notes:\n+\n+The Generation Data (GDA2) and Generation Data Overflow (GDO2) chunks have\n+the number '2' in their chunk IDs because a previous version of Git wrote\n+possibly erroneous data in these chunks with the IDs \"GDAT\" and \"GDOV\". By\n+changing the IDs, newer versions of Git will silently ignore those older\n+chunks and write the new information without trusting the incorrect data.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex b86a6a634fe..fb2ced0bd6d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -39,8 +39,8 @@ void git_test_write_commit_graph_or_die(void)\n #define GRAPH_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n #define GRAPH_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define GRAPH_CHUNKID_DATA 0x43444154 /* \"CDAT\" */\n-#define GRAPH_CHUNKID_GENERATION_DATA 0x47444154 /* \"GDAT\" */\n-#define GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW 0x47444f56 /* \"GDOV\" */\n+#define GRAPH_CHUNKID_GENERATION_DATA 0x47444132 /* \"GDA2\" */\n+#define GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW 0x47444f32 /* \"GDO2\" */\n #define GRAPH_CHUNKID_EXTRAEDGES 0x45444745 /* \"EDGE\" */\n #define GRAPH_CHUNKID_BLOOMINDEXES 0x42494458 /* \"BIDX\" */\n #define GRAPH_CHUNKID_BLOOMDATA 0x42444154 /* \"BDAT\" */\n-- \n2.35.1.138.gfc5de29e9e6\n\n\n\n\n"},{"id":"450145","messageId":"xmqqh78gmdap.fsf@gitster.g","threadId":"57472","inReplyTo":"db5eb248-1b54-9f24-8f8a-28c19a1eee6a@github.com","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-02T18:15:10Z","receivedAt":"2022-03-02T18:15:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> Since our repro relies on private information, but is consistent, I\n> wonder if we should take the patch below, which starts to ignore the\n> older generation number v2 data and only writes freshly-computed\n> numbers.\n\n;-)\n\n> Clearly, there is something else going on. The situation is not\n> completely understood, but the errors do not reproduce if the\n> commit-graphs are all generated by a Git version including these recent\n> fixes.\n\nDo you mean \"we know doing X and then Y and then Z on this\nparticular private data with older version of Git without those two\nfixes will lead to a broken timestamp, but doing exactly the same\nwith the two fixes, the breakage does not reproduce\"? If so, that\nis quite encouraging news. Thanks for working well together.\n\n> If we cannot trust the existing data in the GDAT and GDOV chunks, then\n> we can alter the format to change the chunk IDs for these chunks. This\n> causes the new version of Git to silently ignore the older chunks (and\n> disabling generation number v2 in the process) while writing new\n> commit-graph files with correct data in the GDA2 and GDO2 chunks.\n>\n> Update commit-graph-format.txt including a historical note about these\n> deprecated chunks.\n\nSensible.\n\n> @@ -156,3 +156,11 @@ CHUNK DATA:\n> TRAILER:\n> \n> \tH-byte HASH-checksum of all of the above.\n> +\n> +== Historical Notes:\n> +\n> +The Generation Data (GDA2) and Generation Data Overflow (GDO2) chunks have\n> +the number '2' in their chunk IDs because a previous version of Git wrote\n> +possibly erroneous data in these chunks with the IDs \"GDAT\" and \"GDOV\". By\n> +changing the IDs, newer versions of Git will silently ignore those older\n> +chunks and write the new information without trusting the incorrect data.\n\nGood. How does a new version of Git skip and ignore GDAT and GDOV\nin existing files? By not having any code to recognize what they\nare?\n\nI am wondering if there is some notion of \"if you do not understand\nwhat this chunk is, you are incapable of handling this file\ncorrectly, so do not use it\" kind of bit per chunks (similar to the\nindex extensions where ones that begin with [A-Z] are optional) that\nmay negatively affect this plan.\n\nThanks.\n\n> diff --git a/commit-graph.c b/commit-graph.c\n> index b86a6a634fe..fb2ced0bd6d 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -39,8 +39,8 @@ void git_test_write_commit_graph_or_die(void)\n> #define GRAPH_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n> #define GRAPH_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n> #define GRAPH_CHUNKID_DATA 0x43444154 /* \"CDAT\" */\n> -#define GRAPH_CHUNKID_GENERATION_DATA 0x47444154 /* \"GDAT\" */\n> -#define GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW 0x47444f56 /* \"GDOV\" */\n> +#define GRAPH_CHUNKID_GENERATION_DATA 0x47444132 /* \"GDA2\" */\n> +#define GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW 0x47444f32 /* \"GDO2\" */\n> #define GRAPH_CHUNKID_EXTRAEDGES 0x45444745 /* \"EDGE\" */\n> #define GRAPH_CHUNKID_BLOOMINDEXES 0x42494458 /* \"BIDX\" */\n> #define GRAPH_CHUNKID_BLOOMDATA 0x42444154 /* \"BDAT\" */\n"},{"id":"450146","messageId":"1651a34e-38c1-fb8a-df59-1dcd8a8389b5@github.com","threadId":"57472","inReplyTo":"xmqqh78gmdap.fsf@gitster.g","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-02T18:46:43Z","receivedAt":"2022-03-02T18:46:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/2/2022 1:15 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>> Since our repro relies on private information, but is consistent, I\n>> wonder if we should take the patch below, which starts to ignore the\n>> older generation number v2 data and only writes freshly-computed\n>> numbers.\n> \n> ;-)\n> \n>> Clearly, there is something else going on. The situation is not\n>> completely understood, but the errors do not reproduce if the\n>> commit-graphs are all generated by a Git version including these recent\n>> fixes.\n> \n> Do you mean \"we know doing X and then Y and then Z on this\n> particular private data with older version of Git without those two\n> fixes will lead to a broken timestamp, but doing exactly the same\n> with the two fixes, the breakage does not reproduce\"? If so, that\n> is quite encouraging news. Thanks for working well together.\n\nYes, that is my understanding.\n\n>> If we cannot trust the existing data in the GDAT and GDOV chunks, then\n>> we can alter the format to change the chunk IDs for these chunks. This\n>> causes the new version of Git to silently ignore the older chunks (and\n>> disabling generation number v2 in the process) while writing new\n>> commit-graph files with correct data in the GDA2 and GDO2 chunks.\n>>\n>> Update commit-graph-format.txt including a historical note about these\n>> deprecated chunks.\n> \n> Sensible.\n> \n>> @@ -156,3 +156,11 @@ CHUNK DATA:\n>> TRAILER:\n>> \n>> \tH-byte HASH-checksum of all of the above.\n>> +\n>> +== Historical Notes:\n>> +\n>> +The Generation Data (GDA2) and Generation Data Overflow (GDO2) chunks have\n>> +the number '2' in their chunk IDs because a previous version of Git wrote\n>> +possibly erroneous data in these chunks with the IDs \"GDAT\" and \"GDOV\". By\n>> +changing the IDs, newer versions of Git will silently ignore those older\n>> +chunks and write the new information without trusting the incorrect data.\n> \n> Good. How does a new version of Git skip and ignore GDAT and GDOV\n> in existing files? By not having any code to recognize what they\n> are?\n> \n> I am wondering if there is some notion of \"if you do not understand\n> what this chunk is, you are incapable of handling this file\n> correctly, so do not use it\" kind of bit per chunks (similar to the\n> index extensions where ones that begin with [A-Z] are optional) that\n> may negatively affect this plan.\n\nThe chunk IDs do not have this special casing rule. This is a\nbit unfortunate for certain cases like adding something that _must_\nbe understood. Here, it works to our benefit that GDAT and GDOV are\noptional and can be safely ignored. Thus, clients with this patch\nwill ignore GDAT and GDOV and continue using topological levels\nform the CDAT chunk. Older clients without this patch will ignore\nthe new GDA2 and GDO2 chunks and continue using topological levels.\n\nFor Git versions without this topic branch, this \"continue using\ntopological levels\" means no change of behavior at all.\n\nThanks,\n-Stolee\n"},{"id":"450190","messageId":"xmqqo82ogend.fsf@gitster.g","threadId":"57472","inReplyTo":"1651a34e-38c1-fb8a-df59-1dcd8a8389b5@github.com","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-02T22:42:30Z","receivedAt":"2022-03-02T22:42:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>> I am wondering if there is some notion of \"if you do not understand\n>> what this chunk is, you are incapable of handling this file\n>> correctly, so do not use it\" kind of bit per chunks (similar to the\n>> index extensions where ones that begin with [A-Z] are optional) that\n>> may negatively affect this plan.\n>\n> The chunk IDs do not have this special casing rule. This is a\n> bit unfortunate for certain cases like adding something that _must_\n> be understood. Here, it works to our benefit that GDAT and GDOV are\n> optional and can be safely ignored. Thus, clients with this patch\n> will ignore GDAT and GDOV and continue using topological levels\n> form the CDAT chunk. Older clients without this patch will ignore\n> the new GDA2 and GDO2 chunks and continue using topological levels.\n>\n> For Git versions without this topic branch, this \"continue using\n> topological levels\" means no change of behavior at all.\n\nExcellent. Thanks.\n"},{"id":"450259","messageId":"YiCkRLFxn8Pok7Kc@ncase","threadId":"57472","inReplyTo":"db5eb248-1b54-9f24-8f8a-28c19a1eee6a@github.com","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-03-03T11:19:32Z","receivedAt":"2022-03-03T11:19:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Mar 02, 2022 at 09:57:17AM -0500, Derrick Stolee wrote:\n> On 3/2/2022 8:57 AM, Patrick Steinhardt wrote:\n> > On Tue, Mar 01, 2022 at 10:25:46AM -0500, Derrick Stolee wrote:\n> >> On 3/1/2022 9:53 AM, Patrick Steinhardt wrote:\n> \n> >>> Hum. I have re-verified, and this indeed seems to play out. So I must've\n> >>> accidentally ran all my testing with the state generated without the\n> >>> final patch which fixes the corruption. I do see lots of the following\n> >>> warnings, but overall I can verify and write the commit-graph just fine:\n> >>>\n> >>> commit-graph generation for commit c80a42de8803e2d77818d0c82f88e748d7f9425f is 1623362063 < 1623362139\n> >>\n> >> But I'm not able to generate these warnings from either version. I\n> >> tried generating different levels of a split commit-graph, but\n> >> could not reproduce it. If you have reproduction steps using current\n> >> 'master' (or any released Git version) and the four patches here,\n> >> then I would love to get a full understanding of your errors.\n> >>\n> >> Thanks,\n> >> -Stolee\n> > \n> > I haven't yet been able to reproduce it with publicly available data,\n> > but with the internal references I'm able to evoke the warnings\n> > reliably. It only works when I have two repositories connected via\n> > alternates, when generating the commit-graph in the linked-to repo\n> > first, and then generating the commit-graph in the linking repo.\n> > \n> > The following recipe allows me to reproduce, but rely on private data:\n> > \n> > $ git --version\n> > git version 2.35.1\n> > \n> > # The pool repository is the one we're linked to from the fork.\n> > $ cd \"$pool\"\n> > $ rm -rf objects/info/commit-graph objects/info/commit-graph\n> > $ git commit-graph write --split\n> > \n> > $ cd \"$fork\"\n> > $ rm -rf objects/info/commit-graph objects/info/commit-graph\n> > $ git commit-graph write --split\n> > \n> > $ git commit-graph verify --no-progress\n> > $ echo $?\n> > 0\n> > \n> > # This is 715d08a9e51251ad8290b181b6ac3b9e1f9719d7 with your full v2\n> > # applied on top.\n> > $ ~/Development/git/bin-wrappers/git --version\n> > git version 2.35.1.358.g7ede1bea24\n> > \n> > $ ~/Development/git/bin-wrappers/git commit-graph verify --no-progress\n> > commit-graph generation for commit 06a91bac00ed11128becd48d5ae77eacd8f24c97 is 1623273624 < 1623273710\n> > commit-graph generation for commit 0ae91029f27238e8f8e109c6bb3907f864dda14f is 1622151146 < 1622151220\n> > commit-graph generation for commit 0d4582a33d8c8e3eb01adbf564f5e1deeb3b56a2 is 1631045222 < 1631045225\n> > commit-graph generation for commit 0daf8976439d7e0bb9710c5ee63b570580e0dc03 is 1620347739 < 1620347789\n> > commit-graph generation for commit 0e0ee8ffb3fa22cee7d28e21cbd6df26454932cf is 1623783297 < 1623783380\n> > commit-graph generation for commit 0f08ab3de6ec115ea8a956a1996cb9759e640e74 is 1621543278 < 1621543339\n> > commit-graph generation for commit 133ed0319b5a66ae0c2be76e5a887b880452b111 is 1620949864 < 1620949915\n> > commit-graph generation for commit 1341b3e6c63343ae94a8a473fa057126ddd4669a is 1637344364 < 1637344384\n> > commit-graph generation for commit 15bdfc501c2c9f23e9353bf6e6a5facd9c32a07a is 1623348103 < 1623348133\n> > ...\n> > $ echo $?\n> > 1\n> > \n> > When generating commit-graphs with your patches applied the `verify`\n> > step works alright.\n> > \n> > I've also by accident stumbled over the original error again:\n> > \n> > fatal: commit-graph requires overflow generation data but has none\n> > \n> > This time it's definitely not caused by generating commit-graphs with an\n> > in-between state of your patch series because the data comes straight\n> > from production with no changes to the commit-graphs performed by\n> > myself. There we're running Git v2.33.1 with a couple of backported\n> > patches (see [1]). While those patches cause us to make more use of the\n> > commit-graph, none modify the way we generate them.\n> > \n> > Of note is that the commit-graph contains references to commits which\n> > don't exist in the ODB anymore.\n> > \n> > Patrick\n> > \n> > [1]: https://gitlab.com/gitlab-org/gitlab-git/-/commits/pks-v2.33.1.gl3\n> \n> Thank you for your diligence here, Patrick. I really appreciate the\n> work you're putting in to verify the situation.\n> \n> Since our repro relies on private information, but is consistent, I\n> wonder if we should take the patch below, which starts to ignore the\n> older generation number v2 data and only writes freshly-computed\n> numbers.\n> \n> Thanks,\n> -Stolee\n\nThanks. With your patch below the `fatal:` error is gone, but I'm still\nseeing the same errors with regards to the commit-graph generations.\n\nSo to summarize my findings:\n\n - This bug occurs when writing commit-graphs with v2.35.1, but\n reading them with your patches.\n\n - This bug occurs when I have two repositories connected via an\n alternates file. I haven't yet been able to reproduce it in a\n single repository that is not connected to a separate ODB.\n\n - This bug only occurs when I first generate the commit-graph in the\n repository I'm borrowing objects from.\n\n - This bug only occurs when I write commit-graphs with `--split` in\n both repositories. \"Normal\" commit-graphs don't have this issue,\n and neither can I see it with `--split=replace` or mixed-type\n commit-graphs.\n\nBeware, the following explanation is based on my very basic\nunderstanding of the commit-graph code and thus more likely to be wrong\nthan right:\n\nWith the old Git version, we've been mis-parsing the generation because\n`read_generation_data` wasn't ever set. As a result it can happen that\nthe second split commit-graph we're generating computes its own\ngeneration numbers from the wrong starting point because it uses the\nmis-parsed generation numbers from the parent commit-graph.\n\nWith your patches, we start to correctly account for overflows and would\nthus end up with a different value for the generation depending on where\nwe parse the commit from: if we parse it from the first commit-graph it\nwould be correct because it's contains the \"root\" of the generation\nnumbers. But if we parse a commit from the second commit-graph we may\nhave a mismatch because the generation numbers in there may have been\nderived from generation numbers mis-parsed from the first commit-graph.\nAnd because these would be wrong in case there was an overflow it is\nclear that the new corrected generation number may be wrong, as well.\n\nPatrick\n\n> --- 8< ---\n> \n> From c53d8bd52bbcab3862e8a826ee75692edc7e4173 Mon Sep 17 00:00:00 2001\n> From: Derrick Stolee \n> Date: Wed, 2 Mar 2022 09:45:13 -0500\n> Subject: [PATCH v3 5/4] commit-graph: declare bankruptcy on GDAT chunks\n> \n> The Generation Data (GDAT) and Generation Data Overflow (GDOV) chunks\n> store corrected commit date offsets, used for generation number v2.\n> Recent changes have demonstrated that previous versions of Git were\n> incorrectly parsing data from these chunks, but might have also been\n> writing them incorrectly.\n> \n> I asserted [1] that the previous fixes were sufficient because the known\n> reasons for incorrectly writing generation number v2 data relied on\n> parsing the information incorrectly out of a commit-graph file, but the\n> previous versions of Git were not reading the generation number v2 data.\n> \n> However, Patrick demonstrated [2] a case where in split commit-graphs\n> across an alternate boundary (and possibly some other special\n> conditions) it was possible to have a commit-graph that was generated by\n> a previous version of Git have incorrect generation number v2 data which\n> results in errors like the following:\n> \n> commit-graph generation for commit is 1623273624 < 1623273710\n> \n> [1] https://lore.kernel.org/git/f50e74f0-9ffa-f4f2-4663-269801495ed3@github.com/\n> [2] https://lore.kernel.org/git/Yh93vOkt2DkrGPh2@ncase/\n> \n> Clearly, there is something else going on. The situation is not\n> completely understood, but the errors do not reproduce if the\n> commit-graphs are all generated by a Git version including these recent\n> fixes.\n> \n> If we cannot trust the existing data in the GDAT and GDOV chunks, then\n> we can alter the format to change the chunk IDs for these chunks. This\n> causes the new version of Git to silently ignore the older chunks (and\n> disabling generation number v2 in the process) while writing new\n> commit-graph files with correct data in the GDA2 and GDO2 chunks.\n> \n> Update commit-graph-format.txt including a historical note about these\n> deprecated chunks.\n> \n> Reported-by: Patrick Steinhardt \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/technical/commit-graph-format.txt | 12 ++++++++++--\n> commit-graph.c | 4 ++--\n> 2 files changed, 12 insertions(+), 4 deletions(-)\n> \n> diff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\n> index 87971c27dd7..484b185ba98 100644\n> --- a/Documentation/technical/commit-graph-format.txt\n> +++ b/Documentation/technical/commit-graph-format.txt\n> @@ -93,7 +93,7 @@ CHUNK DATA:\n> 2 bits of the lowest byte, storing the 33rd and 34th bit of the\n> commit time.\n> \n> - Generation Data (ID: {'G', 'D', 'A', 'T' }) (N * 4 bytes) [Optional]\n> + Generation Data (ID: {'G', 'D', 'A', '2' }) (N * 4 bytes) [Optional]\n> * This list of 4-byte values store corrected commit date offsets for the\n> commits, arranged in the same order as commit data chunk.\n> * If the corrected commit date offset cannot be stored within 31 bits,\n> @@ -104,7 +104,7 @@ CHUNK DATA:\n> by compatible versions of Git and in case of split commit-graph chains,\n> the topmost layer also has Generation Data chunk.\n> \n> - Generation Data Overflow (ID: {'G', 'D', 'O', 'V' }) [Optional]\n> + Generation Data Overflow (ID: {'G', 'D', 'O', '2' }) [Optional]\n> * This list of 8-byte values stores the corrected commit date offsets\n> for commits with corrected commit date offsets that cannot be\n> stored within 31 bits.\n> @@ -156,3 +156,11 @@ CHUNK DATA:\n> TRAILER:\n> \n> \tH-byte HASH-checksum of all of the above.\n> +\n> +== Historical Notes:\n> +\n> +The Generation Data (GDA2) and Generation Data Overflow (GDO2) chunks have\n> +the number '2' in their chunk IDs because a previous version of Git wrote\n> +possibly erroneous data in these chunks with the IDs \"GDAT\" and \"GDOV\". By\n> +changing the IDs, newer versions of Git will silently ignore those older\n> +chunks and write the new information without trusting the incorrect data.\n> diff --git a/commit-graph.c b/commit-graph.c\n> index b86a6a634fe..fb2ced0bd6d 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -39,8 +39,8 @@ void git_test_write_commit_graph_or_die(void)\n> #define GRAPH_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n> #define GRAPH_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n> #define GRAPH_CHUNKID_DATA 0x43444154 /* \"CDAT\" */\n> -#define GRAPH_CHUNKID_GENERATION_DATA 0x47444154 /* \"GDAT\" */\n> -#define GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW 0x47444f56 /* \"GDOV\" */\n> +#define GRAPH_CHUNKID_GENERATION_DATA 0x47444132 /* \"GDA2\" */\n> +#define GRAPH_CHUNKID_GENERATION_DATA_OVERFLOW 0x47444f32 /* \"GDO2\" */\n> #define GRAPH_CHUNKID_EXTRAEDGES 0x45444745 /* \"EDGE\" */\n> #define GRAPH_CHUNKID_BLOOMINDEXES 0x42494458 /* \"BIDX\" */\n> #define GRAPH_CHUNKID_BLOOMDATA 0x42444154 /* \"BDAT\" */\n> -- \n> 2.35.1.138.gfc5de29e9e6\n> \n> \n> \n> \n"},{"id":"450270","messageId":"33deae83-1afd-1645-82f3-5af14f14094d@github.com","threadId":"57472","inReplyTo":"YiCkRLFxn8Pok7Kc@ncase","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-03T16:00:53Z","receivedAt":"2022-03-03T16:01:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/3/2022 6:19 AM, Patrick Steinhardt wrote:\n> On Wed, Mar 02, 2022 at 09:57:17AM -0500, Derrick Stolee wrote:\n>> On 3/2/2022 8:57 AM, Patrick Steinhardt wrote:\n>>> On Tue, Mar 01, 2022 at 10:25:46AM -0500, Derrick Stolee wrote:\n>>>> On 3/1/2022 9:53 AM, Patrick Steinhardt wrote:\n>>\n>>>>> Hum. I have re-verified, and this indeed seems to play out. So I must've\n>>>>> accidentally ran all my testing with the state generated without the\n>>>>> final patch which fixes the corruption. I do see lots of the following\n>>>>> warnings, but overall I can verify and write the commit-graph just fine:\n>>>>>\n>>>>> commit-graph generation for commit c80a42de8803e2d77818d0c82f88e748d7f9425f is 1623362063 < 1623362139\n>>>>\n>>>> But I'm not able to generate these warnings from either version. I\n>>>> tried generating different levels of a split commit-graph, but\n>>>> could not reproduce it. If you have reproduction steps using current\n>>>> 'master' (or any released Git version) and the four patches here,\n>>>> then I would love to get a full understanding of your errors.\n>>>>\n>>>> Thanks,\n>>>> -Stolee\n>>>\n>>> I haven't yet been able to reproduce it with publicly available data,\n>>> but with the internal references I'm able to evoke the warnings\n>>> reliably. It only works when I have two repositories connected via\n>>> alternates, when generating the commit-graph in the linked-to repo\n>>> first, and then generating the commit-graph in the linking repo.\n>>>\n>>> The following recipe allows me to reproduce, but rely on private data:\n>>>\n>>> $ git --version\n>>> git version 2.35.1\n>>>\n>>> # The pool repository is the one we're linked to from the fork.\n>>> $ cd \"$pool\"\n>>> $ rm -rf objects/info/commit-graph objects/info/commit-graph\n>>> $ git commit-graph write --split\n>>>\n>>> $ cd \"$fork\"\n>>> $ rm -rf objects/info/commit-graph objects/info/commit-graph\n>>> $ git commit-graph write --split\n>>>\n>>> $ git commit-graph verify --no-progress\n>>> $ echo $?\n>>> 0\n>>>\n>>> # This is 715d08a9e51251ad8290b181b6ac3b9e1f9719d7 with your full v2\n>>> # applied on top.\n>>> $ ~/Development/git/bin-wrappers/git --version\n>>> git version 2.35.1.358.g7ede1bea24\n>>>\n>>> $ ~/Development/git/bin-wrappers/git commit-graph verify --no-progress\n>>> commit-graph generation for commit 06a91bac00ed11128becd48d5ae77eacd8f24c97 is 1623273624 < 1623273710\n>>> commit-graph generation for commit 0ae91029f27238e8f8e109c6bb3907f864dda14f is 1622151146 < 1622151220\n>>> commit-graph generation for commit 0d4582a33d8c8e3eb01adbf564f5e1deeb3b56a2 is 1631045222 < 1631045225\n>>> commit-graph generation for commit 0daf8976439d7e0bb9710c5ee63b570580e0dc03 is 1620347739 < 1620347789\n>>> commit-graph generation for commit 0e0ee8ffb3fa22cee7d28e21cbd6df26454932cf is 1623783297 < 1623783380\n>>> commit-graph generation for commit 0f08ab3de6ec115ea8a956a1996cb9759e640e74 is 1621543278 < 1621543339\n>>> commit-graph generation for commit 133ed0319b5a66ae0c2be76e5a887b880452b111 is 1620949864 < 1620949915\n>>> commit-graph generation for commit 1341b3e6c63343ae94a8a473fa057126ddd4669a is 1637344364 < 1637344384\n>>> commit-graph generation for commit 15bdfc501c2c9f23e9353bf6e6a5facd9c32a07a is 1623348103 < 1623348133\n>>> ...\n>>> $ echo $?\n>>> 1\n>>>\n>>> When generating commit-graphs with your patches applied the `verify`\n>>> step works alright.\n>>>\n>>> I've also by accident stumbled over the original error again:\n>>>\n>>> fatal: commit-graph requires overflow generation data but has none\n>>>\n>>> This time it's definitely not caused by generating commit-graphs with an\n>>> in-between state of your patch series because the data comes straight\n>>> from production with no changes to the commit-graphs performed by\n>>> myself. There we're running Git v2.33.1 with a couple of backported\n>>> patches (see [1]). While those patches cause us to make more use of the\n>>> commit-graph, none modify the way we generate them.\n>>>\n>>> Of note is that the commit-graph contains references to commits which\n>>> don't exist in the ODB anymore.\n>>>\n>>> Patrick\n>>>\n>>> [1]: https://gitlab.com/gitlab-org/gitlab-git/-/commits/pks-v2.33.1.gl3\n>>\n>> Thank you for your diligence here, Patrick. I really appreciate the\n>> work you're putting in to verify the situation.\n>>\n>> Since our repro relies on private information, but is consistent, I\n>> wonder if we should take the patch below, which starts to ignore the\n>> older generation number v2 data and only writes freshly-computed\n>> numbers.\n>>\n>> Thanks,\n>> -Stolee\n> \n> Thanks. With your patch below the `fatal:` error is gone, but I'm still\n> seeing the same errors with regards to the commit-graph generations.\n\nThis is disappointing and unexpected. Thanks for verifying.\n\n> So to summarize my findings:\n> \n> - This bug occurs when writing commit-graphs with v2.35.1, but\n> reading them with your patches.\n> \n> - This bug occurs when I have two repositories connected via an\n> alternates file. I haven't yet been able to reproduce it in a\n> single repository that is not connected to a separate ODB.\n\nThis is an interesting distinction. One that I didn't think would\nmatter, but I'll look into the code to see how that could affect\nthings.\n\n> - This bug only occurs when I first generate the commit-graph in the\n> repository I'm borrowing objects from.\n> \n> - This bug only occurs when I write commit-graphs with `--split` in\n> both repositories. \"Normal\" commit-graphs don't have this issue,\n> and neither can I see it with `--split=replace` or mixed-type\n> commit-graphs.\n> \n> Beware, the following explanation is based on my very basic\n> understanding of the commit-graph code and thus more likely to be wrong\n> than right:\n> \n> With the old Git version, we've been mis-parsing the generation because\n> `read_generation_data` wasn't ever set. As a result it can happen that\n> the second split commit-graph we're generating computes its own\n> generation numbers from the wrong starting point because it uses the\n> mis-parsed generation numbers from the parent commit-graph.\n> \n> With your patches, we start to correctly account for overflows and would\n> thus end up with a different value for the generation depending on where\n> we parse the commit from: if we parse it from the first commit-graph it\n> would be correct because it's contains the \"root\" of the generation\n> numbers. But if we parse a commit from the second commit-graph we may\n> have a mismatch because the generation numbers in there may have been\n> derived from generation numbers mis-parsed from the first commit-graph.\n> And because these would be wrong in case there was an overflow it is\n> clear that the new corrected generation number may be wrong, as well.\n\nHm. My expectation was that the older layers of the split commit-graph\nwould have read_generation_data disabled (because the new Git version\ncannot read the GDAT chunk) and then the validate_mixed_generation_chain()\nmethod would remove read_generation_data from all of the graphs in the\nlist.\n\nCombining this with your thoughts on cross-alternate split commit-graphs,\nthis makes me think we should try this:\n\n--- >8 ---\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex fb2ced0bd6..74c6534f56 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -609,8 +609,6 @@ struct commit_graph *read_commit_graph_one(struct repository *r,\n \tif (!g)\n \t\tg = load_commit_graph_chain(r, odb);\n \n-\tvalidate_mixed_generation_chain(g);\n-\n \treturn g;\n }\n \n@@ -668,7 +666,13 @@ static int prepare_commit_graph(struct repository *r)\n \t !r->objects->commit_graph && odb;\n \t odb = odb->next)\n \t\tprepare_commit_graph_one(r, odb);\n-\treturn !!r->objects->commit_graph;\n+\n+\tif (r->objects->commit_graph) {\n+\t\tvalidate_mixed_generation_chain(r->objects->commit_graph);\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n }\n \n int generation_numbers_enabled(struct repository *r)\n\n\n--- >8 ---\n\nNotice that I'm moving the validate_mixed_generation_chain() call\nout of read_commit_graph_one() and into prepare_commit_graph(). To\nmy understanding, this _should_ have an equivalent end state as the\nold code, but might be worth trying just as a quick check.\n\nI will continue investigating and try to reproduce with this\nadditional constraint of working across an alternate.\n\nThanks,\n-Stolee\n\n\n"},{"id":"450398","messageId":"06ea3190-32d0-c792-0ae9-c5600305f158@github.com","threadId":"57472","inReplyTo":"33deae83-1afd-1645-82f3-5af14f14094d@github.com","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-04T14:03:15Z","receivedAt":"2022-03-04T14:03:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/3/2022 11:00 AM, Derrick Stolee wrote:\n> On 3/3/2022 6:19 AM, Patrick Steinhardt wrote:\n>> On Wed, Mar 02, 2022 at 09:57:17AM -0500, Derrick Stolee wrote:\n>>> On 3/2/2022 8:57 AM, Patrick Steinhardt wrote:\n>>>> On Tue, Mar 01, 2022 at 10:25:46AM -0500, Derrick Stolee wrote:\n>>>>> On 3/1/2022 9:53 AM, Patrick Steinhardt wrote:\n>>>\n>>>>>> Hum. I have re-verified, and this indeed seems to play out. So I must've\n>>>>>> accidentally ran all my testing with the state generated without the\n>>>>>> final patch which fixes the corruption. I do see lots of the following\n>>>>>> warnings, but overall I can verify and write the commit-graph just fine:\n>>>>>>\n>>>>>> commit-graph generation for commit c80a42de8803e2d77818d0c82f88e748d7f9425f is 1623362063 < 1623362139\n>>>>>\n>>>>> But I'm not able to generate these warnings from either version. I\n>>>>> tried generating different levels of a split commit-graph, but\n>>>>> could not reproduce it. If you have reproduction steps using current\n>>>>> 'master' (or any released Git version) and the four patches here,\n>>>>> then I would love to get a full understanding of your errors.\n>>>>>\n>>>>> Thanks,\n>>>>> -Stolee\n>>>>\n>>>> I haven't yet been able to reproduce it with publicly available data,\n>>>> but with the internal references I'm able to evoke the warnings\n>>>> reliably. It only works when I have two repositories connected via\n>>>> alternates, when generating the commit-graph in the linked-to repo\n>>>> first, and then generating the commit-graph in the linking repo.\n>>>>\n>>>> The following recipe allows me to reproduce, but rely on private data:\n>>>>\n>>>> $ git --version\n>>>> git version 2.35.1\n>>>>\n>>>> # The pool repository is the one we're linked to from the fork.\n>>>> $ cd \"$pool\"\n>>>> $ rm -rf objects/info/commit-graph objects/info/commit-graph\n>>>> $ git commit-graph write --split\n>>>>\n>>>> $ cd \"$fork\"\n>>>> $ rm -rf objects/info/commit-graph objects/info/commit-graph\n>>>> $ git commit-graph write --split\n>>>>\n>>>> $ git commit-graph verify --no-progress\n>>>> $ echo $?\n>>>> 0\n>>>>\n>>>> # This is 715d08a9e51251ad8290b181b6ac3b9e1f9719d7 with your full v2\n>>>> # applied on top.\n>>>> $ ~/Development/git/bin-wrappers/git --version\n>>>> git version 2.35.1.358.g7ede1bea24\n>>>>\n>>>> $ ~/Development/git/bin-wrappers/git commit-graph verify --no-progress\n>>>> commit-graph generation for commit 06a91bac00ed11128becd48d5ae77eacd8f24c97 is 1623273624 < 1623273710\n>>>> commit-graph generation for commit 0ae91029f27238e8f8e109c6bb3907f864dda14f is 1622151146 < 1622151220\n>>>> commit-graph generation for commit 0d4582a33d8c8e3eb01adbf564f5e1deeb3b56a2 is 1631045222 < 1631045225\n>>>> commit-graph generation for commit 0daf8976439d7e0bb9710c5ee63b570580e0dc03 is 1620347739 < 1620347789\n>>>> commit-graph generation for commit 0e0ee8ffb3fa22cee7d28e21cbd6df26454932cf is 1623783297 < 1623783380\n>>>> commit-graph generation for commit 0f08ab3de6ec115ea8a956a1996cb9759e640e74 is 1621543278 < 1621543339\n>>>> commit-graph generation for commit 133ed0319b5a66ae0c2be76e5a887b880452b111 is 1620949864 < 1620949915\n>>>> commit-graph generation for commit 1341b3e6c63343ae94a8a473fa057126ddd4669a is 1637344364 < 1637344384\n>>>> commit-graph generation for commit 15bdfc501c2c9f23e9353bf6e6a5facd9c32a07a is 1623348103 < 1623348133\n>>>> ...\n>>>> $ echo $?\n>>>> 1\n>>>>\n>>>> When generating commit-graphs with your patches applied the `verify`\n>>>> step works alright.\n>>>>\n>>>> I've also by accident stumbled over the original error again:\n>>>>\n>>>> fatal: commit-graph requires overflow generation data but has none\n>>>>\n>>>> This time it's definitely not caused by generating commit-graphs with an\n>>>> in-between state of your patch series because the data comes straight\n>>>> from production with no changes to the commit-graphs performed by\n>>>> myself. There we're running Git v2.33.1 with a couple of backported\n>>>> patches (see [1]). While those patches cause us to make more use of the\n>>>> commit-graph, none modify the way we generate them.\n>>>>\n>>>> Of note is that the commit-graph contains references to commits which\n>>>> don't exist in the ODB anymore.\n>>>>\n>>>> Patrick\n>>>>\n>>>> [1]: https://gitlab.com/gitlab-org/gitlab-git/-/commits/pks-v2.33.1.gl3\n>>>\n>>> Thank you for your diligence here, Patrick. I really appreciate the\n>>> work you're putting in to verify the situation.\n>>>\n>>> Since our repro relies on private information, but is consistent, I\n>>> wonder if we should take the patch below, which starts to ignore the\n>>> older generation number v2 data and only writes freshly-computed\n>>> numbers.\n>>>\n>>> Thanks,\n>>> -Stolee\n>>\n>> Thanks. With your patch below the `fatal:` error is gone, but I'm still\n>> seeing the same errors with regards to the commit-graph generations.\n> \n> This is disappointing and unexpected. Thanks for verifying.\n> \n>> So to summarize my findings:\n>>\n>> - This bug occurs when writing commit-graphs with v2.35.1, but\n>> reading them with your patches.\n>>\n>> - This bug occurs when I have two repositories connected via an\n>> alternates file. I haven't yet been able to reproduce it in a\n>> single repository that is not connected to a separate ODB.\n> \n> This is an interesting distinction. One that I didn't think would\n> matter, but I'll look into the code to see how that could affect\n> things.\n> \n>> - This bug only occurs when I first generate the commit-graph in the\n>> repository I'm borrowing objects from.\n>>\n>> - This bug only occurs when I write commit-graphs with `--split` in\n>> both repositories. \"Normal\" commit-graphs don't have this issue,\n>> and neither can I see it with `--split=replace` or mixed-type\n>> commit-graphs.\n>>\n>> Beware, the following explanation is based on my very basic\n>> understanding of the commit-graph code and thus more likely to be wrong\n>> than right:\n>>\n>> With the old Git version, we've been mis-parsing the generation because\n>> `read_generation_data` wasn't ever set. As a result it can happen that\n>> the second split commit-graph we're generating computes its own\n>> generation numbers from the wrong starting point because it uses the\n>> mis-parsed generation numbers from the parent commit-graph.\n>>\n>> With your patches, we start to correctly account for overflows and would\n>> thus end up with a different value for the generation depending on where\n>> we parse the commit from: if we parse it from the first commit-graph it\n>> would be correct because it's contains the \"root\" of the generation\n>> numbers. But if we parse a commit from the second commit-graph we may\n>> have a mismatch because the generation numbers in there may have been\n>> derived from generation numbers mis-parsed from the first commit-graph.\n>> And because these would be wrong in case there was an overflow it is\n>> clear that the new corrected generation number may be wrong, as well.\n> \n> Hm. My expectation was that the older layers of the split commit-graph\n> would have read_generation_data disabled (because the new Git version\n> cannot read the GDAT chunk) and then the validate_mixed_generation_chain()\n> method would remove read_generation_data from all of the graphs in the\n> list.\n> \n> Combining this with your thoughts on cross-alternate split commit-graphs,\n> this makes me think we should try this:\n> \n> --- >8 ---\n> \n> diff --git a/commit-graph.c b/commit-graph.c\n> index fb2ced0bd6..74c6534f56 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -609,8 +609,6 @@ struct commit_graph *read_commit_graph_one(struct repository *r,\n> \tif (!g)\n> \t\tg = load_commit_graph_chain(r, odb);\n> \n> -\tvalidate_mixed_generation_chain(g);\n> -\n> \treturn g;\n> }\n> \n> @@ -668,7 +666,13 @@ static int prepare_commit_graph(struct repository *r)\n> \t !r->objects->commit_graph && odb;\n> \t odb = odb->next)\n> \t\tprepare_commit_graph_one(r, odb);\n> -\treturn !!r->objects->commit_graph;\n> +\n> +\tif (r->objects->commit_graph) {\n> +\t\tvalidate_mixed_generation_chain(r->objects->commit_graph);\n> +\t\treturn 1;\n> +\t}\n> +\n> +\treturn 0;\n> }\n> \n> int generation_numbers_enabled(struct repository *r)\n> \n> \n> --- >8 ---\n> \n> Notice that I'm moving the validate_mixed_generation_chain() call\n> out of read_commit_graph_one() and into prepare_commit_graph(). To\n> my understanding, this _should_ have an equivalent end state as the\n> old code, but might be worth trying just as a quick check.\n> \n> I will continue investigating and try to reproduce with this\n> additional constraint of working across an alternate.\n\nMy attempts to reproduce this across an alternate have failed. I\ntried running the following test against Git without these patches,\nthen verify with the newer version of Git. (I also have generated\na few new layers on top with these patches, and they correctly drop\nthe GDA2 and GDO2 chunks when the lower layers \"don't have gen v2\".)\n\n\ntest_description='commit-graph with offsets across alternates'\n. ./test-lib.sh\n\nif ! test_have_prereq TIME_IS_64BIT || ! test_have_prereq TIME_T_IS_64BIT\nthen\n\tskip_all='skipping 64-bit timestamp tests'\n\ttest_done\nfi\n\n\nUNIX_EPOCH_ZERO=\"@0 +0000\"\nFUTURE_DATE=\"@4147483646 +0000\"\n\nGIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=0\n\ntest_expect_success 'generate alternate split commit-graph' '\n\tgit init alternate &&\n\t(\n\t\tcd alternate &&\n\t\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 1 &&\n\t\ttest_commit --date \"$FUTURE_DATE\" 2 &&\n\t\tgit commit-graph write --reachable &&\n\t\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 3 &&\n\t\ttest_commit --date \"$FUTURE_DATE\" 4 &&\n\t\tgit commit-graph write --reachable --split=no-merge\n\t) &&\n\tgit clone --shared alternate fork &&\n\t(\n\t\tcd fork &&\n\t\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 5 &&\n\t\ttest_commit --date \"$FUTURE_DATE\" 6 &&\n\t\tgit commit-graph write --reachable --split=no-merge &&\n\t\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 7 &&\n\t\ttest_commit --date \"$FUTURE_DATE\" 8 &&\n\t\tgit commit-graph write --reachable --split=no-merge\n\t)\n'\n\ntest_done\n\n\nMy testing after running this with -d allows me to reliably see these\nlayers being created with GDAT and GDOV chunks. Running the 'git\ncommit-graph verify' command with the new code does not show those\nerrors, even after adding commits and another layer to the split\ncommit-graph.\n\nI look forward to any additional insights you might have here.\n\nThanks,\n-Stolee\n"},{"id":"450516","messageId":"YiXfnsbGzPXffdgV@ncase","threadId":"57472","inReplyTo":"06ea3190-32d0-c792-0ae9-c5600305f158@github.com","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-03-07T10:34:06Z","receivedAt":"2022-03-07T11:11:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Mar 04, 2022 at 09:03:15AM -0500, Derrick Stolee wrote:\n> On 3/3/2022 11:00 AM, Derrick Stolee wrote:\n> > On 3/3/2022 6:19 AM, Patrick Steinhardt wrote:\n> >> On Wed, Mar 02, 2022 at 09:57:17AM -0500, Derrick Stolee wrote:\n> >>> On 3/2/2022 8:57 AM, Patrick Steinhardt wrote:\n> >>>> On Tue, Mar 01, 2022 at 10:25:46AM -0500, Derrick Stolee wrote:\n> >>>>> On 3/1/2022 9:53 AM, Patrick Steinhardt wrote:\n> >>>\n> >>>>>> Hum. I have re-verified, and this indeed seems to play out. So I must've\n> >>>>>> accidentally ran all my testing with the state generated without the\n> >>>>>> final patch which fixes the corruption. I do see lots of the following\n> >>>>>> warnings, but overall I can verify and write the commit-graph just fine:\n> >>>>>>\n> >>>>>> commit-graph generation for commit c80a42de8803e2d77818d0c82f88e748d7f9425f is 1623362063 < 1623362139\n> >>>>>\n> >>>>> But I'm not able to generate these warnings from either version. I\n> >>>>> tried generating different levels of a split commit-graph, but\n> >>>>> could not reproduce it. If you have reproduction steps using current\n> >>>>> 'master' (or any released Git version) and the four patches here,\n> >>>>> then I would love to get a full understanding of your errors.\n> >>>>>\n> >>>>> Thanks,\n> >>>>> -Stolee\n> >>>>\n> >>>> I haven't yet been able to reproduce it with publicly available data,\n> >>>> but with the internal references I'm able to evoke the warnings\n> >>>> reliably. It only works when I have two repositories connected via\n> >>>> alternates, when generating the commit-graph in the linked-to repo\n> >>>> first, and then generating the commit-graph in the linking repo.\n> >>>>\n> >>>> The following recipe allows me to reproduce, but rely on private data:\n> >>>>\n> >>>> $ git --version\n> >>>> git version 2.35.1\n> >>>>\n> >>>> # The pool repository is the one we're linked to from the fork.\n> >>>> $ cd \"$pool\"\n> >>>> $ rm -rf objects/info/commit-graph objects/info/commit-graph\n> >>>> $ git commit-graph write --split\n> >>>>\n> >>>> $ cd \"$fork\"\n> >>>> $ rm -rf objects/info/commit-graph objects/info/commit-graph\n> >>>> $ git commit-graph write --split\n> >>>>\n> >>>> $ git commit-graph verify --no-progress\n> >>>> $ echo $?\n> >>>> 0\n> >>>>\n> >>>> # This is 715d08a9e51251ad8290b181b6ac3b9e1f9719d7 with your full v2\n> >>>> # applied on top.\n> >>>> $ ~/Development/git/bin-wrappers/git --version\n> >>>> git version 2.35.1.358.g7ede1bea24\n> >>>>\n> >>>> $ ~/Development/git/bin-wrappers/git commit-graph verify --no-progress\n> >>>> commit-graph generation for commit 06a91bac00ed11128becd48d5ae77eacd8f24c97 is 1623273624 < 1623273710\n> >>>> commit-graph generation for commit 0ae91029f27238e8f8e109c6bb3907f864dda14f is 1622151146 < 1622151220\n> >>>> commit-graph generation for commit 0d4582a33d8c8e3eb01adbf564f5e1deeb3b56a2 is 1631045222 < 1631045225\n> >>>> commit-graph generation for commit 0daf8976439d7e0bb9710c5ee63b570580e0dc03 is 1620347739 < 1620347789\n> >>>> commit-graph generation for commit 0e0ee8ffb3fa22cee7d28e21cbd6df26454932cf is 1623783297 < 1623783380\n> >>>> commit-graph generation for commit 0f08ab3de6ec115ea8a956a1996cb9759e640e74 is 1621543278 < 1621543339\n> >>>> commit-graph generation for commit 133ed0319b5a66ae0c2be76e5a887b880452b111 is 1620949864 < 1620949915\n> >>>> commit-graph generation for commit 1341b3e6c63343ae94a8a473fa057126ddd4669a is 1637344364 < 1637344384\n> >>>> commit-graph generation for commit 15bdfc501c2c9f23e9353bf6e6a5facd9c32a07a is 1623348103 < 1623348133\n> >>>> ...\n> >>>> $ echo $?\n> >>>> 1\n> >>>>\n> >>>> When generating commit-graphs with your patches applied the `verify`\n> >>>> step works alright.\n> >>>>\n> >>>> I've also by accident stumbled over the original error again:\n> >>>>\n> >>>> fatal: commit-graph requires overflow generation data but has none\n> >>>>\n> >>>> This time it's definitely not caused by generating commit-graphs with an\n> >>>> in-between state of your patch series because the data comes straight\n> >>>> from production with no changes to the commit-graphs performed by\n> >>>> myself. There we're running Git v2.33.1 with a couple of backported\n> >>>> patches (see [1]). While those patches cause us to make more use of the\n> >>>> commit-graph, none modify the way we generate them.\n> >>>>\n> >>>> Of note is that the commit-graph contains references to commits which\n> >>>> don't exist in the ODB anymore.\n> >>>>\n> >>>> Patrick\n> >>>>\n> >>>> [1]: https://gitlab.com/gitlab-org/gitlab-git/-/commits/pks-v2.33.1.gl3\n> >>>\n> >>> Thank you for your diligence here, Patrick. I really appreciate the\n> >>> work you're putting in to verify the situation.\n> >>>\n> >>> Since our repro relies on private information, but is consistent, I\n> >>> wonder if we should take the patch below, which starts to ignore the\n> >>> older generation number v2 data and only writes freshly-computed\n> >>> numbers.\n> >>>\n> >>> Thanks,\n> >>> -Stolee\n> >>\n> >> Thanks. With your patch below the `fatal:` error is gone, but I'm still\n> >> seeing the same errors with regards to the commit-graph generations.\n> > \n> > This is disappointing and unexpected. Thanks for verifying.\n> > \n> >> So to summarize my findings:\n> >>\n> >> - This bug occurs when writing commit-graphs with v2.35.1, but\n> >> reading them with your patches.\n> >>\n> >> - This bug occurs when I have two repositories connected via an\n> >> alternates file. I haven't yet been able to reproduce it in a\n> >> single repository that is not connected to a separate ODB.\n> > \n> > This is an interesting distinction. One that I didn't think would\n> > matter, but I'll look into the code to see how that could affect\n> > things.\n> > \n> >> - This bug only occurs when I first generate the commit-graph in the\n> >> repository I'm borrowing objects from.\n> >>\n> >> - This bug only occurs when I write commit-graphs with `--split` in\n> >> both repositories. \"Normal\" commit-graphs don't have this issue,\n> >> and neither can I see it with `--split=replace` or mixed-type\n> >> commit-graphs.\n> >>\n> >> Beware, the following explanation is based on my very basic\n> >> understanding of the commit-graph code and thus more likely to be wrong\n> >> than right:\n> >>\n> >> With the old Git version, we've been mis-parsing the generation because\n> >> `read_generation_data` wasn't ever set. As a result it can happen that\n> >> the second split commit-graph we're generating computes its own\n> >> generation numbers from the wrong starting point because it uses the\n> >> mis-parsed generation numbers from the parent commit-graph.\n> >>\n> >> With your patches, we start to correctly account for overflows and would\n> >> thus end up with a different value for the generation depending on where\n> >> we parse the commit from: if we parse it from the first commit-graph it\n> >> would be correct because it's contains the \"root\" of the generation\n> >> numbers. But if we parse a commit from the second commit-graph we may\n> >> have a mismatch because the generation numbers in there may have been\n> >> derived from generation numbers mis-parsed from the first commit-graph.\n> >> And because these would be wrong in case there was an overflow it is\n> >> clear that the new corrected generation number may be wrong, as well.\n> > \n> > Hm. My expectation was that the older layers of the split commit-graph\n> > would have read_generation_data disabled (because the new Git version\n> > cannot read the GDAT chunk) and then the validate_mixed_generation_chain()\n> > method would remove read_generation_data from all of the graphs in the\n> > list.\n> > \n> > Combining this with your thoughts on cross-alternate split commit-graphs,\n> > this makes me think we should try this:\n> > \n> > --- >8 ---\n> > \n> > diff --git a/commit-graph.c b/commit-graph.c\n> > index fb2ced0bd6..74c6534f56 100644\n> > --- a/commit-graph.c\n> > +++ b/commit-graph.c\n> > @@ -609,8 +609,6 @@ struct commit_graph *read_commit_graph_one(struct repository *r,\n> > \tif (!g)\n> > \t\tg = load_commit_graph_chain(r, odb);\n> > \n> > -\tvalidate_mixed_generation_chain(g);\n> > -\n> > \treturn g;\n> > }\n> > \n> > @@ -668,7 +666,13 @@ static int prepare_commit_graph(struct repository *r)\n> > \t !r->objects->commit_graph && odb;\n> > \t odb = odb->next)\n> > \t\tprepare_commit_graph_one(r, odb);\n> > -\treturn !!r->objects->commit_graph;\n> > +\n> > +\tif (r->objects->commit_graph) {\n> > +\t\tvalidate_mixed_generation_chain(r->objects->commit_graph);\n> > +\t\treturn 1;\n> > +\t}\n> > +\n> > +\treturn 0;\n> > }\n> > \n> > int generation_numbers_enabled(struct repository *r)\n> > \n> > \n> > --- >8 ---\n> > \n> > Notice that I'm moving the validate_mixed_generation_chain() call\n> > out of read_commit_graph_one() and into prepare_commit_graph(). To\n> > my understanding, this _should_ have an equivalent end state as the\n> > old code, but might be worth trying just as a quick check.\n> > \n> > I will continue investigating and try to reproduce with this\n> > additional constraint of working across an alternate.\n> \n> My attempts to reproduce this across an alternate have failed. I\n> tried running the following test against Git without these patches,\n> then verify with the newer version of Git. (I also have generated\n> a few new layers on top with these patches, and they correctly drop\n> the GDA2 and GDO2 chunks when the lower layers \"don't have gen v2\".)\n> \n> \n> test_description='commit-graph with offsets across alternates'\n> . ./test-lib.sh\n> \n> if ! test_have_prereq TIME_IS_64BIT || ! test_have_prereq TIME_T_IS_64BIT\n> then\n> \tskip_all='skipping 64-bit timestamp tests'\n> \ttest_done\n> fi\n> \n> \n> UNIX_EPOCH_ZERO=\"@0 +0000\"\n> FUTURE_DATE=\"@4147483646 +0000\"\n> \n> GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=0\n> \n> test_expect_success 'generate alternate split commit-graph' '\n> \tgit init alternate &&\n> \t(\n> \t\tcd alternate &&\n> \t\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 1 &&\n> \t\ttest_commit --date \"$FUTURE_DATE\" 2 &&\n> \t\tgit commit-graph write --reachable &&\n> \t\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 3 &&\n> \t\ttest_commit --date \"$FUTURE_DATE\" 4 &&\n> \t\tgit commit-graph write --reachable --split=no-merge\n> \t) &&\n> \tgit clone --shared alternate fork &&\n> \t(\n> \t\tcd fork &&\n> \t\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 5 &&\n> \t\ttest_commit --date \"$FUTURE_DATE\" 6 &&\n> \t\tgit commit-graph write --reachable --split=no-merge &&\n> \t\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 7 &&\n> \t\ttest_commit --date \"$FUTURE_DATE\" 8 &&\n> \t\tgit commit-graph write --reachable --split=no-merge\n> \t)\n> '\n> \n> test_done\n> \n> \n> My testing after running this with -d allows me to reliably see these\n> layers being created with GDAT and GDOV chunks. Running the 'git\n> commit-graph verify' command with the new code does not show those\n> errors, even after adding commits and another layer to the split\n> commit-graph.\n> \n> I look forward to any additional insights you might have here.\n\nI don't really know why, but now I've become unable to reproduce it\nagain. I think we should just go with your patch 5/4 on top -- it does\nfix the most important issue, which is the `die()` I saw on almost all\ncommands. The second part about the warnings I'm just not sure about,\nbut I don't think it should stop this patch series given my own\nuncertainty.\n\nPatrick\n"},{"id":"450571","messageId":"94ed6a1e-327c-3f94-b98b-db019a6f5ada@github.com","threadId":"57472","inReplyTo":"YiXfnsbGzPXffdgV@ncase","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-07T13:45:07Z","receivedAt":"2022-03-07T13:45:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/7/2022 5:34 AM, Patrick Steinhardt wrote:\n> On Fri, Mar 04, 2022 at 09:03:15AM -0500, Derrick Stolee wrote:\n>> On 3/3/2022 11:00 AM, Derrick Stolee wrote:\n...\n>>> I will continue investigating and try to reproduce with this\n>>> additional constraint of working across an alternate.\n>>\n>> My attempts to reproduce this across an alternate have failed. I\n>> tried running the following test against Git without these patches,\n>> then verify with the newer version of Git. (I also have generated\n>> a few new layers on top with these patches, and they correctly drop\n>> the GDA2 and GDO2 chunks when the lower layers \"don't have gen v2\".)\n>>\n>>\n>> test_description='commit-graph with offsets across alternates'\n>> . ./test-lib.sh\n>>\n>> if ! test_have_prereq TIME_IS_64BIT || ! test_have_prereq TIME_T_IS_64BIT\n>> then\n>> \tskip_all='skipping 64-bit timestamp tests'\n>> \ttest_done\n>> fi\n>>\n>>\n>> UNIX_EPOCH_ZERO=\"@0 +0000\"\n>> FUTURE_DATE=\"@4147483646 +0000\"\n>>\n>> GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=0\n>>\n>> test_expect_success 'generate alternate split commit-graph' '\n>> \tgit init alternate &&\n>> \t(\n>> \t\tcd alternate &&\n>> \t\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 1 &&\n>> \t\ttest_commit --date \"$FUTURE_DATE\" 2 &&\n>> \t\tgit commit-graph write --reachable &&\n>> \t\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 3 &&\n>> \t\ttest_commit --date \"$FUTURE_DATE\" 4 &&\n>> \t\tgit commit-graph write --reachable --split=no-merge\n>> \t) &&\n>> \tgit clone --shared alternate fork &&\n>> \t(\n>> \t\tcd fork &&\n>> \t\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 5 &&\n>> \t\ttest_commit --date \"$FUTURE_DATE\" 6 &&\n>> \t\tgit commit-graph write --reachable --split=no-merge &&\n>> \t\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 7 &&\n>> \t\ttest_commit --date \"$FUTURE_DATE\" 8 &&\n>> \t\tgit commit-graph write --reachable --split=no-merge\n>> \t)\n>> '\n>>\n>> test_done\n>>\n>>\n>> My testing after running this with -d allows me to reliably see these\n>> layers being created with GDAT and GDOV chunks. Running the 'git\n>> commit-graph verify' command with the new code does not show those\n>> errors, even after adding commits and another layer to the split\n>> commit-graph.\n>>\n>> I look forward to any additional insights you might have here.\n> \n> I don't really know why, but now I've become unable to reproduce it\n> again. I think we should just go with your patch 5/4 on top -- it does\n> fix the most important issue, which is the `die()` I saw on almost all\n> commands. The second part about the warnings I'm just not sure about,\n> but I don't think it should stop this patch series given my own\n> uncertainty.\n\nThanks for following up. I agree that with 5/4 we should be safe.\n\nI'll remain available to quickly respond if anything else surprising\ncomes up in this area.\n\nThanks!\n-Stolee\n"},{"id":"450621","messageId":"xmqqa6e1u18j.fsf@gitster.g","threadId":"57472","inReplyTo":"94ed6a1e-327c-3f94-b98b-db019a6f5ada@github.com","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2022-03-07T17:22:04Z","receivedAt":"2022-03-07T17:22:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> Thanks for following up. I agree that with 5/4 we should be safe.\n>\n> I'll remain available to quickly respond if anything else surprising\n> comes up in this area.\n\nThanks. I just picked up the \"bankruptcy\" step, so we should be\ngood to go.\n\n"},{"id":"451015","messageId":"YioEHVo3GnzBl3rW@ncase","threadId":"57472","inReplyTo":"94ed6a1e-327c-3f94-b98b-db019a6f5ada@github.com","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2022-03-10T13:58:53Z","receivedAt":"2022-03-10T13:59:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Mar 07, 2022 at 08:45:07AM -0500, Derrick Stolee wrote:\n> On 3/7/2022 5:34 AM, Patrick Steinhardt wrote:\n> > On Fri, Mar 04, 2022 at 09:03:15AM -0500, Derrick Stolee wrote:\n> >> On 3/3/2022 11:00 AM, Derrick Stolee wrote:\n> ...\n> >>> I will continue investigating and try to reproduce with this\n> >>> additional constraint of working across an alternate.\n> >>\n> >> My attempts to reproduce this across an alternate have failed. I\n> >> tried running the following test against Git without these patches,\n> >> then verify with the newer version of Git. (I also have generated\n> >> a few new layers on top with these patches, and they correctly drop\n> >> the GDA2 and GDO2 chunks when the lower layers \"don't have gen v2\".)\n> >>\n> >>\n> >> test_description='commit-graph with offsets across alternates'\n> >> . ./test-lib.sh\n> >>\n> >> if ! test_have_prereq TIME_IS_64BIT || ! test_have_prereq TIME_T_IS_64BIT\n> >> then\n> >> \tskip_all='skipping 64-bit timestamp tests'\n> >> \ttest_done\n> >> fi\n> >>\n> >>\n> >> UNIX_EPOCH_ZERO=\"@0 +0000\"\n> >> FUTURE_DATE=\"@4147483646 +0000\"\n> >>\n> >> GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=0\n> >>\n> >> test_expect_success 'generate alternate split commit-graph' '\n> >> \tgit init alternate &&\n> >> \t(\n> >> \t\tcd alternate &&\n> >> \t\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 1 &&\n> >> \t\ttest_commit --date \"$FUTURE_DATE\" 2 &&\n> >> \t\tgit commit-graph write --reachable &&\n> >> \t\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 3 &&\n> >> \t\ttest_commit --date \"$FUTURE_DATE\" 4 &&\n> >> \t\tgit commit-graph write --reachable --split=no-merge\n> >> \t) &&\n> >> \tgit clone --shared alternate fork &&\n> >> \t(\n> >> \t\tcd fork &&\n> >> \t\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 5 &&\n> >> \t\ttest_commit --date \"$FUTURE_DATE\" 6 &&\n> >> \t\tgit commit-graph write --reachable --split=no-merge &&\n> >> \t\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 7 &&\n> >> \t\ttest_commit --date \"$FUTURE_DATE\" 8 &&\n> >> \t\tgit commit-graph write --reachable --split=no-merge\n> >> \t)\n> >> '\n> >>\n> >> test_done\n> >>\n> >>\n> >> My testing after running this with -d allows me to reliably see these\n> >> layers being created with GDAT and GDOV chunks. Running the 'git\n> >> commit-graph verify' command with the new code does not show those\n> >> errors, even after adding commits and another layer to the split\n> >> commit-graph.\n> >>\n> >> I look forward to any additional insights you might have here.\n> > \n> > I don't really know why, but now I've become unable to reproduce it\n> > again. I think we should just go with your patch 5/4 on top -- it does\n> > fix the most important issue, which is the `die()` I saw on almost all\n> > commands. The second part about the warnings I'm just not sure about,\n> > but I don't think it should stop this patch series given my own\n> > uncertainty.\n> \n> Thanks for following up. I agree that with 5/4 we should be safe.\n> \n> I'll remain available to quickly respond if anything else surprising\n> comes up in this area.\n> \n> Thanks!\n> -Stolee\n\nThere is another surprise I hit today in the context of generation\nnumbers. In production, I found the following bug:\n\n signal: aborted (core dumped): BUG: chunk-format.c:88: expected to write 8 bytes to chunk 47444f56, but wrote 168304 instead\n\n47444f56 is the GENERATION_DATA_OVERFLOW chunk ID, and seemingly the\nprecomputed size we intended to write was mismatching the data we have\nactually been writing to disk. And I think this stems from a mismatch in\nhow we precompute the number of generation data overflows compared to\nhow we're actually writing the data to disk:\n\n - We precompute how many generation number overflows there are in\n `compute_generation_numbers()`. Here we only increment the number\n of overflows in case all parents of a given commit have a non-zero\n generation number and if the generation is bigger than OFFSET_MAX.\n Seemingly we have found only a single commit which matches this\n criteria because we pass `sizeof(timestamp_t) * overflows` as\n expected size, and `sizeof(timestamp_t) == 8`.\n\n - On the other hand, when we write generation numbers to disk in\n `write_graph_chunk_generation_data_overflow()`, we always write a\n chunk in case its offset is bigger than OFFSET_MAX. So we don't\n care about the parents here, and this seems to extend the number\n of commits which match this criteria to 21038 commits we write\n into the file.\n\nThe result is that the sanity check we do where we compare that the\nactually written amount of data matches what we expect fails because of\nthe different ways we count this data.\n\nThis time I don't have access to the repository myself, I only tried to\ncombine what's happening based on the bug message and the code.\n\nPatrick\n"},{"id":"451039","messageId":"2318f4e9-2267-0097-4d6f-a1b916cda3ca@github.com","threadId":"57472","inReplyTo":"YioEHVo3GnzBl3rW@ncase","subject":"Re: [PATCH 3/7] commit-graph: start parsing generation v2 (again)","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-03-10T17:18:09Z","receivedAt":"2022-03-10T17:18:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/10/2022 8:58 AM, Patrick Steinhardt wrote:\n> On Mon, Mar 07, 2022 at 08:45:07AM -0500, Derrick Stolee wrote:\n>> On 3/7/2022 5:34 AM, Patrick Steinhardt wrote:\n>>> On Fri, Mar 04, 2022 at 09:03:15AM -0500, Derrick Stolee wrote:\n>>>> On 3/3/2022 11:00 AM, Derrick Stolee wrote:\n>> ...\n>>>>> I will continue investigating and try to reproduce with this\n>>>>> additional constraint of working across an alternate.\n>>>>\n>>>> My attempts to reproduce this across an alternate have failed. I\n>>>> tried running the following test against Git without these patches,\n>>>> then verify with the newer version of Git. (I also have generated\n>>>> a few new layers on top with these patches, and they correctly drop\n>>>> the GDA2 and GDO2 chunks when the lower layers \"don't have gen v2\".)\n>>>>\n>>>>\n>>>> test_description='commit-graph with offsets across alternates'\n>>>> . ./test-lib.sh\n>>>>\n>>>> if ! test_have_prereq TIME_IS_64BIT || ! test_have_prereq TIME_T_IS_64BIT\n>>>> then\n>>>> \tskip_all='skipping 64-bit timestamp tests'\n>>>> \ttest_done\n>>>> fi\n>>>>\n>>>>\n>>>> UNIX_EPOCH_ZERO=\"@0 +0000\"\n>>>> FUTURE_DATE=\"@4147483646 +0000\"\n>>>>\n>>>> GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=0\n>>>>\n>>>> test_expect_success 'generate alternate split commit-graph' '\n>>>> \tgit init alternate &&\n>>>> \t(\n>>>> \t\tcd alternate &&\n>>>> \t\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 1 &&\n>>>> \t\ttest_commit --date \"$FUTURE_DATE\" 2 &&\n>>>> \t\tgit commit-graph write --reachable &&\n>>>> \t\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 3 &&\n>>>> \t\ttest_commit --date \"$FUTURE_DATE\" 4 &&\n>>>> \t\tgit commit-graph write --reachable --split=no-merge\n>>>> \t) &&\n>>>> \tgit clone --shared alternate fork &&\n>>>> \t(\n>>>> \t\tcd fork &&\n>>>> \t\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 5 &&\n>>>> \t\ttest_commit --date \"$FUTURE_DATE\" 6 &&\n>>>> \t\tgit commit-graph write --reachable --split=no-merge &&\n>>>> \t\ttest_commit --date \"$UNIX_EPOCH_ZERO\" 7 &&\n>>>> \t\ttest_commit --date \"$FUTURE_DATE\" 8 &&\n>>>> \t\tgit commit-graph write --reachable --split=no-merge\n>>>> \t)\n>>>> '\n>>>>\n>>>> test_done\n>>>>\n>>>>\n>>>> My testing after running this with -d allows me to reliably see these\n>>>> layers being created with GDAT and GDOV chunks. Running the 'git\n>>>> commit-graph verify' command with the new code does not show those\n>>>> errors, even after adding commits and another layer to the split\n>>>> commit-graph.\n>>>>\n>>>> I look forward to any additional insights you might have here.\n>>>\n>>> I don't really know why, but now I've become unable to reproduce it\n>>> again. I think we should just go with your patch 5/4 on top -- it does\n>>> fix the most important issue, which is the `die()` I saw on almost all\n>>> commands. The second part about the warnings I'm just not sure about,\n>>> but I don't think it should stop this patch series given my own\n>>> uncertainty.\n>>\n>> Thanks for following up. I agree that with 5/4 we should be safe.\n>>\n>> I'll remain available to quickly respond if anything else surprising\n>> comes up in this area.\n>>\n>> Thanks!\n>> -Stolee\n> \n> There is another surprise I hit today in the context of generation\n> numbers. In production, I found the following bug:\n\n\"In production\" makes me think this is on a version of Git without\nthese patches. Am I right?\n\n> signal: aborted (core dumped): BUG: chunk-format.c:88: expected to write 8 bytes to chunk 47444f56, but wrote 168304 instead\n> \n> 47444f56 is the GENERATION_DATA_OVERFLOW chunk ID, and seemingly the\n> precomputed size we intended to write was mismatching the data we have\n> actually been writing to disk. And I think this stems from a mismatch in\n> how we precompute the number of generation data overflows compared to\n> how we're actually writing the data to disk:\n\nYes, and this count was supposed to be fixed in patch v3 3/5.\n\nThe issue is that we skip the increment if the commit was already\nparsed, so we would undercount. The fix was to do a loop at the\nend focused on counting these overflows.\n\nIf you have v3 applied and you still get this error, then we need\nto look closely at this issue.\n\nFor your production's sake, you might want to set\n\"commitGraph.generationVersion=1\" in your config until you have the\nfixes from this series.\n\nThanks,\n-Stolee\n"}]}