{"thread":{"id":"48647","subject":"[PATCH v4 01/21] commit-graph: UNLEAK before die()","startedAt":"2018-06-04T16:52:20Z","lastAt":"2018-09-11T15:14:02Z","messageCount":125,"participants":["Derrick Stolee","Ævar Arnfjörð Bjarmason","Jakub Narębski","SZEDER Gábor","Junio C Hamano","Jonathan Tan","Eric Sunshine","Christian Couder"],"isPatch":true,"patchVersion":4,"patchTotal":21},"messages":[{"id":"349257","messageId":"20180604165200.29261-2-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 01/21] commit-graph: UNLEAK before die()","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:52:14Z","receivedAt":"2018-06-04T16:52:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 37420ae0fd..f0875b8bf3 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -51,8 +51,11 @@ static int graph_read(int argc, const char **argv)\n \tgraph_name = get_commit_graph_filename(opts.obj_dir);\n \tgraph = load_commit_graph_one(graph_name);\n \n-\tif (!graph)\n+\tif (!graph) {\n+\t\tUNLEAK(graph_name);\n \t\tdie(\"graph file %s does not exist\", graph_name);\n+\t}\n+\n \tFREE_AND_NULL(graph_name);\n \n \tprintf(\"header: %08x %d %d %d %d\\n\",\n-- \n2.18.0.rc1\n\n"},{"id":"349258","messageId":"20180604165200.29261-1-dstolee@microsoft.com","threadId":"48647","inReplyTo":null,"subject":"[PATCH v4 00/21] Integrate commit-graph into 'fsck' and 'gc'","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:52:12Z","receivedAt":"2018-06-04T16:52:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Thanks for the feedback on v3. There were several small cleanups, but\nperhaps the biggest change is the addition of \"commit-graph: use\nstring-list API for input\" which makes \"commit-graph: add '--reachable'\noption\" much simpler.\n\nThe inter-diff is still reasonably large, but I'll send it in a\nfollow-up PR.\n\nThanks,\n-Stolee\n\nDerrick Stolee (21):\n commit-graph: UNLEAK before die()\n commit-graph: fix GRAPH_MIN_SIZE\n commit-graph: parse commit from chosen graph\n commit: force commit to parse from object database\n commit-graph: load a root tree from specific graph\n commit-graph: add 'verify' subcommand\n commit-graph: verify catches corrupt signature\n commit-graph: verify required chunks are present\n commit-graph: verify corrupt OID fanout and lookup\n commit-graph: verify objects exist\n commit-graph: verify root tree OIDs\n commit-graph: verify parent list\n commit-graph: verify generation number\n commit-graph: verify commit date\n commit-graph: test for corrupted octopus edge\n commit-graph: verify contents match checksum\n fsck: verify commit-graph\n commit-graph: use string-list API for input\n commit-graph: add '--reachable' option\n gc: automatically write commit-graph files\n commit-graph: update design document\n\n Documentation/config.txt | 10 +-\n Documentation/git-commit-graph.txt | 14 +-\n Documentation/git-fsck.txt | 3 +\n Documentation/git-gc.txt | 4 +\n Documentation/technical/commit-graph.txt | 22 --\n builtin/commit-graph.c | 98 ++++++---\n builtin/fsck.c | 21 ++\n builtin/gc.c | 6 +\n commit-graph.c | 248 +++++++++++++++++++++--\n commit-graph.h | 10 +-\n commit.c | 9 +-\n commit.h | 1 +\n t/t5318-commit-graph.sh | 201 ++++++++++++++++++\n 13 files changed, 569 insertions(+), 78 deletions(-)\n\n-- \n2.18.0.rc1\n\n"},{"id":"349259","messageId":"20180604165200.29261-3-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 02/21] commit-graph: fix GRAPH_MIN_SIZE","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:52:17Z","receivedAt":"2018-06-04T16:52:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The GRAPH_MIN_SIZE macro should be the smallest size of a parsable\ncommit-graph file. However, the minimum number of chunks was wrong.\nIt is possible to write a commit-graph file with zero commits, and\nthat violates this macro's value.\n\nRewrite the macro, and use extra macros to better explain the magic\nconstants.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex bb54c1214c..c09e87c3c2 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -34,10 +34,11 @@\n \n #define GRAPH_LAST_EDGE 0x80000000\n \n+#define GRAPH_HEADER_SIZE 8\n #define GRAPH_FANOUT_SIZE (4 * 256)\n #define GRAPH_CHUNKLOOKUP_WIDTH 12\n-#define GRAPH_MIN_SIZE (5 * GRAPH_CHUNKLOOKUP_WIDTH + GRAPH_FANOUT_SIZE + \\\n-\t\t\tGRAPH_OID_LEN + 8)\n+#define GRAPH_MIN_SIZE (GRAPH_HEADER_SIZE + 4 * GRAPH_CHUNKLOOKUP_WIDTH \\\n+\t\t\t+ GRAPH_FANOUT_SIZE + GRAPH_OID_LEN)\n \n char *get_commit_graph_filename(const char *obj_dir)\n {\n-- \n2.18.0.rc1\n\n"},{"id":"349260","messageId":"20180604165200.29261-4-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 03/21] commit-graph: parse commit from chosen graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:52:19Z","receivedAt":"2018-06-04T16:52:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Before verifying a commit-graph file against the object database, we\nneed to parse all commits from the given commit-graph file. Create\nparse_commit_in_graph_one() to target a given struct commit_graph.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 18 +++++++++++++++---\n 1 file changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex c09e87c3c2..472b41ec30 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -311,7 +311,7 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \t}\n }\n \n-int parse_commit_in_graph(struct commit *item)\n+static int parse_commit_in_graph_one(struct commit_graph *g, struct commit *item)\n {\n \tuint32_t pos;\n \n@@ -319,9 +319,21 @@ int parse_commit_in_graph(struct commit *item)\n \t\treturn 0;\n \tif (item->object.parsed)\n \t\treturn 1;\n+\n+\tif (find_commit_in_graph(item, g, &pos))\n+\t\treturn fill_commit_in_graph(item, g, pos);\n+\n+\treturn 0;\n+}\n+\n+int parse_commit_in_graph(struct commit *item)\n+{\n+\tif (!core_commit_graph)\n+\t\treturn 0;\n+\n \tprepare_commit_graph();\n-\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n-\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n+\tif (commit_graph)\n+\t\treturn parse_commit_in_graph_one(commit_graph, item);\n \treturn 0;\n }\n \n-- \n2.18.0.rc1\n\n"},{"id":"349261","messageId":"20180604165200.29261-6-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 05/21] commit-graph: load a root tree from specific graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:52:24Z","receivedAt":"2018-06-04T16:52:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When lazy-loading a tree for a commit, it will be important to select\nthe tree from a specific struct commit_graph. Create a new method that\nspecifies the commit-graph file and use that in\nget_commit_tree_in_graph().\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 12 +++++++++---\n 1 file changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 472b41ec30..fee8437ce3 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -359,14 +359,20 @@ static struct tree *load_tree_for_commit(struct commit_graph *g, struct commit *\n \treturn c->maybe_tree;\n }\n \n-struct tree *get_commit_tree_in_graph(const struct commit *c)\n+static struct tree *get_commit_tree_in_graph_one(struct commit_graph *g,\n+\t\t\t\t\t\t const struct commit *c)\n {\n \tif (c->maybe_tree)\n \t\treturn c->maybe_tree;\n \tif (c->graph_pos == COMMIT_NOT_FROM_GRAPH)\n-\t\tBUG(\"get_commit_tree_in_graph called from non-commit-graph commit\");\n+\t\tBUG(\"get_commit_tree_in_graph_one called from non-commit-graph commit\");\n+\n+\treturn load_tree_for_commit(g, (struct commit *)c);\n+}\n \n-\treturn load_tree_for_commit(commit_graph, (struct commit *)c);\n+struct tree *get_commit_tree_in_graph(const struct commit *c)\n+{\n+\treturn get_commit_tree_in_graph_one(commit_graph, c);\n }\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n-- \n2.18.0.rc1\n\n"},{"id":"349262","messageId":"20180604165200.29261-8-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 07/21] commit-graph: verify catches corrupt signature","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:52:28Z","receivedAt":"2018-06-04T16:52:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is the first of several commits that add a test to check that\n'git commit-graph verify' catches corruption in the commit-graph\nfile. The first test checks that the command catches an error in\nthe file signature. This is a check that exists in the existing\ncommit-graph reading code.\n\nAdd a helper method 'corrupt_graph_and_verify' to the test script\nt5318-commit-graph.sh. This helper corrupts the commit-graph file\nat a certain location, runs 'git commit-graph verify', and reports\nthe output to the 'err' file. This data is filtered to remove the\nlines added by 'test_must_fail' when the test is run verbosely.\nThen, the output is checked to contain a specific error message.\n\nMost messages from 'git commit-graph verify' will not be marked\nfor translation. There will be one exception: the message that\nreports an invalid checksum will be marked for translation, as that\nis the only message that is intended for a typical user.\n\nHelped-by: Szeder Gábor \nSigned-off-by: Derrick Stolee \n---\n t/t5318-commit-graph.sh | 43 +++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 43 insertions(+)\n\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 0830ef9fdd..c0c1ff09b9 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -235,9 +235,52 @@ test_expect_success 'perform fast-forward merge in full repo' '\n \ttest_cmp expect output\n '\n \n+# the verify tests below expect the commit-graph to contain\n+# exactly the commits reachable from the commits/8 branch.\n+# If the file changes the set of commits in the list, then the\n+# offsets into the binary file will result in different edits\n+# and the tests will likely break.\n+\n test_expect_success 'git commit-graph verify' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit rev-parse commits/8 | git commit-graph write --stdin-commits &&\n \tgit commit-graph verify >output\n '\n \n+GRAPH_BYTE_VERSION=4\n+GRAPH_BYTE_HASH=5\n+\n+# usage: corrupt_graph_and_verify \n+# Manipulates the commit-graph file at the position\n+# by inserting the data, then runs 'git commit-graph verify'\n+# and places the output in the file 'err'. Test 'err' for\n+# the given string.\n+corrupt_graph_and_verify() {\n+\tpos=$1\n+\tdata=\"${2:-\\0}\"\n+\tgrepstr=$3\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n+\tcp $objdir/info/commit-graph commit-graph-backup &&\n+\tprintf \"$data\" | dd of=\"$objdir/info/commit-graph\" bs=1 seek=\"$pos\" conv=notrunc &&\n+\ttest_must_fail git commit-graph verify 2>test_err &&\n+\tgrep -v \"^+\" test_err >err\n+\ttest_i18ngrep \"$grepstr\" err\n+}\n+\n+test_expect_success 'detect bad signature' '\n+\tcorrupt_graph_and_verify 0 \"\\0\" \\\n+\t\t\"graph signature\"\n+'\n+\n+test_expect_success 'detect bad version' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\02\" \\\n+\t\t\"graph version\"\n+'\n+\n+test_expect_success 'detect bad hash version' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_HASH \"\\02\" \\\n+\t\t\"hash version\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349263","messageId":"20180604165200.29261-7-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 06/21] commit-graph: add 'verify' subcommand","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:52:26Z","receivedAt":"2018-06-04T16:52:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"If the commit-graph file becomes corrupt, we need a way to verify\nthat its contents match the object database. In the manner of\n'git fsck' we will implement a 'git commit-graph verify' subcommand\nto report all issues with the file.\n\nAdd the 'verify' subcommand to the 'commit-graph' builtin and its\ndocumentation. The subcommand is currently a no-op except for\nloading the commit-graph into memory, which may trigger run-time\nerrors that would be caught by normal use. Add a simple test that\nensures the command returns a zero error code.\n\nIf no commit-graph file exists, this is an acceptable state. Do\nnot report any errors.\n\nHelped-by: Ramsay Jones \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 6 +++++\n builtin/commit-graph.c | 38 ++++++++++++++++++++++++++++++\n commit-graph.c | 23 ++++++++++++++++++\n commit-graph.h | 2 ++\n t/t5318-commit-graph.sh | 10 ++++++++\n 5 files changed, 79 insertions(+)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 4c97b555cc..a222cfab08 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git commit-graph read' [--object-dir ]\n+'git commit-graph verify' [--object-dir ]\n 'git commit-graph write' [--object-dir ]\n \n \n@@ -52,6 +53,11 @@ existing commit-graph file.\n Read a graph file given by the commit-graph file and output basic\n details about the graph file. Used for debugging purposes.\n \n+'verify'::\n+\n+Read the commit-graph file and verify its contents against the object\n+database. Used to check for corrupted data.\n+\n \n EXAMPLES\n --------\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex f0875b8bf3..3079cde6f9 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -8,10 +8,16 @@\n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n+\tN_(\"git commit-graph verify [--object-dir ]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n+static const char * const builtin_commit_graph_verify_usage[] = {\n+\tN_(\"git commit-graph verify [--object-dir ]\"),\n+\tNULL\n+};\n+\n static const char * const builtin_commit_graph_read_usage[] = {\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tNULL\n@@ -29,6 +35,36 @@ static struct opts_commit_graph {\n \tint append;\n } opts;\n \n+\n+static int graph_verify(int argc, const char **argv)\n+{\n+\tstruct commit_graph *graph = NULL;\n+\tchar *graph_name;\n+\n+\tstatic struct option builtin_commit_graph_verify_options[] = {\n+\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n+\t\t\t N_(\"dir\"),\n+\t\t\t N_(\"The object directory to store the graph\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t builtin_commit_graph_verify_options,\n+\t\t\t builtin_commit_graph_verify_usage, 0);\n+\n+\tif (!opts.obj_dir)\n+\t\topts.obj_dir = get_object_directory();\n+\n+\tgraph_name = get_commit_graph_filename(opts.obj_dir);\n+\tgraph = load_commit_graph_one(graph_name);\n+\tFREE_AND_NULL(graph_name);\n+\n+\tif (!graph)\n+\t\treturn 0;\n+\n+\treturn verify_commit_graph(graph);\n+}\n+\n static int graph_read(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n@@ -165,6 +201,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \tif (argc > 0) {\n \t\tif (!strcmp(argv[0], \"read\"))\n \t\t\treturn graph_read(argc, argv);\n+\t\tif (!strcmp(argv[0], \"verify\"))\n+\t\t\treturn graph_verify(argc, argv);\n \t\tif (!strcmp(argv[0], \"write\"))\n \t\t\treturn graph_write(argc, argv);\n \t}\ndiff --git a/commit-graph.c b/commit-graph.c\nindex fee8437ce3..c860cbe721 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -824,3 +824,26 @@ void write_commit_graph(const char *obj_dir,\n \toids.alloc = 0;\n \toids.nr = 0;\n }\n+\n+static int verify_commit_graph_error;\n+\n+static void graph_report(const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tverify_commit_graph_error = 1;\n+\n+\tva_start(ap, fmt);\n+\tvfprintf(stderr, fmt, ap);\n+\tfprintf(stderr, \"\\n\");\n+\tva_end(ap);\n+}\n+\n+int verify_commit_graph(struct commit_graph *g)\n+{\n+\tif (!g) {\n+\t\tgraph_report(\"no commit-graph file loaded\");\n+\t\treturn 1;\n+\t}\n+\n+\treturn verify_commit_graph_error;\n+}\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 96cccb10f3..71a39c5a57 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -53,4 +53,6 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tint nr_commits,\n \t\t\tint append);\n \n+int verify_commit_graph(struct commit_graph *g);\n+\n #endif\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 59d0be2877..0830ef9fdd 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -11,6 +11,11 @@ test_expect_success 'setup full repo' '\n \tobjdir=\".git/objects\"\n '\n \n+test_expect_success 'verify graph with no graph file' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph verify\n+'\n+\n test_expect_success 'write graph with no packs' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write --object-dir . &&\n@@ -230,4 +235,9 @@ test_expect_success 'perform fast-forward merge in full repo' '\n \ttest_cmp expect output\n '\n \n+test_expect_success 'git commit-graph verify' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph verify >output\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349264","messageId":"20180604165200.29261-9-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 08/21] commit-graph: verify required chunks are present","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:52:30Z","receivedAt":"2018-06-04T16:52:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file requires the following three chunks:\n\n* OID Fanout\n* OID Lookup\n* Commit Data\n\nIf any of these are missing, then the 'verify' subcommand should\nreport a failure. This includes the chunk IDs malformed or the\nchunk count is truncated.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 9 +++++++++\n t/t5318-commit-graph.sh | 29 +++++++++++++++++++++++++++++\n 2 files changed, 38 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex c860cbe721..25d5edea82 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -845,5 +845,14 @@ int verify_commit_graph(struct commit_graph *g)\n \t\treturn 1;\n \t}\n \n+\tverify_commit_graph_error = 0;\n+\n+\tif (!g->chunk_oid_fanout)\n+\t\tgraph_report(\"commit-graph is missing the OID Fanout chunk\");\n+\tif (!g->chunk_oid_lookup)\n+\t\tgraph_report(\"commit-graph is missing the OID Lookup chunk\");\n+\tif (!g->chunk_commit_data)\n+\t\tgraph_report(\"commit-graph is missing the Commit Data chunk\");\n+\n \treturn verify_commit_graph_error;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex c0c1ff09b9..846396665e 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -249,6 +249,15 @@ test_expect_success 'git commit-graph verify' '\n \n GRAPH_BYTE_VERSION=4\n GRAPH_BYTE_HASH=5\n+GRAPH_BYTE_CHUNK_COUNT=6\n+GRAPH_CHUNK_LOOKUP_OFFSET=8\n+GRAPH_CHUNK_LOOKUP_WIDTH=12\n+GRAPH_CHUNK_LOOKUP_ROWS=5\n+GRAPH_BYTE_OID_FANOUT_ID=$GRAPH_CHUNK_LOOKUP_OFFSET\n+GRAPH_BYTE_OID_LOOKUP_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n+\t\t\t 1 \\* $GRAPH_CHUNK_LOOKUP_WIDTH))\n+GRAPH_BYTE_COMMIT_DATA_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n+\t\t\t 2 \\* $GRAPH_CHUNK_LOOKUP_WIDTH))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -283,4 +292,24 @@ test_expect_success 'detect bad hash version' '\n \t\t\"hash version\"\n '\n \n+test_expect_success 'detect low chunk count' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_CHUNK_COUNT \"\\02\" \\\n+\t\t\"missing the .* chunk\"\n+'\n+\n+test_expect_success 'detect missing OID fanout chunk' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_FANOUT_ID \"\\0\" \\\n+\t\t\"missing the OID Fanout chunk\"\n+'\n+\n+test_expect_success 'detect missing OID lookup chunk' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_LOOKUP_ID \"\\0\" \\\n+\t\t\"missing the OID Lookup chunk\"\n+'\n+\n+test_expect_success 'detect missing commit data chunk' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_DATA_ID \"\\0\" \\\n+\t\t\"missing the Commit Data chunk\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349265","messageId":"20180604165200.29261-10-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 09/21] commit-graph: verify corrupt OID fanout and lookup","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:52:33Z","receivedAt":"2018-06-04T16:52:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In the commit-graph file, the OID fanout chunk provides an index into\nthe OID lookup. The 'verify' subcommand should find incorrect values\nin the fanout.\n\nSimilarly, the 'verify' subcommand should find out-of-order values in\nthe OID lookup.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 36 ++++++++++++++++++++++++++++++++++++\n t/t5318-commit-graph.sh | 22 ++++++++++++++++++++++\n 2 files changed, 58 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 25d5edea82..a90cc2e6f4 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -840,6 +840,9 @@ static void graph_report(const char *fmt, ...)\n \n int verify_commit_graph(struct commit_graph *g)\n {\n+\tuint32_t i, cur_fanout_pos = 0;\n+\tstruct object_id prev_oid, cur_oid;\n+\n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n \t\treturn 1;\n@@ -854,5 +857,38 @@ int verify_commit_graph(struct commit_graph *g)\n \tif (!g->chunk_commit_data)\n \t\tgraph_report(\"commit-graph is missing the Commit Data chunk\");\n \n+\tif (verify_commit_graph_error)\n+\t\treturn verify_commit_graph_error;\n+\n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n+\n+\t\tif (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n+\t\t\tgraph_report(\"commit-graph has incorrect OID order: %s then %s\",\n+\t\t\t\t oid_to_hex(&prev_oid),\n+\t\t\t\t oid_to_hex(&cur_oid));\n+\n+\t\toidcpy(&prev_oid, &cur_oid);\n+\n+\t\twhile (cur_oid.hash[0] > cur_fanout_pos) {\n+\t\t\tuint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n+\t\t\tif (i != fanout_value)\n+\t\t\t\tgraph_report(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\",\n+\t\t\t\t\t cur_fanout_pos, fanout_value, i);\n+\n+\t\t\tcur_fanout_pos++;\n+\t\t}\n+\t}\n+\n+\twhile (cur_fanout_pos < 256) {\n+\t\tuint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n+\n+\t\tif (g->num_commits != fanout_value)\n+\t\t\tgraph_report(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\",\n+\t\t\t\t cur_fanout_pos, fanout_value, i);\n+\n+\t\tcur_fanout_pos++;\n+\t}\n+\n \treturn verify_commit_graph_error;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 846396665e..c29eae47c9 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -247,6 +247,7 @@ test_expect_success 'git commit-graph verify' '\n \tgit commit-graph verify >output\n '\n \n+HASH_LEN=20\n GRAPH_BYTE_VERSION=4\n GRAPH_BYTE_HASH=5\n GRAPH_BYTE_CHUNK_COUNT=6\n@@ -258,6 +259,12 @@ GRAPH_BYTE_OID_LOOKUP_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n \t\t\t 1 \\* $GRAPH_CHUNK_LOOKUP_WIDTH))\n GRAPH_BYTE_COMMIT_DATA_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n \t\t\t 2 \\* $GRAPH_CHUNK_LOOKUP_WIDTH))\n+GRAPH_FANOUT_OFFSET=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n+\t\t $GRAPH_CHUNK_LOOKUP_WIDTH \\* $GRAPH_CHUNK_LOOKUP_ROWS))\n+GRAPH_BYTE_FANOUT1=$(($GRAPH_FANOUT_OFFSET + 4 \\* 4))\n+GRAPH_BYTE_FANOUT2=$(($GRAPH_FANOUT_OFFSET + 4 \\* 255))\n+GRAPH_OID_LOOKUP_OFFSET=$(($GRAPH_FANOUT_OFFSET + 4 \\* 256))\n+GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -312,4 +319,19 @@ test_expect_success 'detect missing commit data chunk' '\n \t\t\"missing the Commit Data chunk\"\n '\n \n+test_expect_success 'detect incorrect fanout' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_FANOUT1 \"\\01\" \\\n+\t\t\"fanout value\"\n+'\n+\n+test_expect_success 'detect incorrect fanout final value' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_FANOUT2 \"\\01\" \\\n+\t\t\"fanout value\"\n+'\n+\n+test_expect_success 'detect incorrect OID order' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_LOOKUP_ORDER \"\\01\" \\\n+\t\t\"incorrect OID order\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349266","messageId":"20180604165200.29261-12-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 11/21] commit-graph: verify root tree OIDs","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:52:39Z","receivedAt":"2018-06-04T16:52:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The 'verify' subcommand must compare the commit content parsed from the\ncommit-graph against the content in the object database. Use\nlookup_commit() and parse_commit_in_graph_one() to parse the commits\nfrom the graph and compare against a commit that is loaded separately\nand parsed directly from the object database.\n\nAdd checks for the root tree OID.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 17 ++++++++++++++++-\n t/t5318-commit-graph.sh | 7 +++++++\n 2 files changed, 23 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0cf1b61d80..50a8e27910 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -862,6 +862,8 @@ int verify_commit_graph(struct commit_graph *g)\n \t\treturn verify_commit_graph_error;\n \n \tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct commit *graph_commit;\n+\n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n \t\tif (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n@@ -879,6 +881,11 @@ int verify_commit_graph(struct commit_graph *g)\n \n \t\t\tcur_fanout_pos++;\n \t\t}\n+\n+\t\tgraph_commit = lookup_commit(&cur_oid);\n+\t\tif (!parse_commit_in_graph_one(g, graph_commit))\n+\t\t\tgraph_report(\"failed to parse %s from commit-graph\",\n+\t\t\t\t oid_to_hex(&cur_oid));\n \t}\n \n \twhile (cur_fanout_pos < 256) {\n@@ -895,16 +902,24 @@ int verify_commit_graph(struct commit_graph *g)\n \t\treturn verify_commit_graph_error;\n \n \tfor (i = 0; i < g->num_commits; i++) {\n-\t\tstruct commit *odb_commit;\n+\t\tstruct commit *graph_commit, *odb_commit;\n \n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n+\t\tgraph_commit = lookup_commit(&cur_oid);\n \t\todb_commit = (struct commit *)create_object(cur_oid.hash, alloc_commit_node());\n \t\tif (parse_commit_internal(odb_commit, 0, 0)) {\n \t\t\tgraph_report(\"failed to parse %s from object database\",\n \t\t\t\t oid_to_hex(&cur_oid));\n \t\t\tcontinue;\n \t\t}\n+\n+\t\tif (oidcmp(&get_commit_tree_in_graph_one(g, graph_commit)->object.oid,\n+\t\t\t get_commit_tree_oid(odb_commit)))\n+\t\t\tgraph_report(\"root tree OID for commit %s in commit-graph is %s != %s\",\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t oid_to_hex(get_commit_tree_oid(graph_commit)),\n+\t\t\t\t oid_to_hex(get_commit_tree_oid(odb_commit)));\n \t}\n \n \treturn verify_commit_graph_error;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex cf60e48496..c0c1248eda 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -267,6 +267,8 @@ GRAPH_BYTE_FANOUT2=$(($GRAPH_FANOUT_OFFSET + 4 \\* 255))\n GRAPH_OID_LOOKUP_OFFSET=$(($GRAPH_FANOUT_OFFSET + 4 \\* 256))\n GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8))\n GRAPH_BYTE_OID_LOOKUP_MISSING=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 4 + 10))\n+GRAPH_COMMIT_DATA_OFFSET=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* $NUM_COMMITS))\n+GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -341,4 +343,9 @@ test_expect_success 'detect OID not in object database' '\n \t\t\"from object database\"\n '\n \n+test_expect_success 'detect incorrect tree OID' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_TREE \"\\01\" \\\n+\t\t\"root tree OID for commit\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349267","messageId":"20180604165200.29261-13-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 12/21] commit-graph: verify parent list","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:52:41Z","receivedAt":"2018-06-04T16:52:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file stores parents in a two-column portion of the\ncommit data chunk. If there is only one parent, then the second column\nstores 0xFFFFFFFF to indicate no second parent.\n\nThe 'verify' subcommand checks the parent list for the commit loaded\nfrom the commit-graph and the one parsed from the object database. Test\nthese checks for corrupt parents, too many parents, and wrong parents.\n\nAdd a boundary check to insert_parent_or_die() for when the parent\nposition value is out of range.\n\nThe octopus merge will be tested in a later commit.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 28 ++++++++++++++++++++++++++++\n t/t5318-commit-graph.sh | 18 ++++++++++++++++++\n 2 files changed, 46 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 50a8e27910..d2a86e329e 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -240,6 +240,9 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n \tstruct commit *c;\n \tstruct object_id oid;\n \n+\tif (pos >= g->num_commits)\n+\t\tdie(\"invalid parent position %\"PRIu64, pos);\n+\n \thashcpy(oid.hash, g->chunk_oid_lookup + g->hash_len * pos);\n \tc = lookup_commit(&oid);\n \tif (!c)\n@@ -903,6 +906,7 @@ int verify_commit_graph(struct commit_graph *g)\n \n \tfor (i = 0; i < g->num_commits; i++) {\n \t\tstruct commit *graph_commit, *odb_commit;\n+\t\tstruct commit_list *graph_parents, *odb_parents;\n \n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n@@ -920,6 +924,30 @@ int verify_commit_graph(struct commit_graph *g)\n \t\t\t\t oid_to_hex(&cur_oid),\n \t\t\t\t oid_to_hex(get_commit_tree_oid(graph_commit)),\n \t\t\t\t oid_to_hex(get_commit_tree_oid(odb_commit)));\n+\n+\t\tgraph_parents = graph_commit->parents;\n+\t\todb_parents = odb_commit->parents;\n+\n+\t\twhile (graph_parents) {\n+\t\t\tif (odb_parents == NULL) {\n+\t\t\t\tgraph_report(\"commit-graph parent list for commit %s is too long\",\n+\t\t\t\t\t oid_to_hex(&cur_oid));\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (oidcmp(&graph_parents->item->object.oid, &odb_parents->item->object.oid))\n+\t\t\t\tgraph_report(\"commit-graph parent for %s is %s != %s\",\n+\t\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid),\n+\t\t\t\t\t oid_to_hex(&odb_parents->item->object.oid));\n+\n+\t\t\tgraph_parents = graph_parents->next;\n+\t\t\todb_parents = odb_parents->next;\n+\t\t}\n+\n+\t\tif (odb_parents != NULL)\n+\t\t\tgraph_report(\"commit-graph parent list for commit %s terminates early\",\n+\t\t\t\t oid_to_hex(&cur_oid));\n \t}\n \n \treturn verify_commit_graph_error;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex c0c1248eda..ec0964112a 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -269,6 +269,9 @@ GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8))\n GRAPH_BYTE_OID_LOOKUP_MISSING=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 4 + 10))\n GRAPH_COMMIT_DATA_OFFSET=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* $NUM_COMMITS))\n GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n+GRAPH_BYTE_COMMIT_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN))\n+GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4))\n+GRAPH_BYTE_COMMIT_WRONG_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -348,4 +351,19 @@ test_expect_success 'detect incorrect tree OID' '\n \t\t\"root tree OID for commit\"\n '\n \n+test_expect_success 'detect incorrect parent int-id' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_PARENT \"\\01\" \\\n+\t\t\"invalid parent\"\n+'\n+\n+test_expect_success 'detect extra parent int-id' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_EXTRA_PARENT \"\\00\" \\\n+\t\t\"is too long\"\n+'\n+\n+test_expect_success 'detect wrong parent' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_WRONG_PARENT \"\\01\" \\\n+\t\t\"commit-graph parent for\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349268","messageId":"20180604165200.29261-11-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 10/21] commit-graph: verify objects exist","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:52:35Z","receivedAt":"2018-06-04T16:52:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In the 'verify' subcommand, load commits directly from the object\ndatabase to ensure they exist. Parse by skipping the commit-graph.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 17 +++++++++++++++++\n t/t5318-commit-graph.sh | 7 +++++++\n 2 files changed, 24 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a90cc2e6f4..0cf1b61d80 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -239,6 +239,7 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n {\n \tstruct commit *c;\n \tstruct object_id oid;\n+\n \thashcpy(oid.hash, g->chunk_oid_lookup + g->hash_len * pos);\n \tc = lookup_commit(&oid);\n \tif (!c)\n@@ -890,5 +891,21 @@ int verify_commit_graph(struct commit_graph *g)\n \t\tcur_fanout_pos++;\n \t}\n \n+\tif (verify_commit_graph_error)\n+\t\treturn verify_commit_graph_error;\n+\n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct commit *odb_commit;\n+\n+\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n+\n+\t\todb_commit = (struct commit *)create_object(cur_oid.hash, alloc_commit_node());\n+\t\tif (parse_commit_internal(odb_commit, 0, 0)) {\n+\t\t\tgraph_report(\"failed to parse %s from object database\",\n+\t\t\t\t oid_to_hex(&cur_oid));\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+\n \treturn verify_commit_graph_error;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex c29eae47c9..cf60e48496 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -247,6 +247,7 @@ test_expect_success 'git commit-graph verify' '\n \tgit commit-graph verify >output\n '\n \n+NUM_COMMITS=9\n HASH_LEN=20\n GRAPH_BYTE_VERSION=4\n GRAPH_BYTE_HASH=5\n@@ -265,6 +266,7 @@ GRAPH_BYTE_FANOUT1=$(($GRAPH_FANOUT_OFFSET + 4 \\* 4))\n GRAPH_BYTE_FANOUT2=$(($GRAPH_FANOUT_OFFSET + 4 \\* 255))\n GRAPH_OID_LOOKUP_OFFSET=$(($GRAPH_FANOUT_OFFSET + 4 \\* 256))\n GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8))\n+GRAPH_BYTE_OID_LOOKUP_MISSING=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 4 + 10))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -334,4 +336,9 @@ test_expect_success 'detect incorrect OID order' '\n \t\t\"incorrect OID order\"\n '\n \n+test_expect_success 'detect OID not in object database' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_LOOKUP_MISSING \"\\01\" \\\n+\t\t\"from object database\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349269","messageId":"20180604165200.29261-19-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 18/21] commit-graph: use string-list API for input","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:52:56Z","receivedAt":"2018-06-04T16:53:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 39 +++++++++++++--------------------------\n commit-graph.c | 15 +++++++--------\n commit-graph.h | 7 +++----\n 3 files changed, 23 insertions(+), 38 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 3079cde6f9..d8eb8278b3 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -118,13 +118,9 @@ static int graph_read(int argc, const char **argv)\n \n static int graph_write(int argc, const char **argv)\n {\n-\tconst char **pack_indexes = NULL;\n-\tint packs_nr = 0;\n-\tconst char **commit_hex = NULL;\n-\tint commits_nr = 0;\n-\tconst char **lines = NULL;\n-\tint lines_nr = 0;\n-\tint lines_alloc = 0;\n+\tstruct string_list *pack_indexes = NULL;\n+\tstruct string_list *commit_hex = NULL;\n+\tstruct string_list lines;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n@@ -150,32 +146,23 @@ static int graph_write(int argc, const char **argv)\n \n \tif (opts.stdin_packs || opts.stdin_commits) {\n \t\tstruct strbuf buf = STRBUF_INIT;\n-\t\tlines_nr = 0;\n-\t\tlines_alloc = 128;\n-\t\tALLOC_ARRAY(lines, lines_alloc);\n-\n-\t\twhile (strbuf_getline(&buf, stdin) != EOF) {\n-\t\t\tALLOC_GROW(lines, lines_nr + 1, lines_alloc);\n-\t\t\tlines[lines_nr++] = strbuf_detach(&buf, NULL);\n-\t\t}\n-\n-\t\tif (opts.stdin_packs) {\n-\t\t\tpack_indexes = lines;\n-\t\t\tpacks_nr = lines_nr;\n-\t\t}\n-\t\tif (opts.stdin_commits) {\n-\t\t\tcommit_hex = lines;\n-\t\t\tcommits_nr = lines_nr;\n-\t\t}\n+\t\tstring_list_init(&lines, 0);\n+\n+\t\twhile (strbuf_getline(&buf, stdin) != EOF)\n+\t\t\tstring_list_append(&lines, strbuf_detach(&buf, NULL));\n+\n+\t\tif (opts.stdin_packs)\n+\t\t\tpack_indexes = &lines;\n+\t\tif (opts.stdin_commits)\n+\t\t\tcommit_hex = &lines;\n \t}\n \n \twrite_commit_graph(opts.obj_dir,\n \t\t\t pack_indexes,\n-\t\t\t packs_nr,\n \t\t\t commit_hex,\n-\t\t\t commits_nr,\n \t\t\t opts.append);\n \n+\tstring_list_clear(&lines, 0);\n \treturn 0;\n }\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 05879de26c..c6070735c2 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -653,10 +653,8 @@ static void compute_generation_numbers(struct packed_commit_list* commits)\n }\n \n void write_commit_graph(const char *obj_dir,\n-\t\t\tconst char **pack_indexes,\n-\t\t\tint nr_packs,\n-\t\t\tconst char **commit_hex,\n-\t\t\tint nr_commits,\n+\t\t\tstruct string_list *pack_indexes,\n+\t\t\tstruct string_list *commit_hex,\n \t\t\tint append)\n {\n \tstruct packed_oid_list oids;\n@@ -697,10 +695,10 @@ void write_commit_graph(const char *obj_dir,\n \t\tint dirlen;\n \t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n \t\tdirlen = packname.len;\n-\t\tfor (i = 0; i < nr_packs; i++) {\n+\t\tfor (i = 0; i < pack_indexes->nr; i++) {\n \t\t\tstruct packed_git *p;\n \t\t\tstrbuf_setlen(&packname, dirlen);\n-\t\t\tstrbuf_addstr(&packname, pack_indexes[i]);\n+\t\t\tstrbuf_addstr(&packname, pack_indexes->items[i].string);\n \t\t\tp = add_packed_git(packname.buf, packname.len, 1);\n \t\t\tif (!p)\n \t\t\t\tdie(\"error adding pack %s\", packname.buf);\n@@ -713,12 +711,13 @@ void write_commit_graph(const char *obj_dir,\n \t}\n \n \tif (commit_hex) {\n-\t\tfor (i = 0; i < nr_commits; i++) {\n+\t\tfor (i = 0; i < commit_hex->nr; i++) {\n \t\t\tconst char *end;\n \t\t\tstruct object_id oid;\n \t\t\tstruct commit *result;\n \n-\t\t\tif (commit_hex[i] && parse_oid_hex(commit_hex[i], &oid, &end))\n+\t\t\tif (commit_hex->items[i].string &&\n+\t\t\t parse_oid_hex(commit_hex->items[i].string, &oid, &end))\n \t\t\t\tcontinue;\n \n \t\t\tresult = lookup_commit_reference_gently(&oid, 1);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 71a39c5a57..66661e1fc5 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -2,6 +2,7 @@\n #define COMMIT_GRAPH_H\n \n #include \"git-compat-util.h\"\n+#include \"string-list.h\"\n \n char *get_commit_graph_filename(const char *obj_dir);\n \n@@ -47,10 +48,8 @@ struct commit_graph {\n struct commit_graph *load_commit_graph_one(const char *graph_file);\n \n void write_commit_graph(const char *obj_dir,\n-\t\t\tconst char **pack_indexes,\n-\t\t\tint nr_packs,\n-\t\t\tconst char **commit_hex,\n-\t\t\tint nr_commits,\n+\t\t\tstruct string_list *pack_indexes,\n+\t\t\tstruct string_list *commit_hex,\n \t\t\tint append);\n \n int verify_commit_graph(struct commit_graph *g);\n-- \n2.18.0.rc1\n\n"},{"id":"349270","messageId":"20180604165200.29261-21-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 20/21] gc: automatically write commit-graph files","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:53:01Z","receivedAt":"2018-06-04T16:53:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file is a very helpful feature for speeding up git\noperations. In order to make it more useful, make it possible to\nwrite the commit-graph file during standard garbage collection\noperations.\n\nAdd a 'gc.commitGraph' config setting that triggers writing a\ncommit-graph file after any non-trivial 'git gc' command. Defaults to\nfalse while the commit-graph feature matures. We specifically do not\nwant to have this on by default until the commit-graph feature is fully\nintegrated with history-modifying features like shallow clones.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 10 +++++++++-\n Documentation/git-gc.txt | 4 ++++\n builtin/gc.c | 6 ++++++\n t/t5318-commit-graph.sh | 14 ++++++++++++++\n 4 files changed, 33 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 11f027194e..d2eb3c8e9b 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -900,7 +900,8 @@ the `GIT_NOTES_REF` environment variable. See linkgit:git-notes[1].\n \n core.commitGraph::\n \tEnable git commit graph feature. Allows reading from the\n-\tcommit-graph file.\n+\tcommit-graph file. See `gc.commitGraph` for automatically\n+\tmaintaining the file.\n \n core.sparseCheckout::\n \tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n@@ -1553,6 +1554,13 @@ gc.autoDetach::\n \tMake `git gc --auto` return immediately and run in background\n \tif the system supports it. Default is true.\n \n+gc.commitGraph::\n+\tIf true, then gc will rewrite the commit-graph file when\n+\tlinkgit:git-gc[1] is run. When using linkgit:git-gc[1]\n+\t'--auto' the commit-graph will be updated if housekeeping is\n+\trequired. Default is false. See linkgit:git-commit-graph[1]\n+\tfor details.\n+\n gc.logExpiry::\n \tIf the file gc.log exists, then `git gc --auto` won't run\n \tunless that file is more than 'gc.logExpiry' old. Default is\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 571b5a7e3c..a6526b3592 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -119,6 +119,10 @@ The optional configuration variable `gc.packRefs` determines if\n it within all non-bare repos or it can be set to a boolean value.\n This defaults to true.\n \n+The optional configuration variable `gc.commitGraph` determines if\n+'git gc' should run 'git commit-graph write'. This can be set to a\n+boolean value. This defaults to false.\n+\n The optional configuration variable `gc.aggressiveWindow` controls how\n much time is spent optimizing the delta compression of the objects in\n the repository when the --aggressive option is specified. The larger\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 77fa720bd0..efd214a59f 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -20,6 +20,7 @@\n #include \"argv-array.h\"\n #include \"commit.h\"\n #include \"packfile.h\"\n+#include \"commit-graph.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -34,6 +35,7 @@ static int aggressive_depth = 50;\n static int aggressive_window = 250;\n static int gc_auto_threshold = 6700;\n static int gc_auto_pack_limit = 50;\n+static int gc_commit_graph = 0;\n static int detach_auto = 1;\n static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n@@ -121,6 +123,7 @@ static void gc_config(void)\n \tgit_config_get_int(\"gc.aggressivedepth\", &aggressive_depth);\n \tgit_config_get_int(\"gc.auto\", &gc_auto_threshold);\n \tgit_config_get_int(\"gc.autopacklimit\", &gc_auto_pack_limit);\n+\tgit_config_get_bool(\"gc.commitgraph\", &gc_commit_graph);\n \tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n \tgit_config_get_expiry(\"gc.pruneexpire\", &prune_expire);\n \tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n@@ -480,6 +483,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tif (pack_garbage.nr > 0)\n \t\tclean_pack_garbage();\n \n+\tif (gc_commit_graph)\n+\t\twrite_commit_graph_reachable(get_object_directory(), 0);\n+\n \tif (auto_gc && too_many_loose_objects())\n \t\twarning(_(\"There are too many unreachable loose objects; \"\n \t\t\t\"run 'git prune' to remove them.\"));\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex ffb2ed7c95..b24e8b6689 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -245,6 +245,20 @@ test_expect_success 'perform fast-forward merge in full repo' '\n \ttest_cmp expect output\n '\n \n+test_expect_success 'check that gc computes commit-graph' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit --allow-empty -m \"blank\" &&\n+\tgit commit-graph write --reachable &&\n+\tcp $objdir/info/commit-graph commit-graph-before-gc &&\n+\tgit reset --hard HEAD~1 &&\n+\tgit config gc.commitGraph true &&\n+\tgit gc &&\n+\tcp $objdir/info/commit-graph commit-graph-after-gc &&\n+\t! test_cmp commit-graph-before-gc commit-graph-after-gc &&\n+\tgit commit-graph write --reachable &&\n+\ttest_cmp commit-graph-after-gc $objdir/info/commit-graph\n+'\n+\n # the verify tests below expect the commit-graph to contain\n # exactly the commits reachable from the commits/8 branch.\n # If the file changes the set of commits in the list, then the\n-- \n2.18.0.rc1\n\n"},{"id":"349271","messageId":"20180604165200.29261-20-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 19/21] commit-graph: add '--reachable' option","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:52:59Z","receivedAt":"2018-06-04T16:53:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When writing commit-graph files, it can be convenient to ask for all\nreachable commits (starting at the ref set) in the resulting file. This\nis particularly helpful when writing to stdin is complicated, such as a\nfuture integration with 'git gc'.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 8 ++++++--\n builtin/commit-graph.c | 16 ++++++++++++----\n commit-graph.c | 18 ++++++++++++++++++\n commit-graph.h | 1 +\n t/t5318-commit-graph.sh | 10 ++++++++++\n 5 files changed, 47 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex a222cfab08..dececb79d7 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -38,12 +38,16 @@ Write a commit graph file based on the commits found in packfiles.\n +\n With the `--stdin-packs` option, generate the new commit graph by\n walking objects only in the specified pack-indexes. (Cannot be combined\n-with --stdin-commits.)\n+with `--stdin-commits` or `--reachable`.)\n +\n With the `--stdin-commits` option, generate the new commit graph by\n walking commits starting at the commits specified in stdin as a list\n of OIDs in hex, one OID per line. (Cannot be combined with\n---stdin-packs.)\n+`--stdin-packs` or `--reachable`.)\n++\n+With the `--reachable` option, generate the new commit graph by walking\n+commits starting at all refs. (Cannot be combined with `--stdin-commits`\n+or `--stdin-packs`.)\n +\n With the `--append` option, include all commits that are present in the\n existing commit-graph file.\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex d8eb8278b3..76423b3fa5 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -9,7 +9,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph verify [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -24,12 +24,13 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n static struct opts_commit_graph {\n \tconst char *obj_dir;\n+\tint reachable;\n \tint stdin_packs;\n \tint stdin_commits;\n \tint append;\n@@ -126,6 +127,8 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n \t\t\tN_(\"dir\"),\n \t\t\tN_(\"The object directory to store the graph\")),\n+\t\tOPT_BOOL(0, \"reachable\", &opts.reachable,\n+\t\t\tN_(\"start walk at all refs\")),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\tN_(\"scan pack-indexes listed by stdin for commits\")),\n \t\tOPT_BOOL(0, \"stdin-commits\", &opts.stdin_commits,\n@@ -139,11 +142,16 @@ static int graph_write(int argc, const char **argv)\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\n \n-\tif (opts.stdin_packs && opts.stdin_commits)\n-\t\tdie(_(\"cannot use both --stdin-commits and --stdin-packs\"));\n+\tif (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)\n+\t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n \n+\tif (opts.reachable) {\n+\t\twrite_commit_graph_reachable(opts.obj_dir, opts.append);\n+\t\treturn 0;\n+\t}\n+\n \tif (opts.stdin_packs || opts.stdin_commits) {\n \t\tstruct strbuf buf = STRBUF_INIT;\n \t\tstring_list_init(&lines, 0);\ndiff --git a/commit-graph.c b/commit-graph.c\nindex c6070735c2..946bcfa98c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -7,6 +7,7 @@\n #include \"packfile.h\"\n #include \"commit.h\"\n #include \"object.h\"\n+#include \"refs.h\"\n #include \"revision.h\"\n #include \"sha1-lookup.h\"\n #include \"commit-graph.h\"\n@@ -652,6 +653,23 @@ static void compute_generation_numbers(struct packed_commit_list* commits)\n \t}\n }\n \n+static int add_ref_to_list(const char *refname,\n+\t\t\t const struct object_id *oid,\n+\t\t\t int flags, void *cb_data)\n+{\n+\tstruct string_list *list = (struct string_list*)cb_data;\n+\tstring_list_append(list, oid_to_hex(oid));\n+\treturn 0;\n+}\n+\n+void write_commit_graph_reachable(const char *obj_dir, int append)\n+{\n+\tstruct string_list list;\n+\tstring_list_init(&list, 1);\n+\tfor_each_ref(add_ref_to_list, &list);\n+\twrite_commit_graph(obj_dir, NULL, &list, append);\n+}\n+\n void write_commit_graph(const char *obj_dir,\n \t\t\tstruct string_list *pack_indexes,\n \t\t\tstruct string_list *commit_hex,\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 66661e1fc5..ee20f5e280 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -47,6 +47,7 @@ struct commit_graph {\n \n struct commit_graph *load_commit_graph_one(const char *graph_file);\n \n+void write_commit_graph_reachable(const char *obj_dir, int append);\n void write_commit_graph(const char *obj_dir,\n \t\t\tstruct string_list *pack_indexes,\n \t\t\tstruct string_list *commit_hex,\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 44d4c71f0b..ffb2ed7c95 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -205,6 +205,16 @@ test_expect_success 'build graph from commits with append' '\n graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n \n+test_expect_success 'build graph using --reachable' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph write --reachable &&\n+\ttest_path_is_file $objdir/info/commit-graph &&\n+\tgraph_read_expect \"11\" \"large_edges\"\n+'\n+\n+graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n+graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n+\n test_expect_success 'setup bare repo' '\n \tcd \"$TRASH_DIRECTORY\" &&\n \tgit clone --bare --no-local full bare &&\n-- \n2.18.0.rc1\n\n"},{"id":"349272","messageId":"20180604165200.29261-22-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 21/21] commit-graph: update design document","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:53:04Z","receivedAt":"2018-06-04T16:53:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph feature is now integrated with 'fsck' and 'gc',\nso remove those items from the \"Future Work\" section of the\ncommit-graph design document.\n\nAlso remove the section on lazy-loading trees, as that was completed\nin an earlier patch series.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 22 ----------------------\n 1 file changed, 22 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex e1a883eb46..c664acbd76 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -118,9 +118,6 @@ Future Work\n - The commit graph feature currently does not honor commit grafts. This can\n be remedied by duplicating or refactoring the current graft logic.\n \n-- The 'commit-graph' subcommand does not have a \"verify\" mode that is\n- necessary for integration with fsck.\n-\n - After computing and storing generation numbers, we must make graph\n walks aware of generation numbers to gain the performance benefits they\n enable. This will mostly be accomplished by swapping a commit-date-ordered\n@@ -130,25 +127,6 @@ Future Work\n - 'log --topo-order'\n - 'tag --merged'\n \n-- Currently, parse_commit_gently() requires filling in the root tree\n- object for a commit. This passes through lookup_tree() and consequently\n- lookup_object(). Also, it calls lookup_commit() when loading the parents.\n- These method calls check the ODB for object existence, even if the\n- consumer does not need the content. For example, we do not need the\n- tree contents when computing merge bases. Now that commit parsing is\n- removed from the computation time, these lookup operations are the\n- slowest operations keeping graph walks from being fast. Consider\n- loading these objects without verifying their existence in the ODB and\n- only loading them fully when consumers need them. Consider a method\n- such as \"ensure_tree_loaded(commit)\" that fully loads a tree before\n- using commit->tree.\n-\n-- The current design uses the 'commit-graph' subcommand to generate the graph.\n- When this feature stabilizes enough to recommend to most users, we should\n- add automatic graph writes to common operations that create many commits.\n- For example, one could compute a graph on 'clone', 'fetch', or 'repack'\n- commands.\n-\n - A server could provide a commit graph file as part of the network protocol\n to avoid extra calculations by clients. This feature is only of benefit if\n the user is willing to trust the file, because verifying the file is correct\n-- \n2.18.0.rc1\n\n"},{"id":"349273","messageId":"20180604165200.29261-14-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 13/21] commit-graph: verify generation number","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:52:43Z","receivedAt":"2018-06-04T16:53:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"While iterating through the commit parents, perform the generation\nnumber calculation and compare against the value stored in the\ncommit-graph.\n\nThe tests demonstrate that having a different set of parents affects\nthe generation number calculation, and this value propagates to\ndescendants. Hence, we drop the single-line condition on the output.\n\nSince Git will ship with the commit-graph feature without generation\nnumbers, we need to accept commit-graphs with all generation numbers\nequal to zero. In this case, ignore the generation number calculation.\n\nHowever, verify that we should never have a mix of zero and non-zero\ngeneration numbers. Create a test that sets one commit to generation\nzero and all following commits report a failure as they have non-zero\ngeneration in a file that contains generation number zero.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 34 ++++++++++++++++++++++++++++++++++\n t/t5318-commit-graph.sh | 11 +++++++++++\n 2 files changed, 45 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d2a86e329e..5faecae2a7 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -842,10 +842,14 @@ static void graph_report(const char *fmt, ...)\n \tva_end(ap);\n }\n \n+#define GENERATION_ZERO_EXISTS 1\n+#define GENERATION_NUMBER_EXISTS 2\n+\n int verify_commit_graph(struct commit_graph *g)\n {\n \tuint32_t i, cur_fanout_pos = 0;\n \tstruct object_id prev_oid, cur_oid;\n+\tint generation_zero = 0;\n \n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n@@ -907,6 +911,7 @@ int verify_commit_graph(struct commit_graph *g)\n \tfor (i = 0; i < g->num_commits; i++) {\n \t\tstruct commit *graph_commit, *odb_commit;\n \t\tstruct commit_list *graph_parents, *odb_parents;\n+\t\tuint32_t max_generation = 0;\n \n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n@@ -941,6 +946,9 @@ int verify_commit_graph(struct commit_graph *g)\n \t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid),\n \t\t\t\t\t oid_to_hex(&odb_parents->item->object.oid));\n \n+\t\t\tif (graph_parents->item->generation > max_generation)\n+\t\t\t\tmax_generation = graph_parents->item->generation;\n+\n \t\t\tgraph_parents = graph_parents->next;\n \t\t\todb_parents = odb_parents->next;\n \t\t}\n@@ -948,6 +956,32 @@ int verify_commit_graph(struct commit_graph *g)\n \t\tif (odb_parents != NULL)\n \t\t\tgraph_report(\"commit-graph parent list for commit %s terminates early\",\n \t\t\t\t oid_to_hex(&cur_oid));\n+\n+\t\tif (!graph_commit->generation) {\n+\t\t\tif (generation_zero == GENERATION_NUMBER_EXISTS)\n+\t\t\t\tgraph_report(\"commit-graph has generation number zero for commit %s, but non-zero elsewhere\",\n+\t\t\t\t\t oid_to_hex(&cur_oid));\n+\t\t\tgeneration_zero = GENERATION_ZERO_EXISTS;\n+\t\t} else if (generation_zero == GENERATION_ZERO_EXISTS)\n+\t\t\tgraph_report(\"commit-graph has non-zero generation number for commit %s, but zero elsewhere\",\n+\t\t\t\t oid_to_hex(&cur_oid));\n+\n+\t\tif (generation_zero == GENERATION_ZERO_EXISTS)\n+\t\t\tcontinue;\n+\n+\t\t/*\n+\t\t * If one of our parents has generation GENERATION_NUMBER_MAX, then\n+\t\t * our generation is also GENERATION_NUMBER_MAX. Decrement to avoid\n+\t\t * extra logic in the following condition.\n+\t\t */\n+\t\tif (max_generation == GENERATION_NUMBER_MAX)\n+\t\t\tmax_generation--;\n+\n+\t\tif (graph_commit->generation != max_generation + 1)\n+\t\t\tgraph_report(\"commit-graph generation for commit %s is %u != %u\",\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t graph_commit->generation,\n+\t\t\t\t max_generation + 1);\n \t}\n \n \treturn verify_commit_graph_error;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex ec0964112a..a6ea1341dc 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -272,6 +272,7 @@ GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n GRAPH_BYTE_COMMIT_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN))\n GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4))\n GRAPH_BYTE_COMMIT_WRONG_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3))\n+GRAPH_BYTE_COMMIT_GENERATION=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 11))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -366,4 +367,14 @@ test_expect_success 'detect wrong parent' '\n \t\t\"commit-graph parent for\"\n '\n \n+test_expect_success 'detect incorrect generation number' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_GENERATION \"\\070\" \\\n+\t\t\"generation for commit\"\n+'\n+\n+test_expect_success 'detect incorrect generation number' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_GENERATION \"\\01\" \\\n+\t\t\"non-zero generation number\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349274","messageId":"20180604165200.29261-18-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 17/21] fsck: verify commit-graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:52:53Z","receivedAt":"2018-06-04T16:53:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"If core.commitGraph is true, verify the contents of the commit-graph\nduring 'git fsck' using the 'git commit-graph verify' subcommand. Run\nthis check on all alternates, as well.\n\nWe use a new process for two reasons:\n\n1. The subcommand decouples the details of loading and verifying a\n commit-graph file from the other fsck details.\n\n2. The commit-graph verification requires the commits to be loaded\n in a specific order to guarantee we parse from the commit-graph\n file for some objects and from the object database for others.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-fsck.txt | 3 +++\n builtin/fsck.c | 21 +++++++++++++++++++++\n t/t5318-commit-graph.sh | 8 ++++++++\n 3 files changed, 32 insertions(+)\n\ndiff --git a/Documentation/git-fsck.txt b/Documentation/git-fsck.txt\nindex b9f060e3b2..ab9a93fb9b 100644\n--- a/Documentation/git-fsck.txt\n+++ b/Documentation/git-fsck.txt\n@@ -110,6 +110,9 @@ Any corrupt objects you will have to find in backups or other archives\n (i.e., you can just remove them and do an 'rsync' with some other site in\n the hopes that somebody else has the object you have corrupted).\n \n+If core.commitGraph is true, the commit-graph file will also be inspected\n+using 'git commit-graph verify'. See linkgit:git-commit-graph[1].\n+\n Extracted Diagnostics\n ---------------------\n \ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex ef78c6c00c..a6d5045b77 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -16,6 +16,7 @@\n #include \"streaming.h\"\n #include \"decorate.h\"\n #include \"packfile.h\"\n+#include \"run-command.h\"\n \n #define REACHABLE 0x0001\n #define SEEN 0x0002\n@@ -45,6 +46,7 @@ static int name_objects;\n #define ERROR_REACHABLE 02\n #define ERROR_PACK 04\n #define ERROR_REFS 010\n+#define ERROR_COMMIT_GRAPH 020\n \n static const char *describe_object(struct object *obj)\n {\n@@ -815,5 +817,24 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t}\n \n \tcheck_connectivity();\n+\n+\tif (core_commit_graph) {\n+\t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n+\t\tconst char *verify_argv[] = { \"commit-graph\", \"verify\", NULL, NULL, NULL, NULL };\n+\t\tcommit_graph_verify.argv = verify_argv;\n+\t\tcommit_graph_verify.git_cmd = 1;\n+\n+\t\tif (run_command(&commit_graph_verify))\n+\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n+\n+\t\tprepare_alt_odb();\n+\t\tfor (alt = alt_odb_list; alt; alt = alt->next) {\n+\t\t\tverify_argv[2] = \"--object-dir\";\n+\t\t\tverify_argv[3] = alt->path;\n+\t\t\tif (run_command(&commit_graph_verify))\n+\t\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n+\t\t}\n+\t}\n+\n \treturn errors_found;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 2297a44e7d..44d4c71f0b 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -399,4 +399,12 @@ test_expect_success 'detect invalid checksum hash' '\n \t\t\"incorrect checksum\"\n '\n \n+test_expect_success 'git fsck (checks commit-graph)' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit fsck &&\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_FOOTER \"\\00\" \\\n+\t\t\"incorrect checksum\" &&\n+\ttest_must_fail git fsck\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349275","messageId":"20180604165200.29261-17-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 16/21] commit-graph: verify contents match checksum","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:52:50Z","receivedAt":"2018-06-04T16:53:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file ends with a SHA1 hash of the previous contents. If\na commit-graph file has errors but the checksum hash is correct, then we\nknow that the problem is a bug in Git and not simply file corruption\nafter-the-fact.\n\nCompute the checksum right away so it is the first error that appears,\nand make the message translatable since this error can be \"corrected\" by\na user by simply deleting the file and recomputing. The rest of the\nerrors are useful only to developers.\n\nBe sure to continue checking the rest of the file data if the checksum\nis wrong. This is important for our tests, as we break the checksum as\nwe modify bytes of the commit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 16 ++++++++++++++--\n t/t5318-commit-graph.sh | 6 ++++++\n 2 files changed, 20 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 47fdd62e88..05879de26c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -829,6 +829,7 @@ void write_commit_graph(const char *obj_dir,\n \toids.nr = 0;\n }\n \n+#define VERIFY_COMMIT_GRAPH_ERROR_HASH 2\n static int verify_commit_graph_error;\n \n static void graph_report(const char *fmt, ...)\n@@ -848,8 +849,10 @@ static void graph_report(const char *fmt, ...)\n int verify_commit_graph(struct commit_graph *g)\n {\n \tuint32_t i, cur_fanout_pos = 0;\n-\tstruct object_id prev_oid, cur_oid;\n+\tstruct object_id prev_oid, cur_oid, checksum;\n \tint generation_zero = 0;\n+\tstruct hashfile *f;\n+\tint devnull;\n \n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n@@ -868,6 +871,15 @@ int verify_commit_graph(struct commit_graph *g)\n \tif (verify_commit_graph_error)\n \t\treturn verify_commit_graph_error;\n \n+\tdevnull = open(\"/dev/null\", O_WRONLY);\n+\tf = hashfd(devnull, NULL);\n+\thashwrite(f, g->data, g->data_len - g->hash_len);\n+\tfinalize_hashfile(f, checksum.hash, CSUM_CLOSE);\n+\tif (hashcmp(checksum.hash, g->data + g->data_len - g->hash_len)) {\n+\t\tgraph_report(_(\"the commit-graph file has incorrect checksum and is likely corrupt\"));\n+\t\tverify_commit_graph_error = VERIFY_COMMIT_GRAPH_ERROR_HASH;\n+\t}\n+\n \tfor (i = 0; i < g->num_commits; i++) {\n \t\tstruct commit *graph_commit;\n \n@@ -905,7 +917,7 @@ int verify_commit_graph(struct commit_graph *g)\n \t\tcur_fanout_pos++;\n \t}\n \n-\tif (verify_commit_graph_error)\n+\tif (verify_commit_graph_error & ~VERIFY_COMMIT_GRAPH_ERROR_HASH)\n \t\treturn verify_commit_graph_error;\n \n \tfor (i = 0; i < g->num_commits; i++) {\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex cf67fb391a..2297a44e7d 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -279,6 +279,7 @@ GRAPH_COMMIT_DATA_WIDTH=$(($HASH_LEN + 16))\n GRAPH_OCTOPUS_DATA_OFFSET=$(($GRAPH_COMMIT_DATA_OFFSET + \\\n \t\t\t $GRAPH_COMMIT_DATA_WIDTH \\* $NUM_COMMITS))\n GRAPH_BYTE_OCTOPUS=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4))\n+GRAPH_BYTE_FOOTER=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4 \\* $NUM_OCTOPUS_EDGES))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -393,4 +394,9 @@ test_expect_success 'detect incorrect parent for octopus merge' '\n \t\t\"invalid parent\"\n '\n \n+test_expect_success 'detect invalid checksum hash' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_FOOTER \"\\00\" \\\n+\t\t\"incorrect checksum\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349276","messageId":"20180604165200.29261-15-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 14/21] commit-graph: verify commit date","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:52:45Z","receivedAt":"2018-06-04T16:53:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n commit-graph.c | 6 ++++++\n t/t5318-commit-graph.sh | 6 ++++++\n 2 files changed, 12 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 5faecae2a7..47fdd62e88 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -982,6 +982,12 @@ int verify_commit_graph(struct commit_graph *g)\n \t\t\t\t oid_to_hex(&cur_oid),\n \t\t\t\t graph_commit->generation,\n \t\t\t\t max_generation + 1);\n+\n+\t\tif (graph_commit->date != odb_commit->date)\n+\t\t\tgraph_report(\"commit date for commit %s in commit-graph is %\"PRItime\" != %\"PRItime,\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t graph_commit->date,\n+\t\t\t\t odb_commit->date);\n \t}\n \n \treturn verify_commit_graph_error;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex a6ea1341dc..6a873bfda8 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -273,6 +273,7 @@ GRAPH_BYTE_COMMIT_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN))\n GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4))\n GRAPH_BYTE_COMMIT_WRONG_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3))\n GRAPH_BYTE_COMMIT_GENERATION=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 11))\n+GRAPH_BYTE_COMMIT_DATE=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 12))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -377,4 +378,9 @@ test_expect_success 'detect incorrect generation number' '\n \t\t\"non-zero generation number\"\n '\n \n+test_expect_success 'detect incorrect commit date' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_DATE \"\\01\" \\\n+\t\t\"commit date\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349277","messageId":"20180604165200.29261-16-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 15/21] commit-graph: test for corrupted octopus edge","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:52:48Z","receivedAt":"2018-06-04T16:53:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file has an extra chunk to store the parent int-ids for\nparents beyond the first parent for octopus merges. Our test repo has a\nsingle octopus merge that we can manipulate to demonstrate the 'verify'\nsubcommand detects incorrect values in that chunk.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5318-commit-graph.sh | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 6a873bfda8..cf67fb391a 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -248,6 +248,7 @@ test_expect_success 'git commit-graph verify' '\n '\n \n NUM_COMMITS=9\n+NUM_OCTOPUS_EDGES=2\n HASH_LEN=20\n GRAPH_BYTE_VERSION=4\n GRAPH_BYTE_HASH=5\n@@ -274,6 +275,10 @@ GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4))\n GRAPH_BYTE_COMMIT_WRONG_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3))\n GRAPH_BYTE_COMMIT_GENERATION=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 11))\n GRAPH_BYTE_COMMIT_DATE=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 12))\n+GRAPH_COMMIT_DATA_WIDTH=$(($HASH_LEN + 16))\n+GRAPH_OCTOPUS_DATA_OFFSET=$(($GRAPH_COMMIT_DATA_OFFSET + \\\n+\t\t\t $GRAPH_COMMIT_DATA_WIDTH \\* $NUM_COMMITS))\n+GRAPH_BYTE_OCTOPUS=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -383,4 +388,9 @@ test_expect_success 'detect incorrect commit date' '\n \t\t\"commit date\"\n '\n \n+test_expect_success 'detect incorrect parent for octopus merge' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OCTOPUS \"\\01\" \\\n+\t\t\"invalid parent\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349278","messageId":"20180604165200.29261-5-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v4 04/21] commit: force commit to parse from object database","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-04T16:52:21Z","receivedAt":"2018-06-04T16:53:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In anticipation of verifying commit-graph file contents against the\nobject database, create parse_commit_internal() to allow side-stepping\nthe commit-graph file and parse directly from the object database.\n\nDue to the use of generation numbers, this method should not be called\nunless the intention is explicit in avoiding commits from the\ncommit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 9 +++++++--\n commit.h | 1 +\n 2 files changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex 1d28677dfb..6eaed0174c 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -392,7 +392,7 @@ int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long s\n \treturn 0;\n }\n \n-int parse_commit_gently(struct commit *item, int quiet_on_missing)\n+int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph)\n {\n \tenum object_type type;\n \tvoid *buffer;\n@@ -403,7 +403,7 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \t\treturn -1;\n \tif (item->object.parsed)\n \t\treturn 0;\n-\tif (parse_commit_in_graph(item))\n+\tif (use_commit_graph && parse_commit_in_graph(item))\n \t\treturn 0;\n \tbuffer = read_sha1_file(item->object.oid.hash, &type, &size);\n \tif (!buffer)\n@@ -424,6 +424,11 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \treturn ret;\n }\n \n+int parse_commit_gently(struct commit *item, int quiet_on_missing)\n+{\n+\treturn parse_commit_internal(item, quiet_on_missing, 1);\n+}\n+\n void parse_commit_or_die(struct commit *item)\n {\n \tif (parse_commit(item))\ndiff --git a/commit.h b/commit.h\nindex b5afde1ae9..5fde74fcd7 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -73,6 +73,7 @@ struct commit *lookup_commit_reference_by_name(const char *name);\n struct commit *lookup_commit_or_die(const struct object_id *oid, const char *ref_name);\n \n int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph);\n+int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph);\n int parse_commit_gently(struct commit *item, int quiet_on_missing);\n static inline int parse_commit(struct commit *item)\n {\n-- \n2.18.0.rc1\n\n"},{"id":"349279","messageId":"5d8dd51f-db7c-9da6-c21b-6e537251eddc@gmail.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"Re: [PATCH v4 00/21] Integrate commit-graph into 'fsck' and 'gc'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-04T17:02:55Z","receivedAt":"2018-06-04T17:03:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Sorry I forgot to --in-reply-to the previous version [1]\n\n[1] \nhttps://public-inbox.org/git/20180524162504.158394-1-dstolee@microsoft.com/T/#u\n\nOn 6/4/2018 12:52 PM, Derrick Stolee wrote:\n> Thanks for the feedback on v3. There were several small cleanups, but\n> perhaps the biggest change is the addition of \"commit-graph: use\n> string-list API for input\" which makes \"commit-graph: add '--reachable'\n> option\" much simpler.\n>\n> The inter-diff is still reasonably large, but I'll send it in a\n> follow-up PR.\n\ns/PR/message\n\nHere is that diff:\n\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 9a3abd87e7..d2eb3c8e9b 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -900,7 +900,8 @@ the `GIT_NOTES_REF` environment variable.  See \nlinkgit:git-notes[1].\n\n  core.commitGraph::\n         Enable git commit graph feature. Allows reading from the\n-       commit-graph file.\n+       commit-graph file. See `gc.commitGraph` for automatically\n+       maintaining the file.\n\n  core.sparseCheckout::\n         Enable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n@@ -1554,10 +1555,11 @@ gc.autoDetach::\n         if the system supports it. Default is true.\n\n  gc.commitGraph::\n-       If true, then gc will rewrite the commit-graph file after any\n-       change to the object database. If '--auto' is used, then the\n-       commit-graph will not be updated unless the threshold is met.\n-       See linkgit:git-commit-graph[1] for details.\n+       If true, then gc will rewrite the commit-graph file when\n+       linkgit:git-gc[1] is run. When using linkgit:git-gc[1]\n+       '--auto' the commit-graph will be updated if housekeeping is\n+       required. Default is false. See linkgit:git-commit-graph[1]\n+       for details.\n\n  gc.logExpiry::\n         If the file gc.log exists, then `git gc --auto` won't run\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 17dd654a59..a6526b3592 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -119,9 +119,9 @@ The optional configuration variable `gc.packRefs` \ndetermines if\n  it within all non-bare repos or it can be set to a boolean value.\n  This defaults to true.\n\n-The optional configuration variable 'gc.commitGraph' determines if\n-'git gc' runs 'git commit-graph write'. This can be set to a boolean\n-value. This defaults to false.\n+The optional configuration variable `gc.commitGraph` determines if\n+'git gc' should run 'git commit-graph write'. This can be set to a\n+boolean value. This defaults to false.\n\n  The optional configuration variable `gc.aggressiveWindow` controls how\n  much time is spent optimizing the delta compression of the objects in\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 20ce6437ae..76423b3fa5 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -39,7 +39,7 @@ static struct opts_commit_graph {\n\n  static int graph_verify(int argc, const char **argv)\n  {\n-       struct commit_graph *graph = 0;\n+       struct commit_graph *graph = NULL;\n         char *graph_name;\n\n         static struct option builtin_commit_graph_verify_options[] = {\n@@ -119,13 +119,9 @@ static int graph_read(int argc, const char **argv)\n\n  static int graph_write(int argc, const char **argv)\n  {\n-       const char **pack_indexes = NULL;\n-       int packs_nr = 0;\n-       const char **commit_hex = NULL;\n-       int commits_nr = 0;\n-       const char **lines = NULL;\n-       int lines_nr = 0;\n-       int lines_alloc = 0;\n+       struct string_list *pack_indexes = NULL;\n+       struct string_list *commit_hex = NULL;\n+       struct string_list lines;\n\n         static struct option builtin_commit_graph_write_options[] = {\n                 OPT_STRING(0, \"object-dir\", &opts.obj_dir,\n@@ -158,32 +154,23 @@ static int graph_write(int argc, const char **argv)\n\n         if (opts.stdin_packs || opts.stdin_commits) {\n                 struct strbuf buf = STRBUF_INIT;\n-               lines_nr = 0;\n-               lines_alloc = 128;\n-               ALLOC_ARRAY(lines, lines_alloc);\n-\n-               while (strbuf_getline(&buf, stdin) != EOF) {\n-                       ALLOC_GROW(lines, lines_nr + 1, lines_alloc);\n-                       lines[lines_nr++] = strbuf_detach(&buf, NULL);\n-               }\n-\n-               if (opts.stdin_packs) {\n-                       pack_indexes = lines;\n-                       packs_nr = lines_nr;\n-               }\n-               if (opts.stdin_commits) {\n-                       commit_hex = lines;\n-                       commits_nr = lines_nr;\n-               }\n+               string_list_init(&lines, 0);\n+\n+               while (strbuf_getline(&buf, stdin) != EOF)\n+                       string_list_append(&lines, strbuf_detach(&buf, \nNULL));\n+\n+               if (opts.stdin_packs)\n+                       pack_indexes = &lines;\n+               if (opts.stdin_commits)\n+                       commit_hex = &lines;\n         }\n\n         write_commit_graph(opts.obj_dir,\n                            pack_indexes,\n-                          packs_nr,\n                            commit_hex,\n-                          commits_nr,\n                            opts.append);\n\n+       string_list_clear(&lines, 0);\n         return 0;\n  }\n\n@@ -207,10 +194,10 @@ int cmd_commit_graph(int argc, const char **argv, \nconst char *prefix)\n                              PARSE_OPT_STOP_AT_NON_OPTION);\n\n         if (argc > 0) {\n-               if (!strcmp(argv[0], \"verify\"))\n-                       return graph_verify(argc, argv);\n                 if (!strcmp(argv[0], \"read\"))\n                         return graph_read(argc, argv);\n+               if (!strcmp(argv[0], \"verify\"))\n+                       return graph_verify(argc, argv);\n                 if (!strcmp(argv[0], \"write\"))\n                         return graph_write(argc, argv);\n         }\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 057d734926..946bcfa98c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1,5 +1,6 @@\n  #include \"cache.h\"\n  #include \"config.h\"\n+#include \"dir.h\"\n  #include \"git-compat-util.h\"\n  #include \"lockfile.h\"\n  #include \"pack.h\"\n@@ -652,42 +653,26 @@ static void compute_generation_numbers(struct \npacked_commit_list* commits)\n         }\n  }\n\n-struct hex_list {\n-       char **hex_strs;\n-       int hex_nr;\n-       int hex_alloc;\n-};\n-\n  static int add_ref_to_list(const char *refname,\n                            const struct object_id *oid,\n                            int flags, void *cb_data)\n  {\n-       struct hex_list *list = (struct hex_list*)cb_data;\n-\n-       ALLOC_GROW(list->hex_strs, list->hex_nr + 1, list->hex_alloc);\n-       list->hex_strs[list->hex_nr] = xcalloc(GIT_MAX_HEXSZ + 1, 1);\n-       strcpy(list->hex_strs[list->hex_nr], oid_to_hex(oid));\n-       list->hex_nr++;\n+       struct string_list *list = (struct string_list*)cb_data;\n+       string_list_append(list, oid_to_hex(oid));\n         return 0;\n  }\n\n  void write_commit_graph_reachable(const char *obj_dir, int append)\n  {\n-       struct hex_list list;\n-       list.hex_nr = 0;\n-       list.hex_alloc = 128;\n-       ALLOC_ARRAY(list.hex_strs, list.hex_alloc);\n-\n+       struct string_list list;\n+       string_list_init(&list, 1);\n         for_each_ref(add_ref_to_list, &list);\n-\n-       write_commit_graph(obj_dir, NULL, 0, (const char \n**)list.hex_strs, list.hex_nr, append);\n+       write_commit_graph(obj_dir, NULL, &list, append);\n  }\n\n  void write_commit_graph(const char *obj_dir,\n-                       const char **pack_indexes,\n-                       int nr_packs,\n-                       const char **commit_hex,\n-                       int nr_commits,\n+                       struct string_list *pack_indexes,\n+                       struct string_list *commit_hex,\n                         int append)\n  {\n         struct packed_oid_list oids;\n@@ -695,7 +680,6 @@ void write_commit_graph(const char *obj_dir,\n         struct hashfile *f;\n         uint32_t i, count_distinct = 0;\n         char *graph_name;\n-       int fd;\n         struct lock_file lk = LOCK_INIT;\n         uint32_t chunk_ids[5];\n         uint64_t chunk_offsets[5];\n@@ -729,10 +713,10 @@ void write_commit_graph(const char *obj_dir,\n                 int dirlen;\n                 strbuf_addf(&packname, \"%s/pack/\", obj_dir);\n                 dirlen = packname.len;\n-               for (i = 0; i < nr_packs; i++) {\n+               for (i = 0; i < pack_indexes->nr; i++) {\n                         struct packed_git *p;\n                         strbuf_setlen(&packname, dirlen);\n-                       strbuf_addstr(&packname, pack_indexes[i]);\n+                       strbuf_addstr(&packname, \npack_indexes->items[i].string);\n                         p = add_packed_git(packname.buf, packname.len, 1);\n                         if (!p)\n                                 die(\"error adding pack %s\", packname.buf);\n@@ -745,12 +729,13 @@ void write_commit_graph(const char *obj_dir,\n         }\n\n         if (commit_hex) {\n-               for (i = 0; i < nr_commits; i++) {\n+               for (i = 0; i < commit_hex->nr; i++) {\n                         const char *end;\n                         struct object_id oid;\n                         struct commit *result;\n\n-                       if (commit_hex[i] && \nparse_oid_hex(commit_hex[i], &oid, &end))\n+                       if (commit_hex->items[i].string &&\n+ parse_oid_hex(commit_hex->items[i].string, &oid, &end))\n                                 continue;\n\n                         result = lookup_commit_reference_gently(&oid, 1);\n@@ -809,23 +794,11 @@ void write_commit_graph(const char *obj_dir,\n         compute_generation_numbers(&commits);\n\n         graph_name = get_commit_graph_filename(obj_dir);\n-       fd = hold_lock_file_for_update(&lk, graph_name, 0);\n-\n-       if (fd < 0) {\n-               struct strbuf folder = STRBUF_INIT;\n-               strbuf_addstr(&folder, graph_name);\n-               strbuf_setlen(&folder, strrchr(folder.buf, '/') - \nfolder.buf);\n-\n-               if (mkdir(folder.buf, 0777) < 0)\n-                       die_errno(_(\"cannot mkdir %s\"), folder.buf);\n-               strbuf_release(&folder);\n-\n-               fd = hold_lock_file_for_update(&lk, graph_name, \nLOCK_DIE_ON_ERROR);\n-\n-               if (fd < 0)\n-                       die_errno(\"unable to create '%s'\", graph_name);\n-       }\n+       if (safe_create_leading_directories(graph_name))\n+               die_errno(_(\"unable to create leading directories of %s\"),\n+                         graph_name);\n\n+       hold_lock_file_for_update(&lk, graph_name, LOCK_DIE_ON_ERROR);\n         f = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n\n         hashwrite_be32(f, GRAPH_SIGNATURE);\n@@ -879,21 +852,22 @@ static int verify_commit_graph_error;\n  static void graph_report(const char *fmt, ...)\n  {\n         va_list ap;\n-       struct strbuf sb = STRBUF_INIT;\n         verify_commit_graph_error = 1;\n\n         va_start(ap, fmt);\n-       strbuf_vaddf(&sb, fmt, ap);\n-\n-       fprintf(stderr, \"%s\\n\", sb.buf);\n-       strbuf_release(&sb);\n+       vfprintf(stderr, fmt, ap);\n+       fprintf(stderr, \"\\n\");\n         va_end(ap);\n  }\n\n+#define GENERATION_ZERO_EXISTS 1\n+#define GENERATION_NUMBER_EXISTS 2\n+\n  int verify_commit_graph(struct commit_graph *g)\n  {\n         uint32_t i, cur_fanout_pos = 0;\n         struct object_id prev_oid, cur_oid, checksum;\n+       int generation_zero = 0;\n         struct hashfile *f;\n         int devnull;\n\n@@ -1012,6 +986,18 @@ int verify_commit_graph(struct commit_graph *g)\n                         graph_report(\"commit-graph parent list for \ncommit %s terminates early\",\n                                      oid_to_hex(&cur_oid));\n\n+               if (!graph_commit->generation) {\n+                       if (generation_zero == GENERATION_NUMBER_EXISTS)\n+                               graph_report(\"commit-graph has \ngeneration number zero for commit %s, but non-zero elsewhere\",\n+ oid_to_hex(&cur_oid));\n+                       generation_zero = GENERATION_ZERO_EXISTS;\n+               } else if (generation_zero == GENERATION_ZERO_EXISTS)\n+                       graph_report(\"commit-graph has non-zero \ngeneration number for commit %s, but zero elsewhere\",\n+                                    oid_to_hex(&cur_oid));\n+\n+               if (generation_zero == GENERATION_ZERO_EXISTS)\n+                       continue;\n+\n                 /*\n                  * If one of our parents has generation \nGENERATION_NUMBER_MAX, then\n                  * our generation is also GENERATION_NUMBER_MAX. \nDecrement to avoid\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 9a06a5f188..ee20f5e280 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -2,6 +2,7 @@\n  #define COMMIT_GRAPH_H\n\n  #include \"git-compat-util.h\"\n+#include \"string-list.h\"\n\n  char *get_commit_graph_filename(const char *obj_dir);\n\n@@ -48,10 +49,8 @@ struct commit_graph *load_commit_graph_one(const char \n*graph_file);\n\n  void write_commit_graph_reachable(const char *obj_dir, int append);\n  void write_commit_graph(const char *obj_dir,\n-                       const char **pack_indexes,\n-                       int nr_packs,\n-                       const char **commit_hex,\n-                       int nr_commits,\n+                       struct string_list *pack_indexes,\n+                       struct string_list *commit_hex,\n                         int append);\n\n  int verify_commit_graph(struct commit_graph *g);\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex d20b17586f..b24e8b6689 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -33,8 +33,8 @@ test_expect_success 'create commits and repack' '\n  '\n\n  graph_git_two_modes() {\n-       git -c core.graph=true $1 >output\n-       git -c core.graph=false $1 >expect\n+       git -c core.commitGraph=true $1 >output\n+       git -c core.commitGraph=false $1 >expect\n         test_cmp output expect\n  }\n\n@@ -245,7 +245,7 @@ test_expect_success 'perform fast-forward merge in \nfull repo' '\n         test_cmp expect output\n  '\n\n-test_expect_success 'check that gc clears commit-graph' '\n+test_expect_success 'check that gc computes commit-graph' '\n         cd \"$TRASH_DIRECTORY/full\" &&\n         git commit --allow-empty -m \"blank\" &&\n         git commit-graph write --reachable &&\n@@ -281,29 +281,29 @@ GRAPH_CHUNK_LOOKUP_OFFSET=8\n  GRAPH_CHUNK_LOOKUP_WIDTH=12\n  GRAPH_CHUNK_LOOKUP_ROWS=5\n  GRAPH_BYTE_OID_FANOUT_ID=$GRAPH_CHUNK_LOOKUP_OFFSET\n-GRAPH_BYTE_OID_LOOKUP_ID=`expr $GRAPH_CHUNK_LOOKUP_OFFSET + \\\n-                             1 \\* $GRAPH_CHUNK_LOOKUP_WIDTH`\n-GRAPH_BYTE_COMMIT_DATA_ID=`expr $GRAPH_CHUNK_LOOKUP_OFFSET + \\\n-                               2 \\* $GRAPH_CHUNK_LOOKUP_WIDTH`\n-GRAPH_FANOUT_OFFSET=`expr $GRAPH_CHUNK_LOOKUP_OFFSET + \\\n-                         $GRAPH_CHUNK_LOOKUP_WIDTH \\* \n$GRAPH_CHUNK_LOOKUP_ROWS`\n-GRAPH_BYTE_FANOUT1=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 4`\n-GRAPH_BYTE_FANOUT2=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 255`\n-GRAPH_OID_LOOKUP_OFFSET=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 256`\n-GRAPH_BYTE_OID_LOOKUP_ORDER=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \n\\* 8`\n-GRAPH_BYTE_OID_LOOKUP_MISSING=`expr $GRAPH_OID_LOOKUP_OFFSET + \n$HASH_LEN \\* 4 + 10`\n-GRAPH_COMMIT_DATA_OFFSET=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* \n$NUM_COMMITS`\n+GRAPH_BYTE_OID_LOOKUP_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n+                           1 \\* $GRAPH_CHUNK_LOOKUP_WIDTH))\n+GRAPH_BYTE_COMMIT_DATA_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n+                            2 \\* $GRAPH_CHUNK_LOOKUP_WIDTH))\n+GRAPH_FANOUT_OFFSET=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n+                      $GRAPH_CHUNK_LOOKUP_WIDTH \\* \n$GRAPH_CHUNK_LOOKUP_ROWS))\n+GRAPH_BYTE_FANOUT1=$(($GRAPH_FANOUT_OFFSET + 4 \\* 4))\n+GRAPH_BYTE_FANOUT2=$(($GRAPH_FANOUT_OFFSET + 4 \\* 255))\n+GRAPH_OID_LOOKUP_OFFSET=$(($GRAPH_FANOUT_OFFSET + 4 \\* 256))\n+GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8))\n+GRAPH_BYTE_OID_LOOKUP_MISSING=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \n\\* 4 + 10))\n+GRAPH_COMMIT_DATA_OFFSET=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* \n$NUM_COMMITS))\n  GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n-GRAPH_BYTE_COMMIT_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN`\n-GRAPH_BYTE_COMMIT_EXTRA_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + \n$HASH_LEN + 4`\n-GRAPH_BYTE_COMMIT_WRONG_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + \n$HASH_LEN + 3`\n-GRAPH_BYTE_COMMIT_GENERATION=`expr $GRAPH_COMMIT_DATA_OFFSET + \n$HASH_LEN + 8`\n-GRAPH_BYTE_COMMIT_DATE=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 12`\n-GRAPH_COMMIT_DATA_WIDTH=`expr $HASH_LEN + 16`\n-GRAPH_OCTOPUS_DATA_OFFSET=`expr $GRAPH_COMMIT_DATA_OFFSET + \\\n-                               $GRAPH_COMMIT_DATA_WIDTH \\* $NUM_COMMITS`\n-GRAPH_BYTE_OCTOPUS=`expr $GRAPH_OCTOPUS_DATA_OFFSET + 4`\n-GRAPH_BYTE_FOOTER=`expr $GRAPH_OCTOPUS_DATA_OFFSET + 4 \\* \n$NUM_OCTOPUS_EDGES`\n+GRAPH_BYTE_COMMIT_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN))\n+GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN \n+ 4))\n+GRAPH_BYTE_COMMIT_WRONG_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN \n+ 3))\n+GRAPH_BYTE_COMMIT_GENERATION=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + \n11))\n+GRAPH_BYTE_COMMIT_DATE=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 12))\n+GRAPH_COMMIT_DATA_WIDTH=$(($HASH_LEN + 16))\n+GRAPH_OCTOPUS_DATA_OFFSET=$(($GRAPH_COMMIT_DATA_OFFSET + \\\n+                            $GRAPH_COMMIT_DATA_WIDTH \\* $NUM_COMMITS))\n+GRAPH_BYTE_OCTOPUS=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4))\n+GRAPH_BYTE_FOOTER=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4 \\* $NUM_OCTOPUS_EDGES))\n\n  # usage: corrupt_graph_and_verify \n  # Manipulates the commit-graph file at the position\n@@ -320,7 +320,7 @@ corrupt_graph_and_verify() {\n         printf \"$data\" | dd of=\"$objdir/info/commit-graph\" bs=1 \nseek=\"$pos\" conv=notrunc &&\n         test_must_fail git commit-graph verify 2>test_err &&\n         grep -v \"^+\" test_err >err\n-       grep \"$grepstr\" err\n+       test_i18ngrep \"$grepstr\" err\n  }\n\n  test_expect_success 'detect bad signature' '\n@@ -338,9 +338,9 @@ test_expect_success 'detect bad hash version' '\n                 \"hash version\"\n  '\n\n-test_expect_success 'detect bad chunk count' '\n+test_expect_success 'detect low chunk count' '\n         corrupt_graph_and_verify $GRAPH_BYTE_CHUNK_COUNT \"\\02\" \\\n-               \"missing the Commit Data chunk\"\n+               \"missing the .* chunk\"\n  '\n\n  test_expect_success 'detect missing OID fanout chunk' '\n@@ -363,7 +363,7 @@ test_expect_success 'detect incorrect fanout' '\n                 \"fanout value\"\n  '\n\n-test_expect_success 'detect incorrect fanout' '\n+test_expect_success 'detect incorrect fanout final value' '\n         corrupt_graph_and_verify $GRAPH_BYTE_FANOUT2 \"\\01\" \\\n                 \"fanout value\"\n  '\n@@ -393,14 +393,19 @@ test_expect_success 'detect extra parent int-id' '\n                 \"is too long\"\n  '\n\n-test_expect_success 'detect incorrect tree OID' '\n+test_expect_success 'detect wrong parent' '\n         corrupt_graph_and_verify $GRAPH_BYTE_COMMIT_WRONG_PARENT \"\\01\" \\\n                 \"commit-graph parent for\"\n  '\n\n+test_expect_success 'detect incorrect generation number' '\n+       corrupt_graph_and_verify $GRAPH_BYTE_COMMIT_GENERATION \"\\070\" \\\n+               \"generation for commit\"\n+'\n+\n  test_expect_success 'detect incorrect generation number' '\n         corrupt_graph_and_verify $GRAPH_BYTE_COMMIT_GENERATION \"\\01\" \\\n-               \"generation\"\n+               \"non-zero generation number\"\n  '\n\n  test_expect_success 'detect incorrect commit date' '\n\n"},{"id":"349349","messageId":"CACBZZX7N2xy0iX-zqZG2-g-44D1+X1bK38oqMWrWX0ZE1v9C-A@mail.gmail.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"Re: [PATCH v4 00/21] Integrate commit-graph into 'fsck' and 'gc'","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-06-05T14:51:46Z","receivedAt":"2018-06-05T14:52:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Mon, Jun 4, 2018 at 6:52 PM, Derrick Stolee wrote:\n> Thanks for the feedback on v3. There were several small cleanups, but\n> perhaps the biggest change is the addition of \"commit-graph: use\n> string-list API for input\" which makes \"commit-graph: add '--reachable'\n> option\" much simpler.\n\nDo you have a version of this pushed somewhere? Your fsck/upstream\nbranch conflicts with e.g. long-since-merged nd/repack-keep-pack.\n"},{"id":"349364","messageId":"88258f58-c28f-0f74-a378-003704c41117@gmail.com","threadId":"48647","inReplyTo":"CACBZZX7N2xy0iX-zqZG2-g-44D1+X1bK38oqMWrWX0ZE1v9C-A@mail.gmail.com","subject":"Re: [PATCH v4 00/21] Integrate commit-graph into 'fsck' and 'gc'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-05T16:37:44Z","receivedAt":"2018-06-05T16:37:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/5/2018 10:51 AM, Ævar Arnfjörð Bjarmason wrote:\n> On Mon, Jun 4, 2018 at 6:52 PM, Derrick Stolee wrote:\n>> Thanks for the feedback on v3. There were several small cleanups, but\n>> perhaps the biggest change is the addition of \"commit-graph: use\n>> string-list API for input\" which makes \"commit-graph: add '--reachable'\n>> option\" much simpler.\n> Do you have a version of this pushed somewhere? Your fsck/upstream\n> branch conflicts with e.g. long-since-merged nd/repack-keep-pack.\n\nSorry I forgot to push. It is now available on GitHub [1]. I based my \ncommits on 'next' including the core.commitGraph fix for \nt5318-commit-graph.sh I already sent [2].\n\n[1] https://github.com/derrickstolee/git/tree/fsck/upstream\n     fsck/upstream branch matches this patch series\n\n[2] \nhttps://public-inbox.org/git/20180604123906.136417-1-dstolee@microsoft.com/\n     [PATCH] t5318-commit-graph.sh: use core.commitGraph\n"},{"id":"349470","messageId":"87k1rcyxxj.fsf@evledraar.gmail.com","threadId":"48647","inReplyTo":"20180604165200.29261-18-dstolee@microsoft.com","subject":"Re: [PATCH v4 17/21] fsck: verify commit-graph","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-06-06T11:08:40Z","receivedAt":"2018-06-06T11:08:46Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jun 04 2018, Derrick Stolee wrote:\n\n> +\t\tprepare_alt_odb();\n> +\t\tfor (alt = alt_odb_list; alt; alt = alt->next) {\n> +\t\t\tverify_argv[2] = \"--object-dir\";\n> +\t\t\tverify_argv[3] = alt->path;\n> +\t\t\tif (run_command(&commit_graph_verify))\n> +\t\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n> +\t\t}\n> +\t}\n> +\n\nThis doesn't compile under clang on master. It needs to account for\n0b20903405 (\"sha1_file: add repository argument to prepare_alt_odb\",\n2018-03-23).\n\n builtin/fsck.c:837:19: error: too few arguments to function call, single argument 'r' was not specified\n prepare_alt_odb();\n ~~~~~~~~~~~~~~~ ^\n\nDitto this error due to a missing resolution with 031dc927f4\n(\"object-store: move alt_odb_list and alt_odb_tail to object store\",\n2018-03-23):\n\n builtin/fsck.c:838:14: error: use of undeclared identifier 'alt_odb_list'\n for (alt = alt_odb_list; alt; alt = alt->next) {\n"},{"id":"349471","messageId":"87in6wyxsj.fsf@evledraar.gmail.com","threadId":"48647","inReplyTo":"20180604165200.29261-21-dstolee@microsoft.com","subject":"Re: [PATCH v4 20/21] gc: automatically write commit-graph files","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-06-06T11:11:40Z","receivedAt":"2018-06-06T11:11:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jun 04 2018, Derrick Stolee wrote:\n\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index 11f027194e..d2eb3c8e9b 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -900,7 +900,8 @@ the `GIT_NOTES_REF` environment variable. See linkgit:git-notes[1].\n>\n> core.commitGraph::\n> \tEnable git commit graph feature. Allows reading from the\n> -\tcommit-graph file.\n> +\tcommit-graph file. See `gc.commitGraph` for automatically\n> +\tmaintaining the file.\n>\n> core.sparseCheckout::\n> \tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n> @@ -1553,6 +1554,13 @@ gc.autoDetach::\n> \tMake `git gc --auto` return immediately and run in background\n> \tif the system supports it. Default is true.\n>\n> +gc.commitGraph::\n> +\tIf true, then gc will rewrite the commit-graph file when\n> +\tlinkgit:git-gc[1] is run. When using linkgit:git-gc[1]\n> +\t'--auto' the commit-graph will be updated if housekeeping is\n> +\trequired. Default is false. See linkgit:git-commit-graph[1]\n> +\tfor details.\n> +\n> gc.logExpiry::\n> \tIf the file gc.log exists, then `git gc --auto` won't run\n> \tunless that file is more than 'gc.logExpiry' old. Default is\n\nRe my last E-Mail in\nhttps://public-inbox.org/git/87k1rcyxxj.fsf@evledraar.gmail.com/\n88258f58-c28f-0f74-a378-003704c41117@gmail.com and your reply in\nhttps://public-inbox.org/git/88258f58-c28f-0f74-a378-003704c41117@gmail.com/\nit seems what you've submitted here is the old version. E.g. this hunk\nfails to apply to master because of the conflict with\ngc.bigPackThreshold, which was merged in 30b015bffe (\"Merge branch\n'nd/repack-keep-pack'\", 2018-05-23) .\n"},{"id":"349473","messageId":"73c4edcb-dd13-04e9-b4b6-0f415e5e8768@gmail.com","threadId":"48647","inReplyTo":"87k1rcyxxj.fsf@evledraar.gmail.com","subject":"Re: [PATCH v4 17/21] fsck: verify commit-graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-06T11:31:42Z","receivedAt":"2018-06-06T11:31:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/6/2018 7:08 AM, Ævar Arnfjörð Bjarmason wrote:\n> On Mon, Jun 04 2018, Derrick Stolee wrote:\n>\n>> +\t\tprepare_alt_odb();\n>> +\t\tfor (alt = alt_odb_list; alt; alt = alt->next) {\n>> +\t\t\tverify_argv[2] = \"--object-dir\";\n>> +\t\t\tverify_argv[3] = alt->path;\n>> +\t\t\tif (run_command(&commit_graph_verify))\n>> +\t\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n>> +\t\t}\n>> +\t}\n>> +\n> This doesn't compile under clang on master. It needs to account for\n> 0b20903405 (\"sha1_file: add repository argument to prepare_alt_odb\",\n> 2018-03-23).\n>\n> builtin/fsck.c:837:19: error: too few arguments to function call, single argument 'r' was not specified\n> prepare_alt_odb();\n> ~~~~~~~~~~~~~~~ ^\n>\n> Ditto this error due to a missing resolution with 031dc927f4\n> (\"object-store: move alt_odb_list and alt_odb_tail to object store\",\n> 2018-03-23):\n>\n> builtin/fsck.c:838:14: error: use of undeclared identifier 'alt_odb_list'\n> for (alt = alt_odb_list; alt; alt = alt->next) {\n\nThanks, Ævar. I forgot to rebase onto 'next'. Doing so now and will send \nv5 shortly.\n\n-Stolee\n"},{"id":"349474","messageId":"20180606113611.87822-1-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180604165200.29261-1-dstolee@microsoft.com","subject":"[PATCH v5 00/21] Integrate commit-graph into 'fsck' and 'gc'","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:20Z","receivedAt":"2018-06-06T11:36:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Thanks, Ævar, for pointing out that I forgot to rebase onto 'next'.\n\nThere were a few collisions with the object-store refactoring. Junio\neven pointed them out, so I'm sorry to forget that. I also did a test\nmerge with 'pu' and it seems the only conflicts are with the earlier\nversion of this patch.\n\nThanks,\n-Stolee\n\nDerrick Stolee (21):\n commit-graph: UNLEAK before die()\n commit-graph: fix GRAPH_MIN_SIZE\n commit-graph: parse commit from chosen graph\n commit: force commit to parse from object database\n commit-graph: load a root tree from specific graph\n commit-graph: add 'verify' subcommand\n commit-graph: verify catches corrupt signature\n commit-graph: verify required chunks are present\n commit-graph: verify corrupt OID fanout and lookup\n commit-graph: verify objects exist\n commit-graph: verify root tree OIDs\n commit-graph: verify parent list\n commit-graph: verify generation number\n commit-graph: verify commit date\n commit-graph: test for corrupted octopus edge\n commit-graph: verify contents match checksum\n fsck: verify commit-graph\n commit-graph: use string-list API for input\n commit-graph: add '--reachable' option\n gc: automatically write commit-graph files\n commit-graph: update design document\n\n Documentation/config.txt | 10 +-\n Documentation/git-commit-graph.txt | 14 +-\n Documentation/git-fsck.txt | 3 +\n Documentation/git-gc.txt | 4 +\n Documentation/technical/commit-graph.txt | 22 --\n builtin/commit-graph.c | 98 ++++++---\n builtin/fsck.c | 21 ++\n builtin/gc.c | 6 +\n commit-graph.c | 248 +++++++++++++++++++++--\n commit-graph.h | 10 +-\n commit.c | 9 +-\n commit.h | 1 +\n t/t5318-commit-graph.sh | 201 ++++++++++++++++++\n 13 files changed, 569 insertions(+), 78 deletions(-)\n\n-- \n2.18.0.rc1\n\n"},{"id":"349475","messageId":"20180606113611.87822-6-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 05/21] commit-graph: load a root tree from specific graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:26Z","receivedAt":"2018-06-06T11:36:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When lazy-loading a tree for a commit, it will be important to select\nthe tree from a specific struct commit_graph. Create a new method that\nspecifies the commit-graph file and use that in\nget_commit_tree_in_graph().\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 12 +++++++++---\n 1 file changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex e77b19971d..9e228d3bb5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -362,14 +362,20 @@ static struct tree *load_tree_for_commit(struct commit_graph *g, struct commit *\n \treturn c->maybe_tree;\n }\n \n-struct tree *get_commit_tree_in_graph(const struct commit *c)\n+static struct tree *get_commit_tree_in_graph_one(struct commit_graph *g,\n+\t\t\t\t\t\t const struct commit *c)\n {\n \tif (c->maybe_tree)\n \t\treturn c->maybe_tree;\n \tif (c->graph_pos == COMMIT_NOT_FROM_GRAPH)\n-\t\tBUG(\"get_commit_tree_in_graph called from non-commit-graph commit\");\n+\t\tBUG(\"get_commit_tree_in_graph_one called from non-commit-graph commit\");\n+\n+\treturn load_tree_for_commit(g, (struct commit *)c);\n+}\n \n-\treturn load_tree_for_commit(commit_graph, (struct commit *)c);\n+struct tree *get_commit_tree_in_graph(const struct commit *c)\n+{\n+\treturn get_commit_tree_in_graph_one(commit_graph, c);\n }\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n-- \n2.18.0.rc1\n\n"},{"id":"349476","messageId":"20180606113611.87822-18-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 17/21] fsck: verify commit-graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:37Z","receivedAt":"2018-06-06T11:36:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"If core.commitGraph is true, verify the contents of the commit-graph\nduring 'git fsck' using the 'git commit-graph verify' subcommand. Run\nthis check on all alternates, as well.\n\nWe use a new process for two reasons:\n\n1. The subcommand decouples the details of loading and verifying a\n commit-graph file from the other fsck details.\n\n2. The commit-graph verification requires the commits to be loaded\n in a specific order to guarantee we parse from the commit-graph\n file for some objects and from the object database for others.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-fsck.txt | 3 +++\n builtin/fsck.c | 21 +++++++++++++++++++++\n t/t5318-commit-graph.sh | 8 ++++++++\n 3 files changed, 32 insertions(+)\n\ndiff --git a/Documentation/git-fsck.txt b/Documentation/git-fsck.txt\nindex b9f060e3b2..ab9a93fb9b 100644\n--- a/Documentation/git-fsck.txt\n+++ b/Documentation/git-fsck.txt\n@@ -110,6 +110,9 @@ Any corrupt objects you will have to find in backups or other archives\n (i.e., you can just remove them and do an 'rsync' with some other site in\n the hopes that somebody else has the object you have corrupted).\n \n+If core.commitGraph is true, the commit-graph file will also be inspected\n+using 'git commit-graph verify'. See linkgit:git-commit-graph[1].\n+\n Extracted Diagnostics\n ---------------------\n \ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 3ad4f160f9..9fb2edc69f 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -18,6 +18,7 @@\n #include \"decorate.h\"\n #include \"packfile.h\"\n #include \"object-store.h\"\n+#include \"run-command.h\"\n \n #define REACHABLE 0x0001\n #define SEEN 0x0002\n@@ -47,6 +48,7 @@ static int name_objects;\n #define ERROR_REACHABLE 02\n #define ERROR_PACK 04\n #define ERROR_REFS 010\n+#define ERROR_COMMIT_GRAPH 020\n \n static const char *describe_object(struct object *obj)\n {\n@@ -822,5 +824,24 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t}\n \n \tcheck_connectivity();\n+\n+\tif (core_commit_graph) {\n+\t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n+\t\tconst char *verify_argv[] = { \"commit-graph\", \"verify\", NULL, NULL, NULL };\n+\t\tcommit_graph_verify.argv = verify_argv;\n+\t\tcommit_graph_verify.git_cmd = 1;\n+\n+\t\tif (run_command(&commit_graph_verify))\n+\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n+\n+\t\tprepare_alt_odb(the_repository);\n+\t\tfor (alt = the_repository->objects->alt_odb_list; alt; alt = alt->next) {\n+\t\t\tverify_argv[2] = \"--object-dir\";\n+\t\t\tverify_argv[3] = alt->path;\n+\t\t\tif (run_command(&commit_graph_verify))\n+\t\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n+\t\t}\n+\t}\n+\n \treturn errors_found;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 2297a44e7d..44d4c71f0b 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -399,4 +399,12 @@ test_expect_success 'detect invalid checksum hash' '\n \t\t\"incorrect checksum\"\n '\n \n+test_expect_success 'git fsck (checks commit-graph)' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit fsck &&\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_FOOTER \"\\00\" \\\n+\t\t\"incorrect checksum\" &&\n+\ttest_must_fail git fsck\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349477","messageId":"20180606113611.87822-16-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 15/21] commit-graph: test for corrupted octopus edge","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:35Z","receivedAt":"2018-06-06T11:36:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file has an extra chunk to store the parent int-ids for\nparents beyond the first parent for octopus merges. Our test repo has a\nsingle octopus merge that we can manipulate to demonstrate the 'verify'\nsubcommand detects incorrect values in that chunk.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5318-commit-graph.sh | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 6a873bfda8..cf67fb391a 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -248,6 +248,7 @@ test_expect_success 'git commit-graph verify' '\n '\n \n NUM_COMMITS=9\n+NUM_OCTOPUS_EDGES=2\n HASH_LEN=20\n GRAPH_BYTE_VERSION=4\n GRAPH_BYTE_HASH=5\n@@ -274,6 +275,10 @@ GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4))\n GRAPH_BYTE_COMMIT_WRONG_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3))\n GRAPH_BYTE_COMMIT_GENERATION=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 11))\n GRAPH_BYTE_COMMIT_DATE=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 12))\n+GRAPH_COMMIT_DATA_WIDTH=$(($HASH_LEN + 16))\n+GRAPH_OCTOPUS_DATA_OFFSET=$(($GRAPH_COMMIT_DATA_OFFSET + \\\n+\t\t\t $GRAPH_COMMIT_DATA_WIDTH \\* $NUM_COMMITS))\n+GRAPH_BYTE_OCTOPUS=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -383,4 +388,9 @@ test_expect_success 'detect incorrect commit date' '\n \t\t\"commit date\"\n '\n \n+test_expect_success 'detect incorrect parent for octopus merge' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OCTOPUS \"\\01\" \\\n+\t\t\"invalid parent\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349478","messageId":"20180606113611.87822-22-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 21/21] commit-graph: update design document","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:40Z","receivedAt":"2018-06-06T11:36:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph feature is now integrated with 'fsck' and 'gc',\nso remove those items from the \"Future Work\" section of the\ncommit-graph design document.\n\nAlso remove the section on lazy-loading trees, as that was completed\nin an earlier patch series.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 22 ----------------------\n 1 file changed, 22 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex e1a883eb46..c664acbd76 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -118,9 +118,6 @@ Future Work\n - The commit graph feature currently does not honor commit grafts. This can\n be remedied by duplicating or refactoring the current graft logic.\n \n-- The 'commit-graph' subcommand does not have a \"verify\" mode that is\n- necessary for integration with fsck.\n-\n - After computing and storing generation numbers, we must make graph\n walks aware of generation numbers to gain the performance benefits they\n enable. This will mostly be accomplished by swapping a commit-date-ordered\n@@ -130,25 +127,6 @@ Future Work\n - 'log --topo-order'\n - 'tag --merged'\n \n-- Currently, parse_commit_gently() requires filling in the root tree\n- object for a commit. This passes through lookup_tree() and consequently\n- lookup_object(). Also, it calls lookup_commit() when loading the parents.\n- These method calls check the ODB for object existence, even if the\n- consumer does not need the content. For example, we do not need the\n- tree contents when computing merge bases. Now that commit parsing is\n- removed from the computation time, these lookup operations are the\n- slowest operations keeping graph walks from being fast. Consider\n- loading these objects without verifying their existence in the ODB and\n- only loading them fully when consumers need them. Consider a method\n- such as \"ensure_tree_loaded(commit)\" that fully loads a tree before\n- using commit->tree.\n-\n-- The current design uses the 'commit-graph' subcommand to generate the graph.\n- When this feature stabilizes enough to recommend to most users, we should\n- add automatic graph writes to common operations that create many commits.\n- For example, one could compute a graph on 'clone', 'fetch', or 'repack'\n- commands.\n-\n - A server could provide a commit graph file as part of the network protocol\n to avoid extra calculations by clients. This feature is only of benefit if\n the user is willing to trust the file, because verifying the file is correct\n-- \n2.18.0.rc1\n\n"},{"id":"349479","messageId":"20180606113611.87822-14-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 13/21] commit-graph: verify generation number","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:33Z","receivedAt":"2018-06-06T11:36:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"While iterating through the commit parents, perform the generation\nnumber calculation and compare against the value stored in the\ncommit-graph.\n\nThe tests demonstrate that having a different set of parents affects\nthe generation number calculation, and this value propagates to\ndescendants. Hence, we drop the single-line condition on the output.\n\nSince Git will ship with the commit-graph feature without generation\nnumbers, we need to accept commit-graphs with all generation numbers\nequal to zero. In this case, ignore the generation number calculation.\n\nHowever, verify that we should never have a mix of zero and non-zero\ngeneration numbers. Create a test that sets one commit to generation\nzero and all following commits report a failure as they have non-zero\ngeneration in a file that contains generation number zero.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 34 ++++++++++++++++++++++++++++++++++\n t/t5318-commit-graph.sh | 11 +++++++++++\n 2 files changed, 45 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex fcebd0925c..b97fa05ec9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -845,10 +845,14 @@ static void graph_report(const char *fmt, ...)\n \tva_end(ap);\n }\n \n+#define GENERATION_ZERO_EXISTS 1\n+#define GENERATION_NUMBER_EXISTS 2\n+\n int verify_commit_graph(struct commit_graph *g)\n {\n \tuint32_t i, cur_fanout_pos = 0;\n \tstruct object_id prev_oid, cur_oid;\n+\tint generation_zero = 0;\n \n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n@@ -910,6 +914,7 @@ int verify_commit_graph(struct commit_graph *g)\n \tfor (i = 0; i < g->num_commits; i++) {\n \t\tstruct commit *graph_commit, *odb_commit;\n \t\tstruct commit_list *graph_parents, *odb_parents;\n+\t\tuint32_t max_generation = 0;\n \n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n@@ -944,6 +949,9 @@ int verify_commit_graph(struct commit_graph *g)\n \t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid),\n \t\t\t\t\t oid_to_hex(&odb_parents->item->object.oid));\n \n+\t\t\tif (graph_parents->item->generation > max_generation)\n+\t\t\t\tmax_generation = graph_parents->item->generation;\n+\n \t\t\tgraph_parents = graph_parents->next;\n \t\t\todb_parents = odb_parents->next;\n \t\t}\n@@ -951,6 +959,32 @@ int verify_commit_graph(struct commit_graph *g)\n \t\tif (odb_parents != NULL)\n \t\t\tgraph_report(\"commit-graph parent list for commit %s terminates early\",\n \t\t\t\t oid_to_hex(&cur_oid));\n+\n+\t\tif (!graph_commit->generation) {\n+\t\t\tif (generation_zero == GENERATION_NUMBER_EXISTS)\n+\t\t\t\tgraph_report(\"commit-graph has generation number zero for commit %s, but non-zero elsewhere\",\n+\t\t\t\t\t oid_to_hex(&cur_oid));\n+\t\t\tgeneration_zero = GENERATION_ZERO_EXISTS;\n+\t\t} else if (generation_zero == GENERATION_ZERO_EXISTS)\n+\t\t\tgraph_report(\"commit-graph has non-zero generation number for commit %s, but zero elsewhere\",\n+\t\t\t\t oid_to_hex(&cur_oid));\n+\n+\t\tif (generation_zero == GENERATION_ZERO_EXISTS)\n+\t\t\tcontinue;\n+\n+\t\t/*\n+\t\t * If one of our parents has generation GENERATION_NUMBER_MAX, then\n+\t\t * our generation is also GENERATION_NUMBER_MAX. Decrement to avoid\n+\t\t * extra logic in the following condition.\n+\t\t */\n+\t\tif (max_generation == GENERATION_NUMBER_MAX)\n+\t\t\tmax_generation--;\n+\n+\t\tif (graph_commit->generation != max_generation + 1)\n+\t\t\tgraph_report(\"commit-graph generation for commit %s is %u != %u\",\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t graph_commit->generation,\n+\t\t\t\t max_generation + 1);\n \t}\n \n \treturn verify_commit_graph_error;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex ec0964112a..a6ea1341dc 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -272,6 +272,7 @@ GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n GRAPH_BYTE_COMMIT_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN))\n GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4))\n GRAPH_BYTE_COMMIT_WRONG_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3))\n+GRAPH_BYTE_COMMIT_GENERATION=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 11))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -366,4 +367,14 @@ test_expect_success 'detect wrong parent' '\n \t\t\"commit-graph parent for\"\n '\n \n+test_expect_success 'detect incorrect generation number' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_GENERATION \"\\070\" \\\n+\t\t\"generation for commit\"\n+'\n+\n+test_expect_success 'detect incorrect generation number' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_GENERATION \"\\01\" \\\n+\t\t\"non-zero generation number\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349480","messageId":"20180606113611.87822-21-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 20/21] gc: automatically write commit-graph files","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:39Z","receivedAt":"2018-06-06T11:36:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file is a very helpful feature for speeding up git\noperations. In order to make it more useful, make it possible to\nwrite the commit-graph file during standard garbage collection\noperations.\n\nAdd a 'gc.commitGraph' config setting that triggers writing a\ncommit-graph file after any non-trivial 'git gc' command. Defaults to\nfalse while the commit-graph feature matures. We specifically do not\nwant to have this on by default until the commit-graph feature is fully\nintegrated with history-modifying features like shallow clones.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 10 +++++++++-\n Documentation/git-gc.txt | 4 ++++\n builtin/gc.c | 6 ++++++\n t/t5318-commit-graph.sh | 14 ++++++++++++++\n 4 files changed, 33 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex ab641bf5a9..f2b5ed17c8 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -906,7 +906,8 @@ the `GIT_NOTES_REF` environment variable. See linkgit:git-notes[1].\n \n core.commitGraph::\n \tEnable git commit graph feature. Allows reading from the\n-\tcommit-graph file.\n+\tcommit-graph file. See `gc.commitGraph` for automatically\n+\tmaintaining the file.\n \n core.sparseCheckout::\n \tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n@@ -1647,6 +1648,13 @@ this configuration variable is ignored, all packs except the base pack\n will be repacked. After this the number of packs should go below\n gc.autoPackLimit and gc.bigPackThreshold should be respected again.\n \n+gc.commitGraph::\n+\tIf true, then gc will rewrite the commit-graph file when\n+\tlinkgit:git-gc[1] is run. When using linkgit:git-gc[1]\n+\t'--auto' the commit-graph will be updated if housekeeping is\n+\trequired. Default is false. See linkgit:git-commit-graph[1]\n+\tfor details.\n+\n gc.logExpiry::\n \tIf the file gc.log exists, then `git gc --auto` won't run\n \tunless that file is more than 'gc.logExpiry' old. Default is\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 24b2dd44fe..f5bc98ccb3 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -136,6 +136,10 @@ The optional configuration variable `gc.packRefs` determines if\n it within all non-bare repos or it can be set to a boolean value.\n This defaults to true.\n \n+The optional configuration variable `gc.commitGraph` determines if\n+'git gc' should run 'git commit-graph write'. This can be set to a\n+boolean value. This defaults to false.\n+\n The optional configuration variable `gc.aggressiveWindow` controls how\n much time is spent optimizing the delta compression of the objects in\n the repository when the --aggressive option is specified. The larger\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex ccfb1ceaeb..4e06e8372d 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -20,6 +20,7 @@\n #include \"sigchain.h\"\n #include \"argv-array.h\"\n #include \"commit.h\"\n+#include \"commit-graph.h\"\n #include \"packfile.h\"\n #include \"object-store.h\"\n #include \"pack.h\"\n@@ -40,6 +41,7 @@ static int aggressive_depth = 50;\n static int aggressive_window = 250;\n static int gc_auto_threshold = 6700;\n static int gc_auto_pack_limit = 50;\n+static int gc_commit_graph = 0;\n static int detach_auto = 1;\n static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n@@ -129,6 +131,7 @@ static void gc_config(void)\n \tgit_config_get_int(\"gc.aggressivedepth\", &aggressive_depth);\n \tgit_config_get_int(\"gc.auto\", &gc_auto_threshold);\n \tgit_config_get_int(\"gc.autopacklimit\", &gc_auto_pack_limit);\n+\tgit_config_get_bool(\"gc.commitgraph\", &gc_commit_graph);\n \tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n \tgit_config_get_expiry(\"gc.pruneexpire\", &prune_expire);\n \tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n@@ -641,6 +644,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tif (pack_garbage.nr > 0)\n \t\tclean_pack_garbage();\n \n+\tif (gc_commit_graph)\n+\t\twrite_commit_graph_reachable(get_object_directory(), 0);\n+\n \tif (auto_gc && too_many_loose_objects())\n \t\twarning(_(\"There are too many unreachable loose objects; \"\n \t\t\t\"run 'git prune' to remove them.\"));\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex ffb2ed7c95..b24e8b6689 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -245,6 +245,20 @@ test_expect_success 'perform fast-forward merge in full repo' '\n \ttest_cmp expect output\n '\n \n+test_expect_success 'check that gc computes commit-graph' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit --allow-empty -m \"blank\" &&\n+\tgit commit-graph write --reachable &&\n+\tcp $objdir/info/commit-graph commit-graph-before-gc &&\n+\tgit reset --hard HEAD~1 &&\n+\tgit config gc.commitGraph true &&\n+\tgit gc &&\n+\tcp $objdir/info/commit-graph commit-graph-after-gc &&\n+\t! test_cmp commit-graph-before-gc commit-graph-after-gc &&\n+\tgit commit-graph write --reachable &&\n+\ttest_cmp commit-graph-after-gc $objdir/info/commit-graph\n+'\n+\n # the verify tests below expect the commit-graph to contain\n # exactly the commits reachable from the commits/8 branch.\n # If the file changes the set of commits in the list, then the\n-- \n2.18.0.rc1\n\n"},{"id":"349481","messageId":"20180606113611.87822-11-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 10/21] commit-graph: verify objects exist","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:31Z","receivedAt":"2018-06-06T11:36:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In the 'verify' subcommand, load commits directly from the object\ndatabase to ensure they exist. Parse by skipping the commit-graph.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 17 +++++++++++++++++\n t/t5318-commit-graph.sh | 7 +++++++\n 2 files changed, 24 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d7a5b50a6c..893cc2f346 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -242,6 +242,7 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n {\n \tstruct commit *c;\n \tstruct object_id oid;\n+\n \thashcpy(oid.hash, g->chunk_oid_lookup + g->hash_len * pos);\n \tc = lookup_commit(&oid);\n \tif (!c)\n@@ -893,5 +894,21 @@ int verify_commit_graph(struct commit_graph *g)\n \t\tcur_fanout_pos++;\n \t}\n \n+\tif (verify_commit_graph_error)\n+\t\treturn verify_commit_graph_error;\n+\n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct commit *odb_commit;\n+\n+\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n+\n+\t\todb_commit = (struct commit *)create_object(cur_oid.hash, alloc_commit_node());\n+\t\tif (parse_commit_internal(odb_commit, 0, 0)) {\n+\t\t\tgraph_report(\"failed to parse %s from object database\",\n+\t\t\t\t oid_to_hex(&cur_oid));\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+\n \treturn verify_commit_graph_error;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex c29eae47c9..cf60e48496 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -247,6 +247,7 @@ test_expect_success 'git commit-graph verify' '\n \tgit commit-graph verify >output\n '\n \n+NUM_COMMITS=9\n HASH_LEN=20\n GRAPH_BYTE_VERSION=4\n GRAPH_BYTE_HASH=5\n@@ -265,6 +266,7 @@ GRAPH_BYTE_FANOUT1=$(($GRAPH_FANOUT_OFFSET + 4 \\* 4))\n GRAPH_BYTE_FANOUT2=$(($GRAPH_FANOUT_OFFSET + 4 \\* 255))\n GRAPH_OID_LOOKUP_OFFSET=$(($GRAPH_FANOUT_OFFSET + 4 \\* 256))\n GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8))\n+GRAPH_BYTE_OID_LOOKUP_MISSING=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 4 + 10))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -334,4 +336,9 @@ test_expect_success 'detect incorrect OID order' '\n \t\t\"incorrect OID order\"\n '\n \n+test_expect_success 'detect OID not in object database' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_LOOKUP_MISSING \"\\01\" \\\n+\t\t\"from object database\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349482","messageId":"20180606113611.87822-20-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 19/21] commit-graph: add '--reachable' option","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:38Z","receivedAt":"2018-06-06T11:37:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When writing commit-graph files, it can be convenient to ask for all\nreachable commits (starting at the ref set) in the resulting file. This\nis particularly helpful when writing to stdin is complicated, such as a\nfuture integration with 'git gc'.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 8 ++++++--\n builtin/commit-graph.c | 16 ++++++++++++----\n commit-graph.c | 18 ++++++++++++++++++\n commit-graph.h | 1 +\n t/t5318-commit-graph.sh | 10 ++++++++++\n 5 files changed, 47 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex a222cfab08..dececb79d7 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -38,12 +38,16 @@ Write a commit graph file based on the commits found in packfiles.\n +\n With the `--stdin-packs` option, generate the new commit graph by\n walking objects only in the specified pack-indexes. (Cannot be combined\n-with --stdin-commits.)\n+with `--stdin-commits` or `--reachable`.)\n +\n With the `--stdin-commits` option, generate the new commit graph by\n walking commits starting at the commits specified in stdin as a list\n of OIDs in hex, one OID per line. (Cannot be combined with\n---stdin-packs.)\n+`--stdin-packs` or `--reachable`.)\n++\n+With the `--reachable` option, generate the new commit graph by walking\n+commits starting at all refs. (Cannot be combined with `--stdin-commits`\n+or `--stdin-packs`.)\n +\n With the `--append` option, include all commits that are present in the\n existing commit-graph file.\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex d8eb8278b3..76423b3fa5 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -9,7 +9,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph verify [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -24,12 +24,13 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n static struct opts_commit_graph {\n \tconst char *obj_dir;\n+\tint reachable;\n \tint stdin_packs;\n \tint stdin_commits;\n \tint append;\n@@ -126,6 +127,8 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n \t\t\tN_(\"dir\"),\n \t\t\tN_(\"The object directory to store the graph\")),\n+\t\tOPT_BOOL(0, \"reachable\", &opts.reachable,\n+\t\t\tN_(\"start walk at all refs\")),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\tN_(\"scan pack-indexes listed by stdin for commits\")),\n \t\tOPT_BOOL(0, \"stdin-commits\", &opts.stdin_commits,\n@@ -139,11 +142,16 @@ static int graph_write(int argc, const char **argv)\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\n \n-\tif (opts.stdin_packs && opts.stdin_commits)\n-\t\tdie(_(\"cannot use both --stdin-commits and --stdin-packs\"));\n+\tif (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)\n+\t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n \n+\tif (opts.reachable) {\n+\t\twrite_commit_graph_reachable(opts.obj_dir, opts.append);\n+\t\treturn 0;\n+\t}\n+\n \tif (opts.stdin_packs || opts.stdin_commits) {\n \t\tstruct strbuf buf = STRBUF_INIT;\n \t\tstring_list_init(&lines, 0);\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f23bf4cf50..0d5adc8035 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -7,6 +7,7 @@\n #include \"packfile.h\"\n #include \"commit.h\"\n #include \"object.h\"\n+#include \"refs.h\"\n #include \"revision.h\"\n #include \"sha1-lookup.h\"\n #include \"commit-graph.h\"\n@@ -655,6 +656,23 @@ static void compute_generation_numbers(struct packed_commit_list* commits)\n \t}\n }\n \n+static int add_ref_to_list(const char *refname,\n+\t\t\t const struct object_id *oid,\n+\t\t\t int flags, void *cb_data)\n+{\n+\tstruct string_list *list = (struct string_list*)cb_data;\n+\tstring_list_append(list, oid_to_hex(oid));\n+\treturn 0;\n+}\n+\n+void write_commit_graph_reachable(const char *obj_dir, int append)\n+{\n+\tstruct string_list list;\n+\tstring_list_init(&list, 1);\n+\tfor_each_ref(add_ref_to_list, &list);\n+\twrite_commit_graph(obj_dir, NULL, &list, append);\n+}\n+\n void write_commit_graph(const char *obj_dir,\n \t\t\tstruct string_list *pack_indexes,\n \t\t\tstruct string_list *commit_hex,\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 66661e1fc5..ee20f5e280 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -47,6 +47,7 @@ struct commit_graph {\n \n struct commit_graph *load_commit_graph_one(const char *graph_file);\n \n+void write_commit_graph_reachable(const char *obj_dir, int append);\n void write_commit_graph(const char *obj_dir,\n \t\t\tstruct string_list *pack_indexes,\n \t\t\tstruct string_list *commit_hex,\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 44d4c71f0b..ffb2ed7c95 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -205,6 +205,16 @@ test_expect_success 'build graph from commits with append' '\n graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n \n+test_expect_success 'build graph using --reachable' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph write --reachable &&\n+\ttest_path_is_file $objdir/info/commit-graph &&\n+\tgraph_read_expect \"11\" \"large_edges\"\n+'\n+\n+graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n+graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n+\n test_expect_success 'setup bare repo' '\n \tcd \"$TRASH_DIRECTORY\" &&\n \tgit clone --bare --no-local full bare &&\n-- \n2.18.0.rc1\n\n"},{"id":"349483","messageId":"20180606113611.87822-4-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 03/21] commit-graph: parse commit from chosen graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:24Z","receivedAt":"2018-06-06T11:37:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Before verifying a commit-graph file against the object database, we\nneed to parse all commits from the given commit-graph file. Create\nparse_commit_in_graph_one() to target a given struct commit_graph.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 18 +++++++++++++++---\n 1 file changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f83f6d2373..e77b19971d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -314,7 +314,7 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \t}\n }\n \n-int parse_commit_in_graph(struct commit *item)\n+static int parse_commit_in_graph_one(struct commit_graph *g, struct commit *item)\n {\n \tuint32_t pos;\n \n@@ -322,9 +322,21 @@ int parse_commit_in_graph(struct commit *item)\n \t\treturn 0;\n \tif (item->object.parsed)\n \t\treturn 1;\n+\n+\tif (find_commit_in_graph(item, g, &pos))\n+\t\treturn fill_commit_in_graph(item, g, pos);\n+\n+\treturn 0;\n+}\n+\n+int parse_commit_in_graph(struct commit *item)\n+{\n+\tif (!core_commit_graph)\n+\t\treturn 0;\n+\n \tprepare_commit_graph();\n-\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n-\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n+\tif (commit_graph)\n+\t\treturn parse_commit_in_graph_one(commit_graph, item);\n \treturn 0;\n }\n \n-- \n2.18.0.rc1\n\n"},{"id":"349484","messageId":"20180606113611.87822-19-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 18/21] commit-graph: use string-list API for input","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:37Z","receivedAt":"2018-06-06T11:37:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 39 +++++++++++++--------------------------\n commit-graph.c | 15 +++++++--------\n commit-graph.h | 7 +++----\n 3 files changed, 23 insertions(+), 38 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 3079cde6f9..d8eb8278b3 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -118,13 +118,9 @@ static int graph_read(int argc, const char **argv)\n \n static int graph_write(int argc, const char **argv)\n {\n-\tconst char **pack_indexes = NULL;\n-\tint packs_nr = 0;\n-\tconst char **commit_hex = NULL;\n-\tint commits_nr = 0;\n-\tconst char **lines = NULL;\n-\tint lines_nr = 0;\n-\tint lines_alloc = 0;\n+\tstruct string_list *pack_indexes = NULL;\n+\tstruct string_list *commit_hex = NULL;\n+\tstruct string_list lines;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n@@ -150,32 +146,23 @@ static int graph_write(int argc, const char **argv)\n \n \tif (opts.stdin_packs || opts.stdin_commits) {\n \t\tstruct strbuf buf = STRBUF_INIT;\n-\t\tlines_nr = 0;\n-\t\tlines_alloc = 128;\n-\t\tALLOC_ARRAY(lines, lines_alloc);\n-\n-\t\twhile (strbuf_getline(&buf, stdin) != EOF) {\n-\t\t\tALLOC_GROW(lines, lines_nr + 1, lines_alloc);\n-\t\t\tlines[lines_nr++] = strbuf_detach(&buf, NULL);\n-\t\t}\n-\n-\t\tif (opts.stdin_packs) {\n-\t\t\tpack_indexes = lines;\n-\t\t\tpacks_nr = lines_nr;\n-\t\t}\n-\t\tif (opts.stdin_commits) {\n-\t\t\tcommit_hex = lines;\n-\t\t\tcommits_nr = lines_nr;\n-\t\t}\n+\t\tstring_list_init(&lines, 0);\n+\n+\t\twhile (strbuf_getline(&buf, stdin) != EOF)\n+\t\t\tstring_list_append(&lines, strbuf_detach(&buf, NULL));\n+\n+\t\tif (opts.stdin_packs)\n+\t\t\tpack_indexes = &lines;\n+\t\tif (opts.stdin_commits)\n+\t\t\tcommit_hex = &lines;\n \t}\n \n \twrite_commit_graph(opts.obj_dir,\n \t\t\t pack_indexes,\n-\t\t\t packs_nr,\n \t\t\t commit_hex,\n-\t\t\t commits_nr,\n \t\t\t opts.append);\n \n+\tstring_list_clear(&lines, 0);\n \treturn 0;\n }\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0f93d5d864..f23bf4cf50 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -656,10 +656,8 @@ static void compute_generation_numbers(struct packed_commit_list* commits)\n }\n \n void write_commit_graph(const char *obj_dir,\n-\t\t\tconst char **pack_indexes,\n-\t\t\tint nr_packs,\n-\t\t\tconst char **commit_hex,\n-\t\t\tint nr_commits,\n+\t\t\tstruct string_list *pack_indexes,\n+\t\t\tstruct string_list *commit_hex,\n \t\t\tint append)\n {\n \tstruct packed_oid_list oids;\n@@ -700,10 +698,10 @@ void write_commit_graph(const char *obj_dir,\n \t\tint dirlen;\n \t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n \t\tdirlen = packname.len;\n-\t\tfor (i = 0; i < nr_packs; i++) {\n+\t\tfor (i = 0; i < pack_indexes->nr; i++) {\n \t\t\tstruct packed_git *p;\n \t\t\tstrbuf_setlen(&packname, dirlen);\n-\t\t\tstrbuf_addstr(&packname, pack_indexes[i]);\n+\t\t\tstrbuf_addstr(&packname, pack_indexes->items[i].string);\n \t\t\tp = add_packed_git(packname.buf, packname.len, 1);\n \t\t\tif (!p)\n \t\t\t\tdie(\"error adding pack %s\", packname.buf);\n@@ -716,12 +714,13 @@ void write_commit_graph(const char *obj_dir,\n \t}\n \n \tif (commit_hex) {\n-\t\tfor (i = 0; i < nr_commits; i++) {\n+\t\tfor (i = 0; i < commit_hex->nr; i++) {\n \t\t\tconst char *end;\n \t\t\tstruct object_id oid;\n \t\t\tstruct commit *result;\n \n-\t\t\tif (commit_hex[i] && parse_oid_hex(commit_hex[i], &oid, &end))\n+\t\t\tif (commit_hex->items[i].string &&\n+\t\t\t parse_oid_hex(commit_hex->items[i].string, &oid, &end))\n \t\t\t\tcontinue;\n \n \t\t\tresult = lookup_commit_reference_gently(&oid, 1);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 71a39c5a57..66661e1fc5 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -2,6 +2,7 @@\n #define COMMIT_GRAPH_H\n \n #include \"git-compat-util.h\"\n+#include \"string-list.h\"\n \n char *get_commit_graph_filename(const char *obj_dir);\n \n@@ -47,10 +48,8 @@ struct commit_graph {\n struct commit_graph *load_commit_graph_one(const char *graph_file);\n \n void write_commit_graph(const char *obj_dir,\n-\t\t\tconst char **pack_indexes,\n-\t\t\tint nr_packs,\n-\t\t\tconst char **commit_hex,\n-\t\t\tint nr_commits,\n+\t\t\tstruct string_list *pack_indexes,\n+\t\t\tstruct string_list *commit_hex,\n \t\t\tint append);\n \n int verify_commit_graph(struct commit_graph *g);\n-- \n2.18.0.rc1\n\n"},{"id":"349485","messageId":"20180606113611.87822-7-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 06/21] commit-graph: add 'verify' subcommand","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:27Z","receivedAt":"2018-06-06T11:37:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"If the commit-graph file becomes corrupt, we need a way to verify\nthat its contents match the object database. In the manner of\n'git fsck' we will implement a 'git commit-graph verify' subcommand\nto report all issues with the file.\n\nAdd the 'verify' subcommand to the 'commit-graph' builtin and its\ndocumentation. The subcommand is currently a no-op except for\nloading the commit-graph into memory, which may trigger run-time\nerrors that would be caught by normal use. Add a simple test that\nensures the command returns a zero error code.\n\nIf no commit-graph file exists, this is an acceptable state. Do\nnot report any errors.\n\nHelped-by: Ramsay Jones \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 6 +++++\n builtin/commit-graph.c | 38 ++++++++++++++++++++++++++++++\n commit-graph.c | 23 ++++++++++++++++++\n commit-graph.h | 2 ++\n t/t5318-commit-graph.sh | 10 ++++++++\n 5 files changed, 79 insertions(+)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 4c97b555cc..a222cfab08 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git commit-graph read' [--object-dir ]\n+'git commit-graph verify' [--object-dir ]\n 'git commit-graph write' [--object-dir ]\n \n \n@@ -52,6 +53,11 @@ existing commit-graph file.\n Read a graph file given by the commit-graph file and output basic\n details about the graph file. Used for debugging purposes.\n \n+'verify'::\n+\n+Read the commit-graph file and verify its contents against the object\n+database. Used to check for corrupted data.\n+\n \n EXAMPLES\n --------\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex f0875b8bf3..3079cde6f9 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -8,10 +8,16 @@\n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n+\tN_(\"git commit-graph verify [--object-dir ]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n+static const char * const builtin_commit_graph_verify_usage[] = {\n+\tN_(\"git commit-graph verify [--object-dir ]\"),\n+\tNULL\n+};\n+\n static const char * const builtin_commit_graph_read_usage[] = {\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tNULL\n@@ -29,6 +35,36 @@ static struct opts_commit_graph {\n \tint append;\n } opts;\n \n+\n+static int graph_verify(int argc, const char **argv)\n+{\n+\tstruct commit_graph *graph = NULL;\n+\tchar *graph_name;\n+\n+\tstatic struct option builtin_commit_graph_verify_options[] = {\n+\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n+\t\t\t N_(\"dir\"),\n+\t\t\t N_(\"The object directory to store the graph\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t builtin_commit_graph_verify_options,\n+\t\t\t builtin_commit_graph_verify_usage, 0);\n+\n+\tif (!opts.obj_dir)\n+\t\topts.obj_dir = get_object_directory();\n+\n+\tgraph_name = get_commit_graph_filename(opts.obj_dir);\n+\tgraph = load_commit_graph_one(graph_name);\n+\tFREE_AND_NULL(graph_name);\n+\n+\tif (!graph)\n+\t\treturn 0;\n+\n+\treturn verify_commit_graph(graph);\n+}\n+\n static int graph_read(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n@@ -165,6 +201,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \tif (argc > 0) {\n \t\tif (!strcmp(argv[0], \"read\"))\n \t\t\treturn graph_read(argc, argv);\n+\t\tif (!strcmp(argv[0], \"verify\"))\n+\t\t\treturn graph_verify(argc, argv);\n \t\tif (!strcmp(argv[0], \"write\"))\n \t\t\treturn graph_write(argc, argv);\n \t}\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 9e228d3bb5..432920ad2a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -827,3 +827,26 @@ void write_commit_graph(const char *obj_dir,\n \toids.alloc = 0;\n \toids.nr = 0;\n }\n+\n+static int verify_commit_graph_error;\n+\n+static void graph_report(const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tverify_commit_graph_error = 1;\n+\n+\tva_start(ap, fmt);\n+\tvfprintf(stderr, fmt, ap);\n+\tfprintf(stderr, \"\\n\");\n+\tva_end(ap);\n+}\n+\n+int verify_commit_graph(struct commit_graph *g)\n+{\n+\tif (!g) {\n+\t\tgraph_report(\"no commit-graph file loaded\");\n+\t\treturn 1;\n+\t}\n+\n+\treturn verify_commit_graph_error;\n+}\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 96cccb10f3..71a39c5a57 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -53,4 +53,6 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tint nr_commits,\n \t\t\tint append);\n \n+int verify_commit_graph(struct commit_graph *g);\n+\n #endif\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 59d0be2877..0830ef9fdd 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -11,6 +11,11 @@ test_expect_success 'setup full repo' '\n \tobjdir=\".git/objects\"\n '\n \n+test_expect_success 'verify graph with no graph file' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph verify\n+'\n+\n test_expect_success 'write graph with no packs' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write --object-dir . &&\n@@ -230,4 +235,9 @@ test_expect_success 'perform fast-forward merge in full repo' '\n \ttest_cmp expect output\n '\n \n+test_expect_success 'git commit-graph verify' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph verify >output\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349486","messageId":"20180606113611.87822-17-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 16/21] commit-graph: verify contents match checksum","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:36Z","receivedAt":"2018-06-06T11:37:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file ends with a SHA1 hash of the previous contents. If\na commit-graph file has errors but the checksum hash is correct, then we\nknow that the problem is a bug in Git and not simply file corruption\nafter-the-fact.\n\nCompute the checksum right away so it is the first error that appears,\nand make the message translatable since this error can be \"corrected\" by\na user by simply deleting the file and recomputing. The rest of the\nerrors are useful only to developers.\n\nBe sure to continue checking the rest of the file data if the checksum\nis wrong. This is important for our tests, as we break the checksum as\nwe modify bytes of the commit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 16 ++++++++++++++--\n t/t5318-commit-graph.sh | 6 ++++++\n 2 files changed, 20 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d83f0ce5d5..0f93d5d864 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -832,6 +832,7 @@ void write_commit_graph(const char *obj_dir,\n \toids.nr = 0;\n }\n \n+#define VERIFY_COMMIT_GRAPH_ERROR_HASH 2\n static int verify_commit_graph_error;\n \n static void graph_report(const char *fmt, ...)\n@@ -851,8 +852,10 @@ static void graph_report(const char *fmt, ...)\n int verify_commit_graph(struct commit_graph *g)\n {\n \tuint32_t i, cur_fanout_pos = 0;\n-\tstruct object_id prev_oid, cur_oid;\n+\tstruct object_id prev_oid, cur_oid, checksum;\n \tint generation_zero = 0;\n+\tstruct hashfile *f;\n+\tint devnull;\n \n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n@@ -871,6 +874,15 @@ int verify_commit_graph(struct commit_graph *g)\n \tif (verify_commit_graph_error)\n \t\treturn verify_commit_graph_error;\n \n+\tdevnull = open(\"/dev/null\", O_WRONLY);\n+\tf = hashfd(devnull, NULL);\n+\thashwrite(f, g->data, g->data_len - g->hash_len);\n+\tfinalize_hashfile(f, checksum.hash, CSUM_CLOSE);\n+\tif (hashcmp(checksum.hash, g->data + g->data_len - g->hash_len)) {\n+\t\tgraph_report(_(\"the commit-graph file has incorrect checksum and is likely corrupt\"));\n+\t\tverify_commit_graph_error = VERIFY_COMMIT_GRAPH_ERROR_HASH;\n+\t}\n+\n \tfor (i = 0; i < g->num_commits; i++) {\n \t\tstruct commit *graph_commit;\n \n@@ -908,7 +920,7 @@ int verify_commit_graph(struct commit_graph *g)\n \t\tcur_fanout_pos++;\n \t}\n \n-\tif (verify_commit_graph_error)\n+\tif (verify_commit_graph_error & ~VERIFY_COMMIT_GRAPH_ERROR_HASH)\n \t\treturn verify_commit_graph_error;\n \n \tfor (i = 0; i < g->num_commits; i++) {\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex cf67fb391a..2297a44e7d 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -279,6 +279,7 @@ GRAPH_COMMIT_DATA_WIDTH=$(($HASH_LEN + 16))\n GRAPH_OCTOPUS_DATA_OFFSET=$(($GRAPH_COMMIT_DATA_OFFSET + \\\n \t\t\t $GRAPH_COMMIT_DATA_WIDTH \\* $NUM_COMMITS))\n GRAPH_BYTE_OCTOPUS=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4))\n+GRAPH_BYTE_FOOTER=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4 \\* $NUM_OCTOPUS_EDGES))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -393,4 +394,9 @@ test_expect_success 'detect incorrect parent for octopus merge' '\n \t\t\"invalid parent\"\n '\n \n+test_expect_success 'detect invalid checksum hash' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_FOOTER \"\\00\" \\\n+\t\t\"incorrect checksum\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349487","messageId":"20180606113611.87822-10-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 09/21] commit-graph: verify corrupt OID fanout and lookup","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:30Z","receivedAt":"2018-06-06T11:37:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In the commit-graph file, the OID fanout chunk provides an index into\nthe OID lookup. The 'verify' subcommand should find incorrect values\nin the fanout.\n\nSimilarly, the 'verify' subcommand should find out-of-order values in\nthe OID lookup.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 36 ++++++++++++++++++++++++++++++++++++\n t/t5318-commit-graph.sh | 22 ++++++++++++++++++++++\n 2 files changed, 58 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f41d5a0504..d7a5b50a6c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -843,6 +843,9 @@ static void graph_report(const char *fmt, ...)\n \n int verify_commit_graph(struct commit_graph *g)\n {\n+\tuint32_t i, cur_fanout_pos = 0;\n+\tstruct object_id prev_oid, cur_oid;\n+\n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n \t\treturn 1;\n@@ -857,5 +860,38 @@ int verify_commit_graph(struct commit_graph *g)\n \tif (!g->chunk_commit_data)\n \t\tgraph_report(\"commit-graph is missing the Commit Data chunk\");\n \n+\tif (verify_commit_graph_error)\n+\t\treturn verify_commit_graph_error;\n+\n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n+\n+\t\tif (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n+\t\t\tgraph_report(\"commit-graph has incorrect OID order: %s then %s\",\n+\t\t\t\t oid_to_hex(&prev_oid),\n+\t\t\t\t oid_to_hex(&cur_oid));\n+\n+\t\toidcpy(&prev_oid, &cur_oid);\n+\n+\t\twhile (cur_oid.hash[0] > cur_fanout_pos) {\n+\t\t\tuint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n+\t\t\tif (i != fanout_value)\n+\t\t\t\tgraph_report(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\",\n+\t\t\t\t\t cur_fanout_pos, fanout_value, i);\n+\n+\t\t\tcur_fanout_pos++;\n+\t\t}\n+\t}\n+\n+\twhile (cur_fanout_pos < 256) {\n+\t\tuint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n+\n+\t\tif (g->num_commits != fanout_value)\n+\t\t\tgraph_report(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\",\n+\t\t\t\t cur_fanout_pos, fanout_value, i);\n+\n+\t\tcur_fanout_pos++;\n+\t}\n+\n \treturn verify_commit_graph_error;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 846396665e..c29eae47c9 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -247,6 +247,7 @@ test_expect_success 'git commit-graph verify' '\n \tgit commit-graph verify >output\n '\n \n+HASH_LEN=20\n GRAPH_BYTE_VERSION=4\n GRAPH_BYTE_HASH=5\n GRAPH_BYTE_CHUNK_COUNT=6\n@@ -258,6 +259,12 @@ GRAPH_BYTE_OID_LOOKUP_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n \t\t\t 1 \\* $GRAPH_CHUNK_LOOKUP_WIDTH))\n GRAPH_BYTE_COMMIT_DATA_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n \t\t\t 2 \\* $GRAPH_CHUNK_LOOKUP_WIDTH))\n+GRAPH_FANOUT_OFFSET=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n+\t\t $GRAPH_CHUNK_LOOKUP_WIDTH \\* $GRAPH_CHUNK_LOOKUP_ROWS))\n+GRAPH_BYTE_FANOUT1=$(($GRAPH_FANOUT_OFFSET + 4 \\* 4))\n+GRAPH_BYTE_FANOUT2=$(($GRAPH_FANOUT_OFFSET + 4 \\* 255))\n+GRAPH_OID_LOOKUP_OFFSET=$(($GRAPH_FANOUT_OFFSET + 4 \\* 256))\n+GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -312,4 +319,19 @@ test_expect_success 'detect missing commit data chunk' '\n \t\t\"missing the Commit Data chunk\"\n '\n \n+test_expect_success 'detect incorrect fanout' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_FANOUT1 \"\\01\" \\\n+\t\t\"fanout value\"\n+'\n+\n+test_expect_success 'detect incorrect fanout final value' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_FANOUT2 \"\\01\" \\\n+\t\t\"fanout value\"\n+'\n+\n+test_expect_success 'detect incorrect OID order' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_LOOKUP_ORDER \"\\01\" \\\n+\t\t\"incorrect OID order\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349488","messageId":"20180606113611.87822-5-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 04/21] commit: force commit to parse from object database","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:25Z","receivedAt":"2018-06-06T11:37:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In anticipation of verifying commit-graph file contents against the\nobject database, create parse_commit_internal() to allow side-stepping\nthe commit-graph file and parse directly from the object database.\n\nDue to the use of generation numbers, this method should not be called\nunless the intention is explicit in avoiding commits from the\ncommit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 9 +++++++--\n commit.h | 1 +\n 2 files changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex 298ad747c6..922bb68741 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -405,7 +405,7 @@ int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long s\n \treturn 0;\n }\n \n-int parse_commit_gently(struct commit *item, int quiet_on_missing)\n+int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph)\n {\n \tenum object_type type;\n \tvoid *buffer;\n@@ -416,7 +416,7 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \t\treturn -1;\n \tif (item->object.parsed)\n \t\treturn 0;\n-\tif (parse_commit_in_graph(item))\n+\tif (use_commit_graph && parse_commit_in_graph(item))\n \t\treturn 0;\n \tbuffer = read_object_file(&item->object.oid, &type, &size);\n \tif (!buffer)\n@@ -437,6 +437,11 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \treturn ret;\n }\n \n+int parse_commit_gently(struct commit *item, int quiet_on_missing)\n+{\n+\treturn parse_commit_internal(item, quiet_on_missing, 1);\n+}\n+\n void parse_commit_or_die(struct commit *item)\n {\n \tif (parse_commit(item))\ndiff --git a/commit.h b/commit.h\nindex cb943013d0..4065fd12ac 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -77,6 +77,7 @@ struct commit *lookup_commit_reference_by_name(const char *name);\n struct commit *lookup_commit_or_die(const struct object_id *oid, const char *ref_name);\n \n int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph);\n+int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph);\n int parse_commit_gently(struct commit *item, int quiet_on_missing);\n static inline int parse_commit(struct commit *item)\n {\n-- \n2.18.0.rc1\n\n"},{"id":"349489","messageId":"20180606113611.87822-15-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 14/21] commit-graph: verify commit date","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:34Z","receivedAt":"2018-06-06T11:37:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n commit-graph.c | 6 ++++++\n t/t5318-commit-graph.sh | 6 ++++++\n 2 files changed, 12 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex b97fa05ec9..d83f0ce5d5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -985,6 +985,12 @@ int verify_commit_graph(struct commit_graph *g)\n \t\t\t\t oid_to_hex(&cur_oid),\n \t\t\t\t graph_commit->generation,\n \t\t\t\t max_generation + 1);\n+\n+\t\tif (graph_commit->date != odb_commit->date)\n+\t\t\tgraph_report(\"commit date for commit %s in commit-graph is %\"PRItime\" != %\"PRItime,\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t graph_commit->date,\n+\t\t\t\t odb_commit->date);\n \t}\n \n \treturn verify_commit_graph_error;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex a6ea1341dc..6a873bfda8 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -273,6 +273,7 @@ GRAPH_BYTE_COMMIT_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN))\n GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4))\n GRAPH_BYTE_COMMIT_WRONG_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3))\n GRAPH_BYTE_COMMIT_GENERATION=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 11))\n+GRAPH_BYTE_COMMIT_DATE=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 12))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -377,4 +378,9 @@ test_expect_success 'detect incorrect generation number' '\n \t\t\"non-zero generation number\"\n '\n \n+test_expect_success 'detect incorrect commit date' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_DATE \"\\01\" \\\n+\t\t\"commit date\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349490","messageId":"20180606113611.87822-8-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 07/21] commit-graph: verify catches corrupt signature","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:28Z","receivedAt":"2018-06-06T11:37:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is the first of several commits that add a test to check that\n'git commit-graph verify' catches corruption in the commit-graph\nfile. The first test checks that the command catches an error in\nthe file signature. This is a check that exists in the existing\ncommit-graph reading code.\n\nAdd a helper method 'corrupt_graph_and_verify' to the test script\nt5318-commit-graph.sh. This helper corrupts the commit-graph file\nat a certain location, runs 'git commit-graph verify', and reports\nthe output to the 'err' file. This data is filtered to remove the\nlines added by 'test_must_fail' when the test is run verbosely.\nThen, the output is checked to contain a specific error message.\n\nMost messages from 'git commit-graph verify' will not be marked\nfor translation. There will be one exception: the message that\nreports an invalid checksum will be marked for translation, as that\nis the only message that is intended for a typical user.\n\nHelped-by: Szeder Gábor \nSigned-off-by: Derrick Stolee \n---\n t/t5318-commit-graph.sh | 43 +++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 43 insertions(+)\n\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 0830ef9fdd..c0c1ff09b9 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -235,9 +235,52 @@ test_expect_success 'perform fast-forward merge in full repo' '\n \ttest_cmp expect output\n '\n \n+# the verify tests below expect the commit-graph to contain\n+# exactly the commits reachable from the commits/8 branch.\n+# If the file changes the set of commits in the list, then the\n+# offsets into the binary file will result in different edits\n+# and the tests will likely break.\n+\n test_expect_success 'git commit-graph verify' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit rev-parse commits/8 | git commit-graph write --stdin-commits &&\n \tgit commit-graph verify >output\n '\n \n+GRAPH_BYTE_VERSION=4\n+GRAPH_BYTE_HASH=5\n+\n+# usage: corrupt_graph_and_verify \n+# Manipulates the commit-graph file at the position\n+# by inserting the data, then runs 'git commit-graph verify'\n+# and places the output in the file 'err'. Test 'err' for\n+# the given string.\n+corrupt_graph_and_verify() {\n+\tpos=$1\n+\tdata=\"${2:-\\0}\"\n+\tgrepstr=$3\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n+\tcp $objdir/info/commit-graph commit-graph-backup &&\n+\tprintf \"$data\" | dd of=\"$objdir/info/commit-graph\" bs=1 seek=\"$pos\" conv=notrunc &&\n+\ttest_must_fail git commit-graph verify 2>test_err &&\n+\tgrep -v \"^+\" test_err >err\n+\ttest_i18ngrep \"$grepstr\" err\n+}\n+\n+test_expect_success 'detect bad signature' '\n+\tcorrupt_graph_and_verify 0 \"\\0\" \\\n+\t\t\"graph signature\"\n+'\n+\n+test_expect_success 'detect bad version' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\02\" \\\n+\t\t\"graph version\"\n+'\n+\n+test_expect_success 'detect bad hash version' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_HASH \"\\02\" \\\n+\t\t\"hash version\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349491","messageId":"20180606113611.87822-9-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 08/21] commit-graph: verify required chunks are present","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:29Z","receivedAt":"2018-06-06T11:37:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file requires the following three chunks:\n\n* OID Fanout\n* OID Lookup\n* Commit Data\n\nIf any of these are missing, then the 'verify' subcommand should\nreport a failure. This includes the chunk IDs malformed or the\nchunk count is truncated.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 9 +++++++++\n t/t5318-commit-graph.sh | 29 +++++++++++++++++++++++++++++\n 2 files changed, 38 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 432920ad2a..f41d5a0504 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -848,5 +848,14 @@ int verify_commit_graph(struct commit_graph *g)\n \t\treturn 1;\n \t}\n \n+\tverify_commit_graph_error = 0;\n+\n+\tif (!g->chunk_oid_fanout)\n+\t\tgraph_report(\"commit-graph is missing the OID Fanout chunk\");\n+\tif (!g->chunk_oid_lookup)\n+\t\tgraph_report(\"commit-graph is missing the OID Lookup chunk\");\n+\tif (!g->chunk_commit_data)\n+\t\tgraph_report(\"commit-graph is missing the Commit Data chunk\");\n+\n \treturn verify_commit_graph_error;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex c0c1ff09b9..846396665e 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -249,6 +249,15 @@ test_expect_success 'git commit-graph verify' '\n \n GRAPH_BYTE_VERSION=4\n GRAPH_BYTE_HASH=5\n+GRAPH_BYTE_CHUNK_COUNT=6\n+GRAPH_CHUNK_LOOKUP_OFFSET=8\n+GRAPH_CHUNK_LOOKUP_WIDTH=12\n+GRAPH_CHUNK_LOOKUP_ROWS=5\n+GRAPH_BYTE_OID_FANOUT_ID=$GRAPH_CHUNK_LOOKUP_OFFSET\n+GRAPH_BYTE_OID_LOOKUP_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n+\t\t\t 1 \\* $GRAPH_CHUNK_LOOKUP_WIDTH))\n+GRAPH_BYTE_COMMIT_DATA_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n+\t\t\t 2 \\* $GRAPH_CHUNK_LOOKUP_WIDTH))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -283,4 +292,24 @@ test_expect_success 'detect bad hash version' '\n \t\t\"hash version\"\n '\n \n+test_expect_success 'detect low chunk count' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_CHUNK_COUNT \"\\02\" \\\n+\t\t\"missing the .* chunk\"\n+'\n+\n+test_expect_success 'detect missing OID fanout chunk' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_FANOUT_ID \"\\0\" \\\n+\t\t\"missing the OID Fanout chunk\"\n+'\n+\n+test_expect_success 'detect missing OID lookup chunk' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_LOOKUP_ID \"\\0\" \\\n+\t\t\"missing the OID Lookup chunk\"\n+'\n+\n+test_expect_success 'detect missing commit data chunk' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_DATA_ID \"\\0\" \\\n+\t\t\"missing the Commit Data chunk\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349492","messageId":"20180606113611.87822-13-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 12/21] commit-graph: verify parent list","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:33Z","receivedAt":"2018-06-06T11:37:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file stores parents in a two-column portion of the\ncommit data chunk. If there is only one parent, then the second column\nstores 0xFFFFFFFF to indicate no second parent.\n\nThe 'verify' subcommand checks the parent list for the commit loaded\nfrom the commit-graph and the one parsed from the object database. Test\nthese checks for corrupt parents, too many parents, and wrong parents.\n\nAdd a boundary check to insert_parent_or_die() for when the parent\nposition value is out of range.\n\nThe octopus merge will be tested in a later commit.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 28 ++++++++++++++++++++++++++++\n t/t5318-commit-graph.sh | 18 ++++++++++++++++++\n 2 files changed, 46 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d7e408a99d..fcebd0925c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -243,6 +243,9 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n \tstruct commit *c;\n \tstruct object_id oid;\n \n+\tif (pos >= g->num_commits)\n+\t\tdie(\"invalid parent position %\"PRIu64, pos);\n+\n \thashcpy(oid.hash, g->chunk_oid_lookup + g->hash_len * pos);\n \tc = lookup_commit(&oid);\n \tif (!c)\n@@ -906,6 +909,7 @@ int verify_commit_graph(struct commit_graph *g)\n \n \tfor (i = 0; i < g->num_commits; i++) {\n \t\tstruct commit *graph_commit, *odb_commit;\n+\t\tstruct commit_list *graph_parents, *odb_parents;\n \n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n@@ -923,6 +927,30 @@ int verify_commit_graph(struct commit_graph *g)\n \t\t\t\t oid_to_hex(&cur_oid),\n \t\t\t\t oid_to_hex(get_commit_tree_oid(graph_commit)),\n \t\t\t\t oid_to_hex(get_commit_tree_oid(odb_commit)));\n+\n+\t\tgraph_parents = graph_commit->parents;\n+\t\todb_parents = odb_commit->parents;\n+\n+\t\twhile (graph_parents) {\n+\t\t\tif (odb_parents == NULL) {\n+\t\t\t\tgraph_report(\"commit-graph parent list for commit %s is too long\",\n+\t\t\t\t\t oid_to_hex(&cur_oid));\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (oidcmp(&graph_parents->item->object.oid, &odb_parents->item->object.oid))\n+\t\t\t\tgraph_report(\"commit-graph parent for %s is %s != %s\",\n+\t\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid),\n+\t\t\t\t\t oid_to_hex(&odb_parents->item->object.oid));\n+\n+\t\t\tgraph_parents = graph_parents->next;\n+\t\t\todb_parents = odb_parents->next;\n+\t\t}\n+\n+\t\tif (odb_parents != NULL)\n+\t\t\tgraph_report(\"commit-graph parent list for commit %s terminates early\",\n+\t\t\t\t oid_to_hex(&cur_oid));\n \t}\n \n \treturn verify_commit_graph_error;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex c0c1248eda..ec0964112a 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -269,6 +269,9 @@ GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8))\n GRAPH_BYTE_OID_LOOKUP_MISSING=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 4 + 10))\n GRAPH_COMMIT_DATA_OFFSET=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* $NUM_COMMITS))\n GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n+GRAPH_BYTE_COMMIT_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN))\n+GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4))\n+GRAPH_BYTE_COMMIT_WRONG_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -348,4 +351,19 @@ test_expect_success 'detect incorrect tree OID' '\n \t\t\"root tree OID for commit\"\n '\n \n+test_expect_success 'detect incorrect parent int-id' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_PARENT \"\\01\" \\\n+\t\t\"invalid parent\"\n+'\n+\n+test_expect_success 'detect extra parent int-id' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_EXTRA_PARENT \"\\00\" \\\n+\t\t\"is too long\"\n+'\n+\n+test_expect_success 'detect wrong parent' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_WRONG_PARENT \"\\01\" \\\n+\t\t\"commit-graph parent for\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349493","messageId":"20180606113611.87822-12-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 11/21] commit-graph: verify root tree OIDs","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:32Z","receivedAt":"2018-06-06T11:37:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The 'verify' subcommand must compare the commit content parsed from the\ncommit-graph against the content in the object database. Use\nlookup_commit() and parse_commit_in_graph_one() to parse the commits\nfrom the graph and compare against a commit that is loaded separately\nand parsed directly from the object database.\n\nAdd checks for the root tree OID.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 17 ++++++++++++++++-\n t/t5318-commit-graph.sh | 7 +++++++\n 2 files changed, 23 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 893cc2f346..d7e408a99d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -865,6 +865,8 @@ int verify_commit_graph(struct commit_graph *g)\n \t\treturn verify_commit_graph_error;\n \n \tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct commit *graph_commit;\n+\n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n \t\tif (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n@@ -882,6 +884,11 @@ int verify_commit_graph(struct commit_graph *g)\n \n \t\t\tcur_fanout_pos++;\n \t\t}\n+\n+\t\tgraph_commit = lookup_commit(&cur_oid);\n+\t\tif (!parse_commit_in_graph_one(g, graph_commit))\n+\t\t\tgraph_report(\"failed to parse %s from commit-graph\",\n+\t\t\t\t oid_to_hex(&cur_oid));\n \t}\n \n \twhile (cur_fanout_pos < 256) {\n@@ -898,16 +905,24 @@ int verify_commit_graph(struct commit_graph *g)\n \t\treturn verify_commit_graph_error;\n \n \tfor (i = 0; i < g->num_commits; i++) {\n-\t\tstruct commit *odb_commit;\n+\t\tstruct commit *graph_commit, *odb_commit;\n \n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n+\t\tgraph_commit = lookup_commit(&cur_oid);\n \t\todb_commit = (struct commit *)create_object(cur_oid.hash, alloc_commit_node());\n \t\tif (parse_commit_internal(odb_commit, 0, 0)) {\n \t\t\tgraph_report(\"failed to parse %s from object database\",\n \t\t\t\t oid_to_hex(&cur_oid));\n \t\t\tcontinue;\n \t\t}\n+\n+\t\tif (oidcmp(&get_commit_tree_in_graph_one(g, graph_commit)->object.oid,\n+\t\t\t get_commit_tree_oid(odb_commit)))\n+\t\t\tgraph_report(\"root tree OID for commit %s in commit-graph is %s != %s\",\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t oid_to_hex(get_commit_tree_oid(graph_commit)),\n+\t\t\t\t oid_to_hex(get_commit_tree_oid(odb_commit)));\n \t}\n \n \treturn verify_commit_graph_error;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex cf60e48496..c0c1248eda 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -267,6 +267,8 @@ GRAPH_BYTE_FANOUT2=$(($GRAPH_FANOUT_OFFSET + 4 \\* 255))\n GRAPH_OID_LOOKUP_OFFSET=$(($GRAPH_FANOUT_OFFSET + 4 \\* 256))\n GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8))\n GRAPH_BYTE_OID_LOOKUP_MISSING=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 4 + 10))\n+GRAPH_COMMIT_DATA_OFFSET=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* $NUM_COMMITS))\n+GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -341,4 +343,9 @@ test_expect_success 'detect OID not in object database' '\n \t\t\"from object database\"\n '\n \n+test_expect_success 'detect incorrect tree OID' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_TREE \"\\01\" \\\n+\t\t\"root tree OID for commit\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349494","messageId":"20180606113611.87822-2-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 01/21] commit-graph: UNLEAK before die()","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:23Z","receivedAt":"2018-06-06T11:37:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 37420ae0fd..f0875b8bf3 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -51,8 +51,11 @@ static int graph_read(int argc, const char **argv)\n \tgraph_name = get_commit_graph_filename(opts.obj_dir);\n \tgraph = load_commit_graph_one(graph_name);\n \n-\tif (!graph)\n+\tif (!graph) {\n+\t\tUNLEAK(graph_name);\n \t\tdie(\"graph file %s does not exist\", graph_name);\n+\t}\n+\n \tFREE_AND_NULL(graph_name);\n \n \tprintf(\"header: %08x %d %d %d %d\\n\",\n-- \n2.18.0.rc1\n\n"},{"id":"349495","messageId":"20180606113611.87822-3-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v5 02/21] commit-graph: fix GRAPH_MIN_SIZE","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-06T11:36:24Z","receivedAt":"2018-06-06T11:37:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The GRAPH_MIN_SIZE macro should be the smallest size of a parsable\ncommit-graph file. However, the minimum number of chunks was wrong.\nIt is possible to write a commit-graph file with zero commits, and\nthat violates this macro's value.\n\nRewrite the macro, and use extra macros to better explain the magic\nconstants.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex b63a1fc85e..f83f6d2373 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -35,10 +35,11 @@\n \n #define GRAPH_LAST_EDGE 0x80000000\n \n+#define GRAPH_HEADER_SIZE 8\n #define GRAPH_FANOUT_SIZE (4 * 256)\n #define GRAPH_CHUNKLOOKUP_WIDTH 12\n-#define GRAPH_MIN_SIZE (5 * GRAPH_CHUNKLOOKUP_WIDTH + GRAPH_FANOUT_SIZE + \\\n-\t\t\tGRAPH_OID_LEN + 8)\n+#define GRAPH_MIN_SIZE (GRAPH_HEADER_SIZE + 4 * GRAPH_CHUNKLOOKUP_WIDTH \\\n+\t\t\t+ GRAPH_FANOUT_SIZE + GRAPH_OID_LEN)\n \n char *get_commit_graph_filename(const char *obj_dir)\n {\n-- \n2.18.0.rc1\n\n"},{"id":"349498","messageId":"87h8mgyv0f.fsf@evledraar.gmail.com","threadId":"48647","inReplyTo":"20180606113611.87822-19-dstolee@microsoft.com","subject":"Re: [PATCH v5 18/21] commit-graph: use string-list API for input","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-06-06T12:11:44Z","receivedAt":"2018-06-06T12:11:52Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jun 06 2018, Derrick Stolee wrote:\n\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/commit-graph.c | 39 +++++++++++++--------------------------\n> commit-graph.c | 15 +++++++--------\n> commit-graph.h | 7 +++----\n> 3 files changed, 23 insertions(+), 38 deletions(-)\n\n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index 3079cde6f9..d8eb8278b3 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -118,13 +118,9 @@ static int graph_read(int argc, const char **argv)\n>\n> static int graph_write(int argc, const char **argv)\n> {\n> -\tconst char **pack_indexes = NULL;\n> -\tint packs_nr = 0;\n> -\tconst char **commit_hex = NULL;\n> -\tint commits_nr = 0;\n> -\tconst char **lines = NULL;\n> -\tint lines_nr = 0;\n> -\tint lines_alloc = 0;\n> +\tstruct string_list *pack_indexes = NULL;\n> +\tstruct string_list *commit_hex = NULL;\n> +\tstruct string_list lines;\n>\n> \tstatic struct option builtin_commit_graph_write_options[] = {\n> \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n> @@ -150,32 +146,23 @@ static int graph_write(int argc, const char **argv)\n>\n> \tif (opts.stdin_packs || opts.stdin_commits) {\n> \t\tstruct strbuf buf = STRBUF_INIT;\n> -\t\tlines_nr = 0;\n> -\t\tlines_alloc = 128;\n> -\t\tALLOC_ARRAY(lines, lines_alloc);\n> -\n> -\t\twhile (strbuf_getline(&buf, stdin) != EOF) {\n> -\t\t\tALLOC_GROW(lines, lines_nr + 1, lines_alloc);\n> -\t\t\tlines[lines_nr++] = strbuf_detach(&buf, NULL);\n> -\t\t}\n> -\n> -\t\tif (opts.stdin_packs) {\n> -\t\t\tpack_indexes = lines;\n> -\t\t\tpacks_nr = lines_nr;\n> -\t\t}\n> -\t\tif (opts.stdin_commits) {\n> -\t\t\tcommit_hex = lines;\n> -\t\t\tcommits_nr = lines_nr;\n> -\t\t}\n> +\t\tstring_list_init(&lines, 0);\n> +\n> +\t\twhile (strbuf_getline(&buf, stdin) != EOF)\n> +\t\t\tstring_list_append(&lines, strbuf_detach(&buf, NULL));\n> +\n> +\t\tif (opts.stdin_packs)\n> +\t\t\tpack_indexes = &lines;\n> +\t\tif (opts.stdin_commits)\n> +\t\t\tcommit_hex = &lines;\n> \t}\n>\n> \twrite_commit_graph(opts.obj_dir,\n> \t\t\t pack_indexes,\n> -\t\t\t packs_nr,\n> \t\t\t commit_hex,\n> -\t\t\t commits_nr,\n> \t\t\t opts.append);\n>\n> +\tstring_list_clear(&lines, 0);\n> \treturn 0;\n> }\n\nThis results in an invalid free() & segfault because you're freeing\n&lines which may not have been allocated by string_list_init().\n\nMonkeypatch on top which I used to fix it:\n\n diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n index 76423b3fa5..c7eb68aa3a 100644\n --- a/builtin/commit-graph.c\n +++ b/builtin/commit-graph.c\n @@ -122,6 +122,7 @@ static int graph_write(int argc, const char **argv)\n struct string_list *pack_indexes = NULL;\n struct string_list *commit_hex = NULL;\n struct string_list lines;\n + int free_lines = 0;\n\n static struct option builtin_commit_graph_write_options[] = {\n OPT_STRING(0, \"object-dir\", &opts.obj_dir,\n @@ -155,6 +156,7 @@ static int graph_write(int argc, const char **argv)\n if (opts.stdin_packs || opts.stdin_commits) {\n struct strbuf buf = STRBUF_INIT;\n string_list_init(&lines, 0);\n + free_lines = 1;\n\n while (strbuf_getline(&buf, stdin) != EOF)\n string_list_append(&lines, strbuf_detach(&buf, NULL));\n @@ -170,7 +172,8 @@ static int graph_write(int argc, const char **argv)\n commit_hex,\n opts.append);\n\n - string_list_clear(&lines, 0);\n + if (free_lines)\n + string_list_clear(&lines, 0);\n return 0;\n }\n\nBut probably having a pointer to the struct which is NULL etc. is\nbetter.\n"},{"id":"349499","messageId":"aa4c3a93-0c63-f424-db0b-0e451bbaa91e@gmail.com","threadId":"48647","inReplyTo":"87h8mgyv0f.fsf@evledraar.gmail.com","subject":"Re: [PATCH v5 18/21] commit-graph: use string-list API for input","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-06T12:15:04Z","receivedAt":"2018-06-06T12:15:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/6/2018 8:11 AM, Ævar Arnfjörð Bjarmason wrote:\n> On Wed, Jun 06 2018, Derrick Stolee wrote:\n>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> builtin/commit-graph.c | 39 +++++++++++++--------------------------\n>> commit-graph.c | 15 +++++++--------\n>> commit-graph.h | 7 +++----\n>> 3 files changed, 23 insertions(+), 38 deletions(-)\n>> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n>> index 3079cde6f9..d8eb8278b3 100644\n>> --- a/builtin/commit-graph.c\n>> +++ b/builtin/commit-graph.c\n>> @@ -118,13 +118,9 @@ static int graph_read(int argc, const char **argv)\n>>\n>> static int graph_write(int argc, const char **argv)\n>> {\n>> -\tconst char **pack_indexes = NULL;\n>> -\tint packs_nr = 0;\n>> -\tconst char **commit_hex = NULL;\n>> -\tint commits_nr = 0;\n>> -\tconst char **lines = NULL;\n>> -\tint lines_nr = 0;\n>> -\tint lines_alloc = 0;\n>> +\tstruct string_list *pack_indexes = NULL;\n>> +\tstruct string_list *commit_hex = NULL;\n>> +\tstruct string_list lines;\n>>\n>> \tstatic struct option builtin_commit_graph_write_options[] = {\n>> \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n>> @@ -150,32 +146,23 @@ static int graph_write(int argc, const char **argv)\n>>\n>> \tif (opts.stdin_packs || opts.stdin_commits) {\n>> \t\tstruct strbuf buf = STRBUF_INIT;\n>> -\t\tlines_nr = 0;\n>> -\t\tlines_alloc = 128;\n>> -\t\tALLOC_ARRAY(lines, lines_alloc);\n>> -\n>> -\t\twhile (strbuf_getline(&buf, stdin) != EOF) {\n>> -\t\t\tALLOC_GROW(lines, lines_nr + 1, lines_alloc);\n>> -\t\t\tlines[lines_nr++] = strbuf_detach(&buf, NULL);\n>> -\t\t}\n>> -\n>> -\t\tif (opts.stdin_packs) {\n>> -\t\t\tpack_indexes = lines;\n>> -\t\t\tpacks_nr = lines_nr;\n>> -\t\t}\n>> -\t\tif (opts.stdin_commits) {\n>> -\t\t\tcommit_hex = lines;\n>> -\t\t\tcommits_nr = lines_nr;\n>> -\t\t}\n>> +\t\tstring_list_init(&lines, 0);\n>> +\n>> +\t\twhile (strbuf_getline(&buf, stdin) != EOF)\n>> +\t\t\tstring_list_append(&lines, strbuf_detach(&buf, NULL));\n>> +\n>> +\t\tif (opts.stdin_packs)\n>> +\t\t\tpack_indexes = &lines;\n>> +\t\tif (opts.stdin_commits)\n>> +\t\t\tcommit_hex = &lines;\n>> \t}\n>>\n>> \twrite_commit_graph(opts.obj_dir,\n>> \t\t\t pack_indexes,\n>> -\t\t\t packs_nr,\n>> \t\t\t commit_hex,\n>> -\t\t\t commits_nr,\n>> \t\t\t opts.append);\n>>\n>> +\tstring_list_clear(&lines, 0);\n>> \treturn 0;\n>> }\n> This results in an invalid free() & segfault because you're freeing\n> &lines which may not have been allocated by string_list_init().\n\nGood point. Did my tests not catch this? (seems it requires calling `git \ncommit-graph write` with no `--stdin-packs` or `--stdin-commits`).\n\n>\n> Monkeypatch on top which I used to fix it:\n>\n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index 76423b3fa5..c7eb68aa3a 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -122,6 +122,7 @@ static int graph_write(int argc, const char **argv)\n> struct string_list *pack_indexes = NULL;\n> struct string_list *commit_hex = NULL;\n> struct string_list lines;\n> + int free_lines = 0;\n>\n> static struct option builtin_commit_graph_write_options[] = {\n> OPT_STRING(0, \"object-dir\", &opts.obj_dir,\n> @@ -155,6 +156,7 @@ static int graph_write(int argc, const char **argv)\n> if (opts.stdin_packs || opts.stdin_commits) {\n> struct strbuf buf = STRBUF_INIT;\n> string_list_init(&lines, 0);\n> + free_lines = 1;\n>\n> while (strbuf_getline(&buf, stdin) != EOF)\n> string_list_append(&lines, strbuf_detach(&buf, NULL));\n> @@ -170,7 +172,8 @@ static int graph_write(int argc, const char **argv)\n> commit_hex,\n> opts.append);\n>\n> - string_list_clear(&lines, 0);\n> + if (free_lines)\n> + string_list_clear(&lines, 0);\n> return 0;\n> }\n>\n> But probably having a pointer to the struct which is NULL etc. is\n> better.\n\nWouldn't the easiest fix be to call `string_list_init(&lines, 0)` \noutside of any conditional?\n\nThanks,\n-Stolee\n"},{"id":"349501","messageId":"87fu20yukj.fsf@evledraar.gmail.com","threadId":"48647","inReplyTo":"20180606113611.87822-9-dstolee@microsoft.com","subject":"Re: [PATCH v5 08/21] commit-graph: verify required chunks are present","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-06-06T12:21:16Z","receivedAt":"2018-06-06T12:21:22Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jun 06 2018, Derrick Stolee wrote:\n\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index c0c1ff09b9..846396665e 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -249,6 +249,15 @@ test_expect_success 'git commit-graph verify' '\n>\n> GRAPH_BYTE_VERSION=4\n> GRAPH_BYTE_HASH=5\n> +GRAPH_BYTE_CHUNK_COUNT=6\n> +GRAPH_CHUNK_LOOKUP_OFFSET=8\n> +GRAPH_CHUNK_LOOKUP_WIDTH=12\n> +GRAPH_CHUNK_LOOKUP_ROWS=5\n> +GRAPH_BYTE_OID_FANOUT_ID=$GRAPH_CHUNK_LOOKUP_OFFSET\n> +GRAPH_BYTE_OID_LOOKUP_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n> +\t\t\t 1 \\* $GRAPH_CHUNK_LOOKUP_WIDTH))\n\nOn GNU bash, version 4.1.2(2)-release (x86_64-redhat-linux-gnu) this\nemits:\n\n./t5318-commit-graph.sh: line 285: 8 + 1 \\* 12: syntax error: invalid arithmetic operator (error token is \"\\* 12\")\n\nThe same goes for the rest of this \"\\*\" within $((...)) in this file, it\nshould just be \"*\"..\n\n> +GRAPH_BYTE_COMMIT_DATA_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n> +\t\t\t 2 \\* $GRAPH_CHUNK_LOOKUP_WIDTH))\n"},{"id":"349502","messageId":"87efhkyubq.fsf@evledraar.gmail.com","threadId":"48647","inReplyTo":"aa4c3a93-0c63-f424-db0b-0e451bbaa91e@gmail.com","subject":"Re: [PATCH v5 18/21] commit-graph: use string-list API for input","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-06-06T12:26:33Z","receivedAt":"2018-06-06T12:26:38Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jun 06 2018, Derrick Stolee wrote:\n\n> On 6/6/2018 8:11 AM, Ævar Arnfjörð Bjarmason wrote:\n>> On Wed, Jun 06 2018, Derrick Stolee wrote:\n>>\n>>> Signed-off-by: Derrick Stolee \n>>> ---\n>>> builtin/commit-graph.c | 39 +++++++++++++--------------------------\n>>> commit-graph.c | 15 +++++++--------\n>>> commit-graph.h | 7 +++----\n>>> 3 files changed, 23 insertions(+), 38 deletions(-)\n>>> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n>>> index 3079cde6f9..d8eb8278b3 100644\n>>> --- a/builtin/commit-graph.c\n>>> +++ b/builtin/commit-graph.c\n>>> @@ -118,13 +118,9 @@ static int graph_read(int argc, const char **argv)\n>>>\n>>> static int graph_write(int argc, const char **argv)\n>>> {\n>>> -\tconst char **pack_indexes = NULL;\n>>> -\tint packs_nr = 0;\n>>> -\tconst char **commit_hex = NULL;\n>>> -\tint commits_nr = 0;\n>>> -\tconst char **lines = NULL;\n>>> -\tint lines_nr = 0;\n>>> -\tint lines_alloc = 0;\n>>> +\tstruct string_list *pack_indexes = NULL;\n>>> +\tstruct string_list *commit_hex = NULL;\n>>> +\tstruct string_list lines;\n>>>\n>>> \tstatic struct option builtin_commit_graph_write_options[] = {\n>>> \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n>>> @@ -150,32 +146,23 @@ static int graph_write(int argc, const char **argv)\n>>>\n>>> \tif (opts.stdin_packs || opts.stdin_commits) {\n>>> \t\tstruct strbuf buf = STRBUF_INIT;\n>>> -\t\tlines_nr = 0;\n>>> -\t\tlines_alloc = 128;\n>>> -\t\tALLOC_ARRAY(lines, lines_alloc);\n>>> -\n>>> -\t\twhile (strbuf_getline(&buf, stdin) != EOF) {\n>>> -\t\t\tALLOC_GROW(lines, lines_nr + 1, lines_alloc);\n>>> -\t\t\tlines[lines_nr++] = strbuf_detach(&buf, NULL);\n>>> -\t\t}\n>>> -\n>>> -\t\tif (opts.stdin_packs) {\n>>> -\t\t\tpack_indexes = lines;\n>>> -\t\t\tpacks_nr = lines_nr;\n>>> -\t\t}\n>>> -\t\tif (opts.stdin_commits) {\n>>> -\t\t\tcommit_hex = lines;\n>>> -\t\t\tcommits_nr = lines_nr;\n>>> -\t\t}\n>>> +\t\tstring_list_init(&lines, 0);\n>>> +\n>>> +\t\twhile (strbuf_getline(&buf, stdin) != EOF)\n>>> +\t\t\tstring_list_append(&lines, strbuf_detach(&buf, NULL));\n>>> +\n>>> +\t\tif (opts.stdin_packs)\n>>> +\t\t\tpack_indexes = &lines;\n>>> +\t\tif (opts.stdin_commits)\n>>> +\t\t\tcommit_hex = &lines;\n>>> \t}\n>>>\n>>> \twrite_commit_graph(opts.obj_dir,\n>>> \t\t\t pack_indexes,\n>>> -\t\t\t packs_nr,\n>>> \t\t\t commit_hex,\n>>> -\t\t\t commits_nr,\n>>> \t\t\t opts.append);\n>>>\n>>> +\tstring_list_clear(&lines, 0);\n>>> \treturn 0;\n>>> }\n>> This results in an invalid free() & segfault because you're freeing\n>> &lines which may not have been allocated by string_list_init().\n>\n> Good point. Did my tests not catch this? (seems it requires calling\n> `git commit-graph write` with no `--stdin-packs` or\n> `--stdin-commits`).\n\nMost of your tests (t5318-commit-graph.sh) segfaulted, but presumably\nyou're on a more forgiving compiler/platform/options. I compiled with\n-O0 -g on clang 4.0.1-8 + Debian testing.\n\n>>\n>> Monkeypatch on top which I used to fix it:\n>>\n>> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n>> index 76423b3fa5..c7eb68aa3a 100644\n>> --- a/builtin/commit-graph.c\n>> +++ b/builtin/commit-graph.c\n>> @@ -122,6 +122,7 @@ static int graph_write(int argc, const char **argv)\n>> struct string_list *pack_indexes = NULL;\n>> struct string_list *commit_hex = NULL;\n>> struct string_list lines;\n>> + int free_lines = 0;\n>>\n>> static struct option builtin_commit_graph_write_options[] = {\n>> OPT_STRING(0, \"object-dir\", &opts.obj_dir,\n>> @@ -155,6 +156,7 @@ static int graph_write(int argc, const char **argv)\n>> if (opts.stdin_packs || opts.stdin_commits) {\n>> struct strbuf buf = STRBUF_INIT;\n>> string_list_init(&lines, 0);\n>> + free_lines = 1;\n>>\n>> while (strbuf_getline(&buf, stdin) != EOF)\n>> string_list_append(&lines, strbuf_detach(&buf, NULL));\n>> @@ -170,7 +172,8 @@ static int graph_write(int argc, const char **argv)\n>> commit_hex,\n>> opts.append);\n>>\n>> - string_list_clear(&lines, 0);\n>> + if (free_lines)\n>> + string_list_clear(&lines, 0);\n>> return 0;\n>> }\n>>\n>> But probably having a pointer to the struct which is NULL etc. is\n>> better.\n>\n> Wouldn't the easiest fix be to call `string_list_init(&lines, 0)`\n> outside of any conditional?\n\nSure that works too. We'd be doing the init when we don't need it, but\nit's not like this part is performance critical or anything...\n"},{"id":"349503","messageId":"548f9285-da68-478c-1423-a9b9b5021d97@gmail.com","threadId":"48647","inReplyTo":"87efhkyubq.fsf@evledraar.gmail.com","subject":"Re: [PATCH v5 18/21] commit-graph: use string-list API for input","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-06T12:45:00Z","receivedAt":"2018-06-06T12:45:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/6/2018 8:26 AM, Ævar Arnfjörð Bjarmason wrote:\n> On Wed, Jun 06 2018, Derrick Stolee wrote:\n>\n>> On 6/6/2018 8:11 AM, Ævar Arnfjörð Bjarmason wrote:\n>>> On Wed, Jun 06 2018, Derrick Stolee wrote:\n>>>\n>>>> Signed-off-by: Derrick Stolee \n>>>> ---\n>>>> builtin/commit-graph.c | 39 +++++++++++++--------------------------\n>>>> commit-graph.c | 15 +++++++--------\n>>>> commit-graph.h | 7 +++----\n>>>> 3 files changed, 23 insertions(+), 38 deletions(-)\n>>>> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n>>>> index 3079cde6f9..d8eb8278b3 100644\n>>>> --- a/builtin/commit-graph.c\n>>>> +++ b/builtin/commit-graph.c\n>>>> @@ -118,13 +118,9 @@ static int graph_read(int argc, const char **argv)\n>>>>\n>>>> static int graph_write(int argc, const char **argv)\n>>>> {\n>>>> -\tconst char **pack_indexes = NULL;\n>>>> -\tint packs_nr = 0;\n>>>> -\tconst char **commit_hex = NULL;\n>>>> -\tint commits_nr = 0;\n>>>> -\tconst char **lines = NULL;\n>>>> -\tint lines_nr = 0;\n>>>> -\tint lines_alloc = 0;\n>>>> +\tstruct string_list *pack_indexes = NULL;\n>>>> +\tstruct string_list *commit_hex = NULL;\n>>>> +\tstruct string_list lines;\n>>>>\n>>>> \tstatic struct option builtin_commit_graph_write_options[] = {\n>>>> \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n>>>> @@ -150,32 +146,23 @@ static int graph_write(int argc, const char **argv)\n>>>>\n>>>> \tif (opts.stdin_packs || opts.stdin_commits) {\n>>>> \t\tstruct strbuf buf = STRBUF_INIT;\n>>>> -\t\tlines_nr = 0;\n>>>> -\t\tlines_alloc = 128;\n>>>> -\t\tALLOC_ARRAY(lines, lines_alloc);\n>>>> -\n>>>> -\t\twhile (strbuf_getline(&buf, stdin) != EOF) {\n>>>> -\t\t\tALLOC_GROW(lines, lines_nr + 1, lines_alloc);\n>>>> -\t\t\tlines[lines_nr++] = strbuf_detach(&buf, NULL);\n>>>> -\t\t}\n>>>> -\n>>>> -\t\tif (opts.stdin_packs) {\n>>>> -\t\t\tpack_indexes = lines;\n>>>> -\t\t\tpacks_nr = lines_nr;\n>>>> -\t\t}\n>>>> -\t\tif (opts.stdin_commits) {\n>>>> -\t\t\tcommit_hex = lines;\n>>>> -\t\t\tcommits_nr = lines_nr;\n>>>> -\t\t}\n>>>> +\t\tstring_list_init(&lines, 0);\n>>>> +\n>>>> +\t\twhile (strbuf_getline(&buf, stdin) != EOF)\n>>>> +\t\t\tstring_list_append(&lines, strbuf_detach(&buf, NULL));\n>>>> +\n>>>> +\t\tif (opts.stdin_packs)\n>>>> +\t\t\tpack_indexes = &lines;\n>>>> +\t\tif (opts.stdin_commits)\n>>>> +\t\t\tcommit_hex = &lines;\n>>>> \t}\n>>>>\n>>>> \twrite_commit_graph(opts.obj_dir,\n>>>> \t\t\t pack_indexes,\n>>>> -\t\t\t packs_nr,\n>>>> \t\t\t commit_hex,\n>>>> -\t\t\t commits_nr,\n>>>> \t\t\t opts.append);\n>>>>\n>>>> +\tstring_list_clear(&lines, 0);\n>>>> \treturn 0;\n>>>> }\n>>> This results in an invalid free() & segfault because you're freeing\n>>> &lines which may not have been allocated by string_list_init().\n>> Good point. Did my tests not catch this? (seems it requires calling\n>> `git commit-graph write` with no `--stdin-packs` or\n>> `--stdin-commits`).\n> Most of your tests (t5318-commit-graph.sh) segfaulted, but presumably\n> you're on a more forgiving compiler/platform/options. I compiled with\n> -O0 -g on clang 4.0.1-8 + Debian testing.\n\nI appreciate the extra platform testing. I'm using GCC on Ubuntu (gcc \n(Ubuntu 7.3.0-16ubuntu3) 7.3.0).\n\n>\n>>> Monkeypatch on top which I used to fix it:\n>>>\n>>> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n>>> index 76423b3fa5..c7eb68aa3a 100644\n>>> --- a/builtin/commit-graph.c\n>>> +++ b/builtin/commit-graph.c\n>>> @@ -122,6 +122,7 @@ static int graph_write(int argc, const char **argv)\n>>> struct string_list *pack_indexes = NULL;\n>>> struct string_list *commit_hex = NULL;\n>>> struct string_list lines;\n>>> + int free_lines = 0;\n>>>\n>>> static struct option builtin_commit_graph_write_options[] = {\n>>> OPT_STRING(0, \"object-dir\", &opts.obj_dir,\n>>> @@ -155,6 +156,7 @@ static int graph_write(int argc, const char **argv)\n>>> if (opts.stdin_packs || opts.stdin_commits) {\n>>> struct strbuf buf = STRBUF_INIT;\n>>> string_list_init(&lines, 0);\n>>> + free_lines = 1;\n>>>\n>>> while (strbuf_getline(&buf, stdin) != EOF)\n>>> string_list_append(&lines, strbuf_detach(&buf, NULL));\n>>> @@ -170,7 +172,8 @@ static int graph_write(int argc, const char **argv)\n>>> commit_hex,\n>>> opts.append);\n>>>\n>>> - string_list_clear(&lines, 0);\n>>> + if (free_lines)\n>>> + string_list_clear(&lines, 0);\n>>> return 0;\n>>> }\n>>>\n>>> But probably having a pointer to the struct which is NULL etc. is\n>>> better.\n>> Wouldn't the easiest fix be to call `string_list_init(&lines, 0)`\n>> outside of any conditional?\n> Sure that works too. We'd be doing the init when we don't need it, but\n> it's not like this part is performance critical or anything...\n\n"},{"id":"349692","messageId":"4c5982b2-15a0-9172-c115-e78afe0b4daa@gmail.com","threadId":"48647","inReplyTo":"548f9285-da68-478c-1423-a9b9b5021d97@gmail.com","subject":"Re: [PATCH v5 18/21] commit-graph: use string-list API for input","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-08T12:48:32Z","receivedAt":"2018-06-08T12:48:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/6/2018 8:45 AM, Derrick Stolee wrote:\n> On 6/6/2018 8:26 AM, Ævar Arnfjörð Bjarmason wrote:\n>> On Wed, Jun 06 2018, Derrick Stolee wrote:\n>>\n>>> On 6/6/2018 8:11 AM, Ævar Arnfjörð Bjarmason wrote:\n>>>> On Wed, Jun 06 2018, Derrick Stolee wrote:\n>>>>\n>>>>> Signed-off-by: Derrick Stolee \n>>>>>\n>>>>> +    string_list_clear(&lines, 0);\n>>>>>        return 0;\n>>>>>    }\n>>>> This results in an invalid free() & segfault because you're freeing\n>>>> &lines which may not have been allocated by string_list_init().\n>>> Good point. Did my tests not catch this? (seems it requires calling\n>>> `git commit-graph write` with no `--stdin-packs` or\n>>> `--stdin-commits`).\n>> Most of your tests (t5318-commit-graph.sh) segfaulted, but presumably\n>> you're on a more forgiving compiler/platform/options. I compiled with\n>> -O0 -g on clang 4.0.1-8 + Debian testing.\n>\n> I appreciate the extra platform testing. I'm using GCC on Ubuntu (gcc \n> (Ubuntu 7.3.0-16ubuntu3) 7.3.0).\n\nWhile the errors didn't repro on my box, they did on the VSTS Linux \nbuild machines [1]. I created an internal PR just so I could run those \ntests (and on our OSX agents which use clang) and so I can verify I \nfixed the situation. I'll send a v6 soon so Junio only picks up a \nversion that succeeds in tests.\n\nThanks,\n-Stolee\n\n[1] \nhttps://github.com/Microsoft/vsts-agent-docker/blob/master/ubuntu/16.04/standard/Dockerfile\n     What's installed on a VSTS Linux build agent\n"},{"id":"349694","messageId":"20180608135548.216405-1-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180606113611.87822-1-dstolee@microsoft.com","subject":"[PATCH v6 00/21] Integrate commit-graph into 'fsck' and 'gc'","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:12Z","receivedAt":"2018-06-08T13:56:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Sorry for the fast rerolls, but Ævar found some runtime issues with the\nstring-list use and '\\*' as multiplication on some platforms. Thus, v4\nand v5 are broken. I did get a repro of those issues using VSTS Linux\nbuild servers and checked that they are fixed in this version.\n\nThis version depends on 'next' and sb/object-store-alloc. To accommodate\nthe repository arguments added by sb/object-store-alloc, the following\ndiff occurs from the previous patch:\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 76423b3fa5..c7d0db5ab4 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -3,6 +3,7 @@\n #include \"dir.h\"\n #include \"lockfile.h\"\n #include \"parse-options.h\"\n+#include \"repository.h\"\n #include \"commit-graph.h\"\n\n static char const * const builtin_commit_graph_usage[] = {\n@@ -63,7 +64,7 @@ static int graph_verify(int argc, const char **argv)\n if (!graph)\n return 0;\n\n- return verify_commit_graph(graph);\n+ return verify_commit_graph(the_repository, graph);\n }\n\n static int graph_read(int argc, const char **argv)\n@@ -152,9 +153,9 @@ static int graph_write(int argc, const char **argv)\n return 0;\n }\n\n+ string_list_init(&lines, 0);\n if (opts.stdin_packs || opts.stdin_commits) {\n struct strbuf buf = STRBUF_INIT;\n- string_list_init(&lines, 0);\n\n while (strbuf_getline(&buf, stdin) != EOF)\n string_list_append(&lines, strbuf_detach(&buf, NULL));\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0d5adc8035..adf54e3fe7 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -12,6 +12,7 @@\n #include \"sha1-lookup.h\"\n #include \"commit-graph.h\"\n #include \"object-store.h\"\n+#include \"alloc.h\"\n\n #define GRAPH_SIGNATURE 0x43475048 /* \"CGPH\" */\n #define GRAPH_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n@@ -866,7 +867,7 @@ static void graph_report(const char *fmt, ...)\n #define GENERATION_ZERO_EXISTS 1\n #define GENERATION_NUMBER_EXISTS 2\n\n-int verify_commit_graph(struct commit_graph *g)\n+int verify_commit_graph(struct repository *r, struct commit_graph *g)\n {\n uint32_t i, cur_fanout_pos = 0;\n struct object_id prev_oid, cur_oid, checksum;\n@@ -948,7 +949,7 @@ int verify_commit_graph(struct commit_graph *g)\n hashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n\n graph_commit = lookup_commit(&cur_oid);\n- odb_commit = (struct commit *)create_object(cur_oid.hash, alloc_commit_node());\n+ odb_commit = (struct commit *)create_object(r, cur_oid.hash, alloc_commit_node(r));\n if (parse_commit_internal(odb_commit, 0, 0)) {\n graph_report(\"failed to parse %s from object database\",\n oid_to_hex(&cur_oid));\ndiff --git a/commit-graph.h b/commit-graph.h\nindex ee20f5e280..506cb45fb1 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -2,6 +2,7 @@\n #define COMMIT_GRAPH_H\n\n #include \"git-compat-util.h\"\n+#include \"repository.h\"\n #include \"string-list.h\"\n\n char *get_commit_graph_filename(const char *obj_dir);\n@@ -53,6 +54,6 @@ void write_commit_graph(const char *obj_dir,\n struct string_list *commit_hex,\n int append);\n\n-int verify_commit_graph(struct commit_graph *g);\n+int verify_commit_graph(struct repository *r, struct commit_graph *g);\n\n #endif\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex b24e8b6689..9a0661983c 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -33,8 +33,8 @@ test_expect_success 'create commits and repack' '\n '\n\n graph_git_two_modes() {\n- git -c core.commitGraph=true $1 >output\n- git -c core.commitGraph=false $1 >expect\n+ git -c core.graph=true $1 >output\n+ git -c core.graph=false $1 >expect\n test_cmp output expect\n }\n\n@@ -282,17 +282,17 @@ GRAPH_CHUNK_LOOKUP_WIDTH=12\n GRAPH_CHUNK_LOOKUP_ROWS=5\n GRAPH_BYTE_OID_FANOUT_ID=$GRAPH_CHUNK_LOOKUP_OFFSET\n GRAPH_BYTE_OID_LOOKUP_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n- 1 \\* $GRAPH_CHUNK_LOOKUP_WIDTH))\n+ 1 * $GRAPH_CHUNK_LOOKUP_WIDTH))\n GRAPH_BYTE_COMMIT_DATA_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n- 2 \\* $GRAPH_CHUNK_LOOKUP_WIDTH))\n+ 2 * $GRAPH_CHUNK_LOOKUP_WIDTH))\n GRAPH_FANOUT_OFFSET=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n- $GRAPH_CHUNK_LOOKUP_WIDTH \\* $GRAPH_CHUNK_LOOKUP_ROWS))\n-GRAPH_BYTE_FANOUT1=$(($GRAPH_FANOUT_OFFSET + 4 \\* 4))\n-GRAPH_BYTE_FANOUT2=$(($GRAPH_FANOUT_OFFSET + 4 \\* 255))\n-GRAPH_OID_LOOKUP_OFFSET=$(($GRAPH_FANOUT_OFFSET + 4 \\* 256))\n-GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8))\n-GRAPH_BYTE_OID_LOOKUP_MISSING=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 4 + 10))\n-GRAPH_COMMIT_DATA_OFFSET=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* $NUM_COMMITS))\n+ $GRAPH_CHUNK_LOOKUP_WIDTH * $GRAPH_CHUNK_LOOKUP_ROWS))\n+GRAPH_BYTE_FANOUT1=$(($GRAPH_FANOUT_OFFSET + 4 * 4))\n+GRAPH_BYTE_FANOUT2=$(($GRAPH_FANOUT_OFFSET + 4 * 255))\n+GRAPH_OID_LOOKUP_OFFSET=$(($GRAPH_FANOUT_OFFSET + 4 * 256))\n+GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 8))\n+GRAPH_BYTE_OID_LOOKUP_MISSING=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 4 + 10))\n+GRAPH_COMMIT_DATA_OFFSET=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * $NUM_COMMITS))\n GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n GRAPH_BYTE_COMMIT_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN))\n GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4))\n@@ -301,9 +301,9 @@ GRAPH_BYTE_COMMIT_GENERATION=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 11))\n GRAPH_BYTE_COMMIT_DATE=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 12))\n GRAPH_COMMIT_DATA_WIDTH=$(($HASH_LEN + 16))\n GRAPH_OCTOPUS_DATA_OFFSET=$(($GRAPH_COMMIT_DATA_OFFSET + \\\n- $GRAPH_COMMIT_DATA_WIDTH \\* $NUM_COMMITS))\n+ $GRAPH_COMMIT_DATA_WIDTH * $NUM_COMMITS))\n GRAPH_BYTE_OCTOPUS=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4))\n-GRAPH_BYTE_FOOTER=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4 \\* $NUM_OCTOPUS_EDGES))\n+GRAPH_BYTE_FOOTER=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4 * $NUM_OCTOPUS_EDGES))\n\n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n\n---\n\nDerrick Stolee (21):\n commit-graph: UNLEAK before die()\n commit-graph: fix GRAPH_MIN_SIZE\n commit-graph: parse commit from chosen graph\n commit: force commit to parse from object database\n commit-graph: load a root tree from specific graph\n commit-graph: add 'verify' subcommand\n commit-graph: verify catches corrupt signature\n commit-graph: verify required chunks are present\n commit-graph: verify corrupt OID fanout and lookup\n commit-graph: verify objects exist\n commit-graph: verify root tree OIDs\n commit-graph: verify parent list\n commit-graph: verify generation number\n commit-graph: verify commit date\n commit-graph: test for corrupted octopus edge\n commit-graph: verify contents match checksum\n fsck: verify commit-graph\n commit-graph: use string-list API for input\n commit-graph: add '--reachable' option\n gc: automatically write commit-graph files\n commit-graph: update design document\n\n Documentation/config.txt | 10 +-\n Documentation/git-commit-graph.txt | 14 +-\n Documentation/git-fsck.txt | 3 +\n Documentation/git-gc.txt | 4 +\n Documentation/technical/commit-graph.txt | 22 --\n builtin/commit-graph.c | 99 ++++++---\n builtin/fsck.c | 21 ++\n builtin/gc.c | 6 +\n commit-graph.c | 249 +++++++++++++++++++++--\n commit-graph.h | 11 +-\n commit.c | 9 +-\n commit.h | 1 +\n t/t5318-commit-graph.sh | 201 ++++++++++++++++++\n 13 files changed, 572 insertions(+), 78 deletions(-)\n\n-- \n2.18.0.rc1\n\n"},{"id":"349695","messageId":"20180608135548.216405-3-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 02/21] commit-graph: fix GRAPH_MIN_SIZE","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:14Z","receivedAt":"2018-06-08T13:56:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The GRAPH_MIN_SIZE macro should be the smallest size of a parsable\ncommit-graph file. However, the minimum number of chunks was wrong.\nIt is possible to write a commit-graph file with zero commits, and\nthat violates this macro's value.\n\nRewrite the macro, and use extra macros to better explain the magic\nconstants.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex b63a1fc85e..f83f6d2373 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -35,10 +35,11 @@\n \n #define GRAPH_LAST_EDGE 0x80000000\n \n+#define GRAPH_HEADER_SIZE 8\n #define GRAPH_FANOUT_SIZE (4 * 256)\n #define GRAPH_CHUNKLOOKUP_WIDTH 12\n-#define GRAPH_MIN_SIZE (5 * GRAPH_CHUNKLOOKUP_WIDTH + GRAPH_FANOUT_SIZE + \\\n-\t\t\tGRAPH_OID_LEN + 8)\n+#define GRAPH_MIN_SIZE (GRAPH_HEADER_SIZE + 4 * GRAPH_CHUNKLOOKUP_WIDTH \\\n+\t\t\t+ GRAPH_FANOUT_SIZE + GRAPH_OID_LEN)\n \n char *get_commit_graph_filename(const char *obj_dir)\n {\n-- \n2.18.0.rc1\n\n"},{"id":"349696","messageId":"20180608135548.216405-4-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 03/21] commit-graph: parse commit from chosen graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:14Z","receivedAt":"2018-06-08T13:56:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Before verifying a commit-graph file against the object database, we\nneed to parse all commits from the given commit-graph file. Create\nparse_commit_in_graph_one() to target a given struct commit_graph.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 18 +++++++++++++++---\n 1 file changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f83f6d2373..e77b19971d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -314,7 +314,7 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \t}\n }\n \n-int parse_commit_in_graph(struct commit *item)\n+static int parse_commit_in_graph_one(struct commit_graph *g, struct commit *item)\n {\n \tuint32_t pos;\n \n@@ -322,9 +322,21 @@ int parse_commit_in_graph(struct commit *item)\n \t\treturn 0;\n \tif (item->object.parsed)\n \t\treturn 1;\n+\n+\tif (find_commit_in_graph(item, g, &pos))\n+\t\treturn fill_commit_in_graph(item, g, pos);\n+\n+\treturn 0;\n+}\n+\n+int parse_commit_in_graph(struct commit *item)\n+{\n+\tif (!core_commit_graph)\n+\t\treturn 0;\n+\n \tprepare_commit_graph();\n-\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n-\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n+\tif (commit_graph)\n+\t\treturn parse_commit_in_graph_one(commit_graph, item);\n \treturn 0;\n }\n \n-- \n2.18.0.rc1\n\n"},{"id":"349697","messageId":"20180608135548.216405-6-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 05/21] commit-graph: load a root tree from specific graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:16Z","receivedAt":"2018-06-08T13:56:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When lazy-loading a tree for a commit, it will be important to select\nthe tree from a specific struct commit_graph. Create a new method that\nspecifies the commit-graph file and use that in\nget_commit_tree_in_graph().\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 12 +++++++++---\n 1 file changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex e77b19971d..9e228d3bb5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -362,14 +362,20 @@ static struct tree *load_tree_for_commit(struct commit_graph *g, struct commit *\n \treturn c->maybe_tree;\n }\n \n-struct tree *get_commit_tree_in_graph(const struct commit *c)\n+static struct tree *get_commit_tree_in_graph_one(struct commit_graph *g,\n+\t\t\t\t\t\t const struct commit *c)\n {\n \tif (c->maybe_tree)\n \t\treturn c->maybe_tree;\n \tif (c->graph_pos == COMMIT_NOT_FROM_GRAPH)\n-\t\tBUG(\"get_commit_tree_in_graph called from non-commit-graph commit\");\n+\t\tBUG(\"get_commit_tree_in_graph_one called from non-commit-graph commit\");\n+\n+\treturn load_tree_for_commit(g, (struct commit *)c);\n+}\n \n-\treturn load_tree_for_commit(commit_graph, (struct commit *)c);\n+struct tree *get_commit_tree_in_graph(const struct commit *c)\n+{\n+\treturn get_commit_tree_in_graph_one(commit_graph, c);\n }\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n-- \n2.18.0.rc1\n\n"},{"id":"349698","messageId":"20180608135548.216405-7-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 06/21] commit-graph: add 'verify' subcommand","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:17Z","receivedAt":"2018-06-08T13:56:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"If the commit-graph file becomes corrupt, we need a way to verify\nthat its contents match the object database. In the manner of\n'git fsck' we will implement a 'git commit-graph verify' subcommand\nto report all issues with the file.\n\nAdd the 'verify' subcommand to the 'commit-graph' builtin and its\ndocumentation. The subcommand is currently a no-op except for\nloading the commit-graph into memory, which may trigger run-time\nerrors that would be caught by normal use. Add a simple test that\nensures the command returns a zero error code.\n\nIf no commit-graph file exists, this is an acceptable state. Do\nnot report any errors.\n\nHelped-by: Ramsay Jones \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 6 +++++\n builtin/commit-graph.c | 39 ++++++++++++++++++++++++++++++\n commit-graph.c | 23 ++++++++++++++++++\n commit-graph.h | 3 +++\n t/t5318-commit-graph.sh | 10 ++++++++\n 5 files changed, 81 insertions(+)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 4c97b555cc..a222cfab08 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git commit-graph read' [--object-dir ]\n+'git commit-graph verify' [--object-dir ]\n 'git commit-graph write' [--object-dir ]\n \n \n@@ -52,6 +53,11 @@ existing commit-graph file.\n Read a graph file given by the commit-graph file and output basic\n details about the graph file. Used for debugging purposes.\n \n+'verify'::\n+\n+Read the commit-graph file and verify its contents against the object\n+database. Used to check for corrupted data.\n+\n \n EXAMPLES\n --------\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex f0875b8bf3..9d108f43a9 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -3,15 +3,22 @@\n #include \"dir.h\"\n #include \"lockfile.h\"\n #include \"parse-options.h\"\n+#include \"repository.h\"\n #include \"commit-graph.h\"\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n+\tN_(\"git commit-graph verify [--object-dir ]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n+static const char * const builtin_commit_graph_verify_usage[] = {\n+\tN_(\"git commit-graph verify [--object-dir ]\"),\n+\tNULL\n+};\n+\n static const char * const builtin_commit_graph_read_usage[] = {\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tNULL\n@@ -29,6 +36,36 @@ static struct opts_commit_graph {\n \tint append;\n } opts;\n \n+\n+static int graph_verify(int argc, const char **argv)\n+{\n+\tstruct commit_graph *graph = NULL;\n+\tchar *graph_name;\n+\n+\tstatic struct option builtin_commit_graph_verify_options[] = {\n+\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n+\t\t\t N_(\"dir\"),\n+\t\t\t N_(\"The object directory to store the graph\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t builtin_commit_graph_verify_options,\n+\t\t\t builtin_commit_graph_verify_usage, 0);\n+\n+\tif (!opts.obj_dir)\n+\t\topts.obj_dir = get_object_directory();\n+\n+\tgraph_name = get_commit_graph_filename(opts.obj_dir);\n+\tgraph = load_commit_graph_one(graph_name);\n+\tFREE_AND_NULL(graph_name);\n+\n+\tif (!graph)\n+\t\treturn 0;\n+\n+\treturn verify_commit_graph(the_repository, graph);\n+}\n+\n static int graph_read(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n@@ -165,6 +202,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \tif (argc > 0) {\n \t\tif (!strcmp(argv[0], \"read\"))\n \t\t\treturn graph_read(argc, argv);\n+\t\tif (!strcmp(argv[0], \"verify\"))\n+\t\t\treturn graph_verify(argc, argv);\n \t\tif (!strcmp(argv[0], \"write\"))\n \t\t\treturn graph_write(argc, argv);\n \t}\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 9e228d3bb5..22ef696e18 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -827,3 +827,26 @@ void write_commit_graph(const char *obj_dir,\n \toids.alloc = 0;\n \toids.nr = 0;\n }\n+\n+static int verify_commit_graph_error;\n+\n+static void graph_report(const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tverify_commit_graph_error = 1;\n+\n+\tva_start(ap, fmt);\n+\tvfprintf(stderr, fmt, ap);\n+\tfprintf(stderr, \"\\n\");\n+\tva_end(ap);\n+}\n+\n+int verify_commit_graph(struct repository *r, struct commit_graph *g)\n+{\n+\tif (!g) {\n+\t\tgraph_report(\"no commit-graph file loaded\");\n+\t\treturn 1;\n+\t}\n+\n+\treturn verify_commit_graph_error;\n+}\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 96cccb10f3..4359812fb4 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -2,6 +2,7 @@\n #define COMMIT_GRAPH_H\n \n #include \"git-compat-util.h\"\n+#include \"repository.h\"\n \n char *get_commit_graph_filename(const char *obj_dir);\n \n@@ -53,4 +54,6 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tint nr_commits,\n \t\t\tint append);\n \n+int verify_commit_graph(struct repository *r, struct commit_graph *g);\n+\n #endif\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 77d85aefe7..6ca451dfd2 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -11,6 +11,11 @@ test_expect_success 'setup full repo' '\n \tobjdir=\".git/objects\"\n '\n \n+test_expect_success 'verify graph with no graph file' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph verify\n+'\n+\n test_expect_success 'write graph with no packs' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write --object-dir . &&\n@@ -230,4 +235,9 @@ test_expect_success 'perform fast-forward merge in full repo' '\n \ttest_cmp expect output\n '\n \n+test_expect_success 'git commit-graph verify' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph verify >output\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349699","messageId":"20180608135548.216405-10-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 09/21] commit-graph: verify corrupt OID fanout and lookup","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:20Z","receivedAt":"2018-06-08T13:56:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In the commit-graph file, the OID fanout chunk provides an index into\nthe OID lookup. The 'verify' subcommand should find incorrect values\nin the fanout.\n\nSimilarly, the 'verify' subcommand should find out-of-order values in\nthe OID lookup.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 36 ++++++++++++++++++++++++++++++++++++\n t/t5318-commit-graph.sh | 22 ++++++++++++++++++++++\n 2 files changed, 58 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f30b4ccee9..866a9e7e41 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -843,6 +843,9 @@ static void graph_report(const char *fmt, ...)\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g)\n {\n+\tuint32_t i, cur_fanout_pos = 0;\n+\tstruct object_id prev_oid, cur_oid;\n+\n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n \t\treturn 1;\n@@ -857,5 +860,38 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \tif (!g->chunk_commit_data)\n \t\tgraph_report(\"commit-graph is missing the Commit Data chunk\");\n \n+\tif (verify_commit_graph_error)\n+\t\treturn verify_commit_graph_error;\n+\n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n+\n+\t\tif (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n+\t\t\tgraph_report(\"commit-graph has incorrect OID order: %s then %s\",\n+\t\t\t\t oid_to_hex(&prev_oid),\n+\t\t\t\t oid_to_hex(&cur_oid));\n+\n+\t\toidcpy(&prev_oid, &cur_oid);\n+\n+\t\twhile (cur_oid.hash[0] > cur_fanout_pos) {\n+\t\t\tuint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n+\t\t\tif (i != fanout_value)\n+\t\t\t\tgraph_report(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\",\n+\t\t\t\t\t cur_fanout_pos, fanout_value, i);\n+\n+\t\t\tcur_fanout_pos++;\n+\t\t}\n+\t}\n+\n+\twhile (cur_fanout_pos < 256) {\n+\t\tuint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n+\n+\t\tif (g->num_commits != fanout_value)\n+\t\t\tgraph_report(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\",\n+\t\t\t\t cur_fanout_pos, fanout_value, i);\n+\n+\t\tcur_fanout_pos++;\n+\t}\n+\n \treturn verify_commit_graph_error;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex c03792a8ed..4809cc881f 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -247,6 +247,7 @@ test_expect_success 'git commit-graph verify' '\n \tgit commit-graph verify >output\n '\n \n+HASH_LEN=20\n GRAPH_BYTE_VERSION=4\n GRAPH_BYTE_HASH=5\n GRAPH_BYTE_CHUNK_COUNT=6\n@@ -258,6 +259,12 @@ GRAPH_BYTE_OID_LOOKUP_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n \t\t\t 1 * $GRAPH_CHUNK_LOOKUP_WIDTH))\n GRAPH_BYTE_COMMIT_DATA_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n \t\t\t 2 * $GRAPH_CHUNK_LOOKUP_WIDTH))\n+GRAPH_FANOUT_OFFSET=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n+\t\t $GRAPH_CHUNK_LOOKUP_WIDTH * $GRAPH_CHUNK_LOOKUP_ROWS))\n+GRAPH_BYTE_FANOUT1=$(($GRAPH_FANOUT_OFFSET + 4 * 4))\n+GRAPH_BYTE_FANOUT2=$(($GRAPH_FANOUT_OFFSET + 4 * 255))\n+GRAPH_OID_LOOKUP_OFFSET=$(($GRAPH_FANOUT_OFFSET + 4 * 256))\n+GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 8))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -312,4 +319,19 @@ test_expect_success 'detect missing commit data chunk' '\n \t\t\"missing the Commit Data chunk\"\n '\n \n+test_expect_success 'detect incorrect fanout' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_FANOUT1 \"\\01\" \\\n+\t\t\"fanout value\"\n+'\n+\n+test_expect_success 'detect incorrect fanout final value' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_FANOUT2 \"\\01\" \\\n+\t\t\"fanout value\"\n+'\n+\n+test_expect_success 'detect incorrect OID order' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_LOOKUP_ORDER \"\\01\" \\\n+\t\t\"incorrect OID order\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349700","messageId":"20180608135548.216405-13-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 12/21] commit-graph: verify parent list","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:22Z","receivedAt":"2018-06-08T13:56:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file stores parents in a two-column portion of the\ncommit data chunk. If there is only one parent, then the second column\nstores 0xFFFFFFFF to indicate no second parent.\n\nThe 'verify' subcommand checks the parent list for the commit loaded\nfrom the commit-graph and the one parsed from the object database. Test\nthese checks for corrupt parents, too many parents, and wrong parents.\n\nAdd a boundary check to insert_parent_or_die() for when the parent\nposition value is out of range.\n\nThe octopus merge will be tested in a later commit.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 28 ++++++++++++++++++++++++++++\n t/t5318-commit-graph.sh | 18 ++++++++++++++++++\n 2 files changed, 46 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 5df18394f9..6d8d774eb0 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -244,6 +244,9 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n \tstruct commit *c;\n \tstruct object_id oid;\n \n+\tif (pos >= g->num_commits)\n+\t\tdie(\"invalid parent position %\"PRIu64, pos);\n+\n \thashcpy(oid.hash, g->chunk_oid_lookup + g->hash_len * pos);\n \tc = lookup_commit(&oid);\n \tif (!c)\n@@ -907,6 +910,7 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \n \tfor (i = 0; i < g->num_commits; i++) {\n \t\tstruct commit *graph_commit, *odb_commit;\n+\t\tstruct commit_list *graph_parents, *odb_parents;\n \n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n@@ -924,6 +928,30 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\t\t\t oid_to_hex(&cur_oid),\n \t\t\t\t oid_to_hex(get_commit_tree_oid(graph_commit)),\n \t\t\t\t oid_to_hex(get_commit_tree_oid(odb_commit)));\n+\n+\t\tgraph_parents = graph_commit->parents;\n+\t\todb_parents = odb_commit->parents;\n+\n+\t\twhile (graph_parents) {\n+\t\t\tif (odb_parents == NULL) {\n+\t\t\t\tgraph_report(\"commit-graph parent list for commit %s is too long\",\n+\t\t\t\t\t oid_to_hex(&cur_oid));\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (oidcmp(&graph_parents->item->object.oid, &odb_parents->item->object.oid))\n+\t\t\t\tgraph_report(\"commit-graph parent for %s is %s != %s\",\n+\t\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid),\n+\t\t\t\t\t oid_to_hex(&odb_parents->item->object.oid));\n+\n+\t\t\tgraph_parents = graph_parents->next;\n+\t\t\todb_parents = odb_parents->next;\n+\t\t}\n+\n+\t\tif (odb_parents != NULL)\n+\t\t\tgraph_report(\"commit-graph parent list for commit %s terminates early\",\n+\t\t\t\t oid_to_hex(&cur_oid));\n \t}\n \n \treturn verify_commit_graph_error;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 2b9214bc83..9a3481c30f 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -269,6 +269,9 @@ GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 8))\n GRAPH_BYTE_OID_LOOKUP_MISSING=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 4 + 10))\n GRAPH_COMMIT_DATA_OFFSET=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * $NUM_COMMITS))\n GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n+GRAPH_BYTE_COMMIT_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN))\n+GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4))\n+GRAPH_BYTE_COMMIT_WRONG_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -348,4 +351,19 @@ test_expect_success 'detect incorrect tree OID' '\n \t\t\"root tree OID for commit\"\n '\n \n+test_expect_success 'detect incorrect parent int-id' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_PARENT \"\\01\" \\\n+\t\t\"invalid parent\"\n+'\n+\n+test_expect_success 'detect extra parent int-id' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_EXTRA_PARENT \"\\00\" \\\n+\t\t\"is too long\"\n+'\n+\n+test_expect_success 'detect wrong parent' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_WRONG_PARENT \"\\01\" \\\n+\t\t\"commit-graph parent for\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349701","messageId":"20180608135548.216405-11-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 10/21] commit-graph: verify objects exist","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:21Z","receivedAt":"2018-06-08T13:56:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In the 'verify' subcommand, load commits directly from the object\ndatabase to ensure they exist. Parse by skipping the commit-graph.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 18 ++++++++++++++++++\n t/t5318-commit-graph.sh | 7 +++++++\n 2 files changed, 25 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 866a9e7e41..00e89b71e9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -11,6 +11,7 @@\n #include \"sha1-lookup.h\"\n #include \"commit-graph.h\"\n #include \"object-store.h\"\n+#include \"alloc.h\"\n \n #define GRAPH_SIGNATURE 0x43475048 /* \"CGPH\" */\n #define GRAPH_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n@@ -242,6 +243,7 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n {\n \tstruct commit *c;\n \tstruct object_id oid;\n+\n \thashcpy(oid.hash, g->chunk_oid_lookup + g->hash_len * pos);\n \tc = lookup_commit(&oid);\n \tif (!c)\n@@ -893,5 +895,21 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\tcur_fanout_pos++;\n \t}\n \n+\tif (verify_commit_graph_error)\n+\t\treturn verify_commit_graph_error;\n+\n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct commit *odb_commit;\n+\n+\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n+\n+\t\todb_commit = (struct commit *)create_object(r, cur_oid.hash, alloc_commit_node(r));\n+\t\tif (parse_commit_internal(odb_commit, 0, 0)) {\n+\t\t\tgraph_report(\"failed to parse %s from object database\",\n+\t\t\t\t oid_to_hex(&cur_oid));\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+\n \treturn verify_commit_graph_error;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 4809cc881f..af5e34c0cb 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -247,6 +247,7 @@ test_expect_success 'git commit-graph verify' '\n \tgit commit-graph verify >output\n '\n \n+NUM_COMMITS=9\n HASH_LEN=20\n GRAPH_BYTE_VERSION=4\n GRAPH_BYTE_HASH=5\n@@ -265,6 +266,7 @@ GRAPH_BYTE_FANOUT1=$(($GRAPH_FANOUT_OFFSET + 4 * 4))\n GRAPH_BYTE_FANOUT2=$(($GRAPH_FANOUT_OFFSET + 4 * 255))\n GRAPH_OID_LOOKUP_OFFSET=$(($GRAPH_FANOUT_OFFSET + 4 * 256))\n GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 8))\n+GRAPH_BYTE_OID_LOOKUP_MISSING=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 4 + 10))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -334,4 +336,9 @@ test_expect_success 'detect incorrect OID order' '\n \t\t\"incorrect OID order\"\n '\n \n+test_expect_success 'detect OID not in object database' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_LOOKUP_MISSING \"\\01\" \\\n+\t\t\"from object database\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349702","messageId":"20180608135548.216405-18-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 17/21] fsck: verify commit-graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:27Z","receivedAt":"2018-06-08T13:56:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"If core.commitGraph is true, verify the contents of the commit-graph\nduring 'git fsck' using the 'git commit-graph verify' subcommand. Run\nthis check on all alternates, as well.\n\nWe use a new process for two reasons:\n\n1. The subcommand decouples the details of loading and verifying a\n commit-graph file from the other fsck details.\n\n2. The commit-graph verification requires the commits to be loaded\n in a specific order to guarantee we parse from the commit-graph\n file for some objects and from the object database for others.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-fsck.txt | 3 +++\n builtin/fsck.c | 21 +++++++++++++++++++++\n t/t5318-commit-graph.sh | 8 ++++++++\n 3 files changed, 32 insertions(+)\n\ndiff --git a/Documentation/git-fsck.txt b/Documentation/git-fsck.txt\nindex b9f060e3b2..ab9a93fb9b 100644\n--- a/Documentation/git-fsck.txt\n+++ b/Documentation/git-fsck.txt\n@@ -110,6 +110,9 @@ Any corrupt objects you will have to find in backups or other archives\n (i.e., you can just remove them and do an 'rsync' with some other site in\n the hopes that somebody else has the object you have corrupted).\n \n+If core.commitGraph is true, the commit-graph file will also be inspected\n+using 'git commit-graph verify'. See linkgit:git-commit-graph[1].\n+\n Extracted Diagnostics\n ---------------------\n \ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 3ad4f160f9..9fb2edc69f 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -18,6 +18,7 @@\n #include \"decorate.h\"\n #include \"packfile.h\"\n #include \"object-store.h\"\n+#include \"run-command.h\"\n \n #define REACHABLE 0x0001\n #define SEEN 0x0002\n@@ -47,6 +48,7 @@ static int name_objects;\n #define ERROR_REACHABLE 02\n #define ERROR_PACK 04\n #define ERROR_REFS 010\n+#define ERROR_COMMIT_GRAPH 020\n \n static const char *describe_object(struct object *obj)\n {\n@@ -822,5 +824,24 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t}\n \n \tcheck_connectivity();\n+\n+\tif (core_commit_graph) {\n+\t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n+\t\tconst char *verify_argv[] = { \"commit-graph\", \"verify\", NULL, NULL, NULL };\n+\t\tcommit_graph_verify.argv = verify_argv;\n+\t\tcommit_graph_verify.git_cmd = 1;\n+\n+\t\tif (run_command(&commit_graph_verify))\n+\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n+\n+\t\tprepare_alt_odb(the_repository);\n+\t\tfor (alt = the_repository->objects->alt_odb_list; alt; alt = alt->next) {\n+\t\t\tverify_argv[2] = \"--object-dir\";\n+\t\t\tverify_argv[3] = alt->path;\n+\t\t\tif (run_command(&commit_graph_verify))\n+\t\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n+\t\t}\n+\t}\n+\n \treturn errors_found;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex bd5b8428af..0da5a51552 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -399,4 +399,12 @@ test_expect_success 'detect invalid checksum hash' '\n \t\t\"incorrect checksum\"\n '\n \n+test_expect_success 'git fsck (checks commit-graph)' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit fsck &&\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_FOOTER \"\\00\" \\\n+\t\t\"incorrect checksum\" &&\n+\ttest_must_fail git fsck\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349703","messageId":"20180608135548.216405-22-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 21/21] commit-graph: update design document","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:30Z","receivedAt":"2018-06-08T13:56:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph feature is now integrated with 'fsck' and 'gc',\nso remove those items from the \"Future Work\" section of the\ncommit-graph design document.\n\nAlso remove the section on lazy-loading trees, as that was completed\nin an earlier patch series.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 22 ----------------------\n 1 file changed, 22 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex e1a883eb46..c664acbd76 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -118,9 +118,6 @@ Future Work\n - The commit graph feature currently does not honor commit grafts. This can\n be remedied by duplicating or refactoring the current graft logic.\n \n-- The 'commit-graph' subcommand does not have a \"verify\" mode that is\n- necessary for integration with fsck.\n-\n - After computing and storing generation numbers, we must make graph\n walks aware of generation numbers to gain the performance benefits they\n enable. This will mostly be accomplished by swapping a commit-date-ordered\n@@ -130,25 +127,6 @@ Future Work\n - 'log --topo-order'\n - 'tag --merged'\n \n-- Currently, parse_commit_gently() requires filling in the root tree\n- object for a commit. This passes through lookup_tree() and consequently\n- lookup_object(). Also, it calls lookup_commit() when loading the parents.\n- These method calls check the ODB for object existence, even if the\n- consumer does not need the content. For example, we do not need the\n- tree contents when computing merge bases. Now that commit parsing is\n- removed from the computation time, these lookup operations are the\n- slowest operations keeping graph walks from being fast. Consider\n- loading these objects without verifying their existence in the ODB and\n- only loading them fully when consumers need them. Consider a method\n- such as \"ensure_tree_loaded(commit)\" that fully loads a tree before\n- using commit->tree.\n-\n-- The current design uses the 'commit-graph' subcommand to generate the graph.\n- When this feature stabilizes enough to recommend to most users, we should\n- add automatic graph writes to common operations that create many commits.\n- For example, one could compute a graph on 'clone', 'fetch', or 'repack'\n- commands.\n-\n - A server could provide a commit graph file as part of the network protocol\n to avoid extra calculations by clients. This feature is only of benefit if\n the user is willing to trust the file, because verifying the file is correct\n-- \n2.18.0.rc1\n\n"},{"id":"349704","messageId":"20180608135548.216405-17-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 16/21] commit-graph: verify contents match checksum","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:26Z","receivedAt":"2018-06-08T13:56:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file ends with a SHA1 hash of the previous contents. If\na commit-graph file has errors but the checksum hash is correct, then we\nknow that the problem is a bug in Git and not simply file corruption\nafter-the-fact.\n\nCompute the checksum right away so it is the first error that appears,\nand make the message translatable since this error can be \"corrected\" by\na user by simply deleting the file and recomputing. The rest of the\nerrors are useful only to developers.\n\nBe sure to continue checking the rest of the file data if the checksum\nis wrong. This is important for our tests, as we break the checksum as\nwe modify bytes of the commit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 16 ++++++++++++++--\n t/t5318-commit-graph.sh | 6 ++++++\n 2 files changed, 20 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 6d6c6beff9..d926c4b59f 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -833,6 +833,7 @@ void write_commit_graph(const char *obj_dir,\n \toids.nr = 0;\n }\n \n+#define VERIFY_COMMIT_GRAPH_ERROR_HASH 2\n static int verify_commit_graph_error;\n \n static void graph_report(const char *fmt, ...)\n@@ -852,8 +853,10 @@ static void graph_report(const char *fmt, ...)\n int verify_commit_graph(struct repository *r, struct commit_graph *g)\n {\n \tuint32_t i, cur_fanout_pos = 0;\n-\tstruct object_id prev_oid, cur_oid;\n+\tstruct object_id prev_oid, cur_oid, checksum;\n \tint generation_zero = 0;\n+\tstruct hashfile *f;\n+\tint devnull;\n \n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n@@ -872,6 +875,15 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \tif (verify_commit_graph_error)\n \t\treturn verify_commit_graph_error;\n \n+\tdevnull = open(\"/dev/null\", O_WRONLY);\n+\tf = hashfd(devnull, NULL);\n+\thashwrite(f, g->data, g->data_len - g->hash_len);\n+\tfinalize_hashfile(f, checksum.hash, CSUM_CLOSE);\n+\tif (hashcmp(checksum.hash, g->data + g->data_len - g->hash_len)) {\n+\t\tgraph_report(_(\"the commit-graph file has incorrect checksum and is likely corrupt\"));\n+\t\tverify_commit_graph_error = VERIFY_COMMIT_GRAPH_ERROR_HASH;\n+\t}\n+\n \tfor (i = 0; i < g->num_commits; i++) {\n \t\tstruct commit *graph_commit;\n \n@@ -909,7 +921,7 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\tcur_fanout_pos++;\n \t}\n \n-\tif (verify_commit_graph_error)\n+\tif (verify_commit_graph_error & ~VERIFY_COMMIT_GRAPH_ERROR_HASH)\n \t\treturn verify_commit_graph_error;\n \n \tfor (i = 0; i < g->num_commits; i++) {\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex cbd6462226..bd5b8428af 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -279,6 +279,7 @@ GRAPH_COMMIT_DATA_WIDTH=$(($HASH_LEN + 16))\n GRAPH_OCTOPUS_DATA_OFFSET=$(($GRAPH_COMMIT_DATA_OFFSET + \\\n \t\t\t $GRAPH_COMMIT_DATA_WIDTH * $NUM_COMMITS))\n GRAPH_BYTE_OCTOPUS=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4))\n+GRAPH_BYTE_FOOTER=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4 * $NUM_OCTOPUS_EDGES))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -393,4 +394,9 @@ test_expect_success 'detect incorrect parent for octopus merge' '\n \t\t\"invalid parent\"\n '\n \n+test_expect_success 'detect invalid checksum hash' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_FOOTER \"\\00\" \\\n+\t\t\"incorrect checksum\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349705","messageId":"20180608135548.216405-19-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 18/21] commit-graph: use string-list API for input","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:28Z","receivedAt":"2018-06-08T13:56:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 39 +++++++++++++--------------------------\n commit-graph.c | 15 +++++++--------\n commit-graph.h | 7 +++----\n 3 files changed, 23 insertions(+), 38 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 9d108f43a9..ea28bc311a 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -119,13 +119,9 @@ static int graph_read(int argc, const char **argv)\n \n static int graph_write(int argc, const char **argv)\n {\n-\tconst char **pack_indexes = NULL;\n-\tint packs_nr = 0;\n-\tconst char **commit_hex = NULL;\n-\tint commits_nr = 0;\n-\tconst char **lines = NULL;\n-\tint lines_nr = 0;\n-\tint lines_alloc = 0;\n+\tstruct string_list *pack_indexes = NULL;\n+\tstruct string_list *commit_hex = NULL;\n+\tstruct string_list lines;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n@@ -149,34 +145,25 @@ static int graph_write(int argc, const char **argv)\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n \n+\tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n \t\tstruct strbuf buf = STRBUF_INIT;\n-\t\tlines_nr = 0;\n-\t\tlines_alloc = 128;\n-\t\tALLOC_ARRAY(lines, lines_alloc);\n-\n-\t\twhile (strbuf_getline(&buf, stdin) != EOF) {\n-\t\t\tALLOC_GROW(lines, lines_nr + 1, lines_alloc);\n-\t\t\tlines[lines_nr++] = strbuf_detach(&buf, NULL);\n-\t\t}\n-\n-\t\tif (opts.stdin_packs) {\n-\t\t\tpack_indexes = lines;\n-\t\t\tpacks_nr = lines_nr;\n-\t\t}\n-\t\tif (opts.stdin_commits) {\n-\t\t\tcommit_hex = lines;\n-\t\t\tcommits_nr = lines_nr;\n-\t\t}\n+\n+\t\twhile (strbuf_getline(&buf, stdin) != EOF)\n+\t\t\tstring_list_append(&lines, strbuf_detach(&buf, NULL));\n+\n+\t\tif (opts.stdin_packs)\n+\t\t\tpack_indexes = &lines;\n+\t\tif (opts.stdin_commits)\n+\t\t\tcommit_hex = &lines;\n \t}\n \n \twrite_commit_graph(opts.obj_dir,\n \t\t\t pack_indexes,\n-\t\t\t packs_nr,\n \t\t\t commit_hex,\n-\t\t\t commits_nr,\n \t\t\t opts.append);\n \n+\tstring_list_clear(&lines, 0);\n \treturn 0;\n }\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex d926c4b59f..a06e7e9549 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -657,10 +657,8 @@ static void compute_generation_numbers(struct packed_commit_list* commits)\n }\n \n void write_commit_graph(const char *obj_dir,\n-\t\t\tconst char **pack_indexes,\n-\t\t\tint nr_packs,\n-\t\t\tconst char **commit_hex,\n-\t\t\tint nr_commits,\n+\t\t\tstruct string_list *pack_indexes,\n+\t\t\tstruct string_list *commit_hex,\n \t\t\tint append)\n {\n \tstruct packed_oid_list oids;\n@@ -701,10 +699,10 @@ void write_commit_graph(const char *obj_dir,\n \t\tint dirlen;\n \t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n \t\tdirlen = packname.len;\n-\t\tfor (i = 0; i < nr_packs; i++) {\n+\t\tfor (i = 0; i < pack_indexes->nr; i++) {\n \t\t\tstruct packed_git *p;\n \t\t\tstrbuf_setlen(&packname, dirlen);\n-\t\t\tstrbuf_addstr(&packname, pack_indexes[i]);\n+\t\t\tstrbuf_addstr(&packname, pack_indexes->items[i].string);\n \t\t\tp = add_packed_git(packname.buf, packname.len, 1);\n \t\t\tif (!p)\n \t\t\t\tdie(\"error adding pack %s\", packname.buf);\n@@ -717,12 +715,13 @@ void write_commit_graph(const char *obj_dir,\n \t}\n \n \tif (commit_hex) {\n-\t\tfor (i = 0; i < nr_commits; i++) {\n+\t\tfor (i = 0; i < commit_hex->nr; i++) {\n \t\t\tconst char *end;\n \t\t\tstruct object_id oid;\n \t\t\tstruct commit *result;\n \n-\t\t\tif (commit_hex[i] && parse_oid_hex(commit_hex[i], &oid, &end))\n+\t\t\tif (commit_hex->items[i].string &&\n+\t\t\t parse_oid_hex(commit_hex->items[i].string, &oid, &end))\n \t\t\t\tcontinue;\n \n \t\t\tresult = lookup_commit_reference_gently(&oid, 1);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 4359812fb4..a79b854470 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -3,6 +3,7 @@\n \n #include \"git-compat-util.h\"\n #include \"repository.h\"\n+#include \"string-list.h\"\n \n char *get_commit_graph_filename(const char *obj_dir);\n \n@@ -48,10 +49,8 @@ struct commit_graph {\n struct commit_graph *load_commit_graph_one(const char *graph_file);\n \n void write_commit_graph(const char *obj_dir,\n-\t\t\tconst char **pack_indexes,\n-\t\t\tint nr_packs,\n-\t\t\tconst char **commit_hex,\n-\t\t\tint nr_commits,\n+\t\t\tstruct string_list *pack_indexes,\n+\t\t\tstruct string_list *commit_hex,\n \t\t\tint append);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n-- \n2.18.0.rc1\n\n"},{"id":"349706","messageId":"20180608135548.216405-20-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 19/21] commit-graph: add '--reachable' option","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:28Z","receivedAt":"2018-06-08T13:56:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When writing commit-graph files, it can be convenient to ask for all\nreachable commits (starting at the ref set) in the resulting file. This\nis particularly helpful when writing to stdin is complicated, such as a\nfuture integration with 'git gc'.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 8 ++++++--\n builtin/commit-graph.c | 16 ++++++++++++----\n commit-graph.c | 18 ++++++++++++++++++\n commit-graph.h | 1 +\n t/t5318-commit-graph.sh | 10 ++++++++++\n 5 files changed, 47 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex a222cfab08..dececb79d7 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -38,12 +38,16 @@ Write a commit graph file based on the commits found in packfiles.\n +\n With the `--stdin-packs` option, generate the new commit graph by\n walking objects only in the specified pack-indexes. (Cannot be combined\n-with --stdin-commits.)\n+with `--stdin-commits` or `--reachable`.)\n +\n With the `--stdin-commits` option, generate the new commit graph by\n walking commits starting at the commits specified in stdin as a list\n of OIDs in hex, one OID per line. (Cannot be combined with\n---stdin-packs.)\n+`--stdin-packs` or `--reachable`.)\n++\n+With the `--reachable` option, generate the new commit graph by walking\n+commits starting at all refs. (Cannot be combined with `--stdin-commits`\n+or `--stdin-packs`.)\n +\n With the `--append` option, include all commits that are present in the\n existing commit-graph file.\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex ea28bc311a..c7d0db5ab4 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -10,7 +10,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph verify [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -25,12 +25,13 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n static struct opts_commit_graph {\n \tconst char *obj_dir;\n+\tint reachable;\n \tint stdin_packs;\n \tint stdin_commits;\n \tint append;\n@@ -127,6 +128,8 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n \t\t\tN_(\"dir\"),\n \t\t\tN_(\"The object directory to store the graph\")),\n+\t\tOPT_BOOL(0, \"reachable\", &opts.reachable,\n+\t\t\tN_(\"start walk at all refs\")),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\tN_(\"scan pack-indexes listed by stdin for commits\")),\n \t\tOPT_BOOL(0, \"stdin-commits\", &opts.stdin_commits,\n@@ -140,11 +143,16 @@ static int graph_write(int argc, const char **argv)\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\n \n-\tif (opts.stdin_packs && opts.stdin_commits)\n-\t\tdie(_(\"cannot use both --stdin-commits and --stdin-packs\"));\n+\tif (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)\n+\t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n \n+\tif (opts.reachable) {\n+\t\twrite_commit_graph_reachable(opts.obj_dir, opts.append);\n+\t\treturn 0;\n+\t}\n+\n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n \t\tstruct strbuf buf = STRBUF_INIT;\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a06e7e9549..adf54e3fe7 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -7,6 +7,7 @@\n #include \"packfile.h\"\n #include \"commit.h\"\n #include \"object.h\"\n+#include \"refs.h\"\n #include \"revision.h\"\n #include \"sha1-lookup.h\"\n #include \"commit-graph.h\"\n@@ -656,6 +657,23 @@ static void compute_generation_numbers(struct packed_commit_list* commits)\n \t}\n }\n \n+static int add_ref_to_list(const char *refname,\n+\t\t\t const struct object_id *oid,\n+\t\t\t int flags, void *cb_data)\n+{\n+\tstruct string_list *list = (struct string_list*)cb_data;\n+\tstring_list_append(list, oid_to_hex(oid));\n+\treturn 0;\n+}\n+\n+void write_commit_graph_reachable(const char *obj_dir, int append)\n+{\n+\tstruct string_list list;\n+\tstring_list_init(&list, 1);\n+\tfor_each_ref(add_ref_to_list, &list);\n+\twrite_commit_graph(obj_dir, NULL, &list, append);\n+}\n+\n void write_commit_graph(const char *obj_dir,\n \t\t\tstruct string_list *pack_indexes,\n \t\t\tstruct string_list *commit_hex,\ndiff --git a/commit-graph.h b/commit-graph.h\nindex a79b854470..506cb45fb1 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -48,6 +48,7 @@ struct commit_graph {\n \n struct commit_graph *load_commit_graph_one(const char *graph_file);\n \n+void write_commit_graph_reachable(const char *obj_dir, int append);\n void write_commit_graph(const char *obj_dir,\n \t\t\tstruct string_list *pack_indexes,\n \t\t\tstruct string_list *commit_hex,\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 0da5a51552..06ecbb3f4a 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -205,6 +205,16 @@ test_expect_success 'build graph from commits with append' '\n graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n \n+test_expect_success 'build graph using --reachable' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph write --reachable &&\n+\ttest_path_is_file $objdir/info/commit-graph &&\n+\tgraph_read_expect \"11\" \"large_edges\"\n+'\n+\n+graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n+graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n+\n test_expect_success 'setup bare repo' '\n \tcd \"$TRASH_DIRECTORY\" &&\n \tgit clone --bare --no-local full bare &&\n-- \n2.18.0.rc1\n\n"},{"id":"349707","messageId":"20180608135548.216405-16-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 15/21] commit-graph: test for corrupted octopus edge","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:25Z","receivedAt":"2018-06-08T13:56:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file has an extra chunk to store the parent int-ids for\nparents beyond the first parent for octopus merges. Our test repo has a\nsingle octopus merge that we can manipulate to demonstrate the 'verify'\nsubcommand detects incorrect values in that chunk.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5318-commit-graph.sh | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex b7b4410e75..cbd6462226 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -248,6 +248,7 @@ test_expect_success 'git commit-graph verify' '\n '\n \n NUM_COMMITS=9\n+NUM_OCTOPUS_EDGES=2\n HASH_LEN=20\n GRAPH_BYTE_VERSION=4\n GRAPH_BYTE_HASH=5\n@@ -274,6 +275,10 @@ GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4))\n GRAPH_BYTE_COMMIT_WRONG_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3))\n GRAPH_BYTE_COMMIT_GENERATION=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 11))\n GRAPH_BYTE_COMMIT_DATE=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 12))\n+GRAPH_COMMIT_DATA_WIDTH=$(($HASH_LEN + 16))\n+GRAPH_OCTOPUS_DATA_OFFSET=$(($GRAPH_COMMIT_DATA_OFFSET + \\\n+\t\t\t $GRAPH_COMMIT_DATA_WIDTH * $NUM_COMMITS))\n+GRAPH_BYTE_OCTOPUS=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -383,4 +388,9 @@ test_expect_success 'detect incorrect commit date' '\n \t\t\"commit date\"\n '\n \n+test_expect_success 'detect incorrect parent for octopus merge' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OCTOPUS \"\\01\" \\\n+\t\t\"invalid parent\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349708","messageId":"20180608135548.216405-21-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 20/21] gc: automatically write commit-graph files","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:29Z","receivedAt":"2018-06-08T13:56:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file is a very helpful feature for speeding up git\noperations. In order to make it more useful, make it possible to\nwrite the commit-graph file during standard garbage collection\noperations.\n\nAdd a 'gc.commitGraph' config setting that triggers writing a\ncommit-graph file after any non-trivial 'git gc' command. Defaults to\nfalse while the commit-graph feature matures. We specifically do not\nwant to have this on by default until the commit-graph feature is fully\nintegrated with history-modifying features like shallow clones.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 10 +++++++++-\n Documentation/git-gc.txt | 4 ++++\n builtin/gc.c | 6 ++++++\n t/t5318-commit-graph.sh | 14 ++++++++++++++\n 4 files changed, 33 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex ab641bf5a9..f2b5ed17c8 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -906,7 +906,8 @@ the `GIT_NOTES_REF` environment variable. See linkgit:git-notes[1].\n \n core.commitGraph::\n \tEnable git commit graph feature. Allows reading from the\n-\tcommit-graph file.\n+\tcommit-graph file. See `gc.commitGraph` for automatically\n+\tmaintaining the file.\n \n core.sparseCheckout::\n \tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n@@ -1647,6 +1648,13 @@ this configuration variable is ignored, all packs except the base pack\n will be repacked. After this the number of packs should go below\n gc.autoPackLimit and gc.bigPackThreshold should be respected again.\n \n+gc.commitGraph::\n+\tIf true, then gc will rewrite the commit-graph file when\n+\tlinkgit:git-gc[1] is run. When using linkgit:git-gc[1]\n+\t'--auto' the commit-graph will be updated if housekeeping is\n+\trequired. Default is false. See linkgit:git-commit-graph[1]\n+\tfor details.\n+\n gc.logExpiry::\n \tIf the file gc.log exists, then `git gc --auto` won't run\n \tunless that file is more than 'gc.logExpiry' old. Default is\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 24b2dd44fe..f5bc98ccb3 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -136,6 +136,10 @@ The optional configuration variable `gc.packRefs` determines if\n it within all non-bare repos or it can be set to a boolean value.\n This defaults to true.\n \n+The optional configuration variable `gc.commitGraph` determines if\n+'git gc' should run 'git commit-graph write'. This can be set to a\n+boolean value. This defaults to false.\n+\n The optional configuration variable `gc.aggressiveWindow` controls how\n much time is spent optimizing the delta compression of the objects in\n the repository when the --aggressive option is specified. The larger\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex ccfb1ceaeb..4e06e8372d 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -20,6 +20,7 @@\n #include \"sigchain.h\"\n #include \"argv-array.h\"\n #include \"commit.h\"\n+#include \"commit-graph.h\"\n #include \"packfile.h\"\n #include \"object-store.h\"\n #include \"pack.h\"\n@@ -40,6 +41,7 @@ static int aggressive_depth = 50;\n static int aggressive_window = 250;\n static int gc_auto_threshold = 6700;\n static int gc_auto_pack_limit = 50;\n+static int gc_commit_graph = 0;\n static int detach_auto = 1;\n static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n@@ -129,6 +131,7 @@ static void gc_config(void)\n \tgit_config_get_int(\"gc.aggressivedepth\", &aggressive_depth);\n \tgit_config_get_int(\"gc.auto\", &gc_auto_threshold);\n \tgit_config_get_int(\"gc.autopacklimit\", &gc_auto_pack_limit);\n+\tgit_config_get_bool(\"gc.commitgraph\", &gc_commit_graph);\n \tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n \tgit_config_get_expiry(\"gc.pruneexpire\", &prune_expire);\n \tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n@@ -641,6 +644,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tif (pack_garbage.nr > 0)\n \t\tclean_pack_garbage();\n \n+\tif (gc_commit_graph)\n+\t\twrite_commit_graph_reachable(get_object_directory(), 0);\n+\n \tif (auto_gc && too_many_loose_objects())\n \t\twarning(_(\"There are too many unreachable loose objects; \"\n \t\t\t\"run 'git prune' to remove them.\"));\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 06ecbb3f4a..9a0661983c 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -245,6 +245,20 @@ test_expect_success 'perform fast-forward merge in full repo' '\n \ttest_cmp expect output\n '\n \n+test_expect_success 'check that gc computes commit-graph' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit --allow-empty -m \"blank\" &&\n+\tgit commit-graph write --reachable &&\n+\tcp $objdir/info/commit-graph commit-graph-before-gc &&\n+\tgit reset --hard HEAD~1 &&\n+\tgit config gc.commitGraph true &&\n+\tgit gc &&\n+\tcp $objdir/info/commit-graph commit-graph-after-gc &&\n+\t! test_cmp commit-graph-before-gc commit-graph-after-gc &&\n+\tgit commit-graph write --reachable &&\n+\ttest_cmp commit-graph-after-gc $objdir/info/commit-graph\n+'\n+\n # the verify tests below expect the commit-graph to contain\n # exactly the commits reachable from the commits/8 branch.\n # If the file changes the set of commits in the list, then the\n-- \n2.18.0.rc1\n\n"},{"id":"349709","messageId":"20180608135548.216405-12-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 11/21] commit-graph: verify root tree OIDs","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:22Z","receivedAt":"2018-06-08T13:57:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The 'verify' subcommand must compare the commit content parsed from the\ncommit-graph against the content in the object database. Use\nlookup_commit() and parse_commit_in_graph_one() to parse the commits\nfrom the graph and compare against a commit that is loaded separately\nand parsed directly from the object database.\n\nAdd checks for the root tree OID.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 17 ++++++++++++++++-\n t/t5318-commit-graph.sh | 7 +++++++\n 2 files changed, 23 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 00e89b71e9..5df18394f9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -866,6 +866,8 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\treturn verify_commit_graph_error;\n \n \tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct commit *graph_commit;\n+\n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n \t\tif (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n@@ -883,6 +885,11 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \n \t\t\tcur_fanout_pos++;\n \t\t}\n+\n+\t\tgraph_commit = lookup_commit(&cur_oid);\n+\t\tif (!parse_commit_in_graph_one(g, graph_commit))\n+\t\t\tgraph_report(\"failed to parse %s from commit-graph\",\n+\t\t\t\t oid_to_hex(&cur_oid));\n \t}\n \n \twhile (cur_fanout_pos < 256) {\n@@ -899,16 +906,24 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\treturn verify_commit_graph_error;\n \n \tfor (i = 0; i < g->num_commits; i++) {\n-\t\tstruct commit *odb_commit;\n+\t\tstruct commit *graph_commit, *odb_commit;\n \n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n+\t\tgraph_commit = lookup_commit(&cur_oid);\n \t\todb_commit = (struct commit *)create_object(r, cur_oid.hash, alloc_commit_node(r));\n \t\tif (parse_commit_internal(odb_commit, 0, 0)) {\n \t\t\tgraph_report(\"failed to parse %s from object database\",\n \t\t\t\t oid_to_hex(&cur_oid));\n \t\t\tcontinue;\n \t\t}\n+\n+\t\tif (oidcmp(&get_commit_tree_in_graph_one(g, graph_commit)->object.oid,\n+\t\t\t get_commit_tree_oid(odb_commit)))\n+\t\t\tgraph_report(\"root tree OID for commit %s in commit-graph is %s != %s\",\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t oid_to_hex(get_commit_tree_oid(graph_commit)),\n+\t\t\t\t oid_to_hex(get_commit_tree_oid(odb_commit)));\n \t}\n \n \treturn verify_commit_graph_error;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex af5e34c0cb..2b9214bc83 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -267,6 +267,8 @@ GRAPH_BYTE_FANOUT2=$(($GRAPH_FANOUT_OFFSET + 4 * 255))\n GRAPH_OID_LOOKUP_OFFSET=$(($GRAPH_FANOUT_OFFSET + 4 * 256))\n GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 8))\n GRAPH_BYTE_OID_LOOKUP_MISSING=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 4 + 10))\n+GRAPH_COMMIT_DATA_OFFSET=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * $NUM_COMMITS))\n+GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -341,4 +343,9 @@ test_expect_success 'detect OID not in object database' '\n \t\t\"from object database\"\n '\n \n+test_expect_success 'detect incorrect tree OID' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_TREE \"\\01\" \\\n+\t\t\"root tree OID for commit\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349710","messageId":"20180608135548.216405-15-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 14/21] commit-graph: verify commit date","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:24Z","receivedAt":"2018-06-08T13:57:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n commit-graph.c | 6 ++++++\n t/t5318-commit-graph.sh | 6 ++++++\n 2 files changed, 12 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex e0f71658da..6d6c6beff9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -986,6 +986,12 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\t\t\t oid_to_hex(&cur_oid),\n \t\t\t\t graph_commit->generation,\n \t\t\t\t max_generation + 1);\n+\n+\t\tif (graph_commit->date != odb_commit->date)\n+\t\t\tgraph_report(\"commit date for commit %s in commit-graph is %\"PRItime\" != %\"PRItime,\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t graph_commit->date,\n+\t\t\t\t odb_commit->date);\n \t}\n \n \treturn verify_commit_graph_error;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 5b75c4dca2..b7b4410e75 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -273,6 +273,7 @@ GRAPH_BYTE_COMMIT_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN))\n GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4))\n GRAPH_BYTE_COMMIT_WRONG_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3))\n GRAPH_BYTE_COMMIT_GENERATION=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 11))\n+GRAPH_BYTE_COMMIT_DATE=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 12))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -377,4 +378,9 @@ test_expect_success 'detect incorrect generation number' '\n \t\t\"non-zero generation number\"\n '\n \n+test_expect_success 'detect incorrect commit date' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_DATE \"\\01\" \\\n+\t\t\"commit date\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349711","messageId":"20180608135548.216405-14-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 13/21] commit-graph: verify generation number","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:23Z","receivedAt":"2018-06-08T13:57:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"While iterating through the commit parents, perform the generation\nnumber calculation and compare against the value stored in the\ncommit-graph.\n\nThe tests demonstrate that having a different set of parents affects\nthe generation number calculation, and this value propagates to\ndescendants. Hence, we drop the single-line condition on the output.\n\nSince Git will ship with the commit-graph feature without generation\nnumbers, we need to accept commit-graphs with all generation numbers\nequal to zero. In this case, ignore the generation number calculation.\n\nHowever, verify that we should never have a mix of zero and non-zero\ngeneration numbers. Create a test that sets one commit to generation\nzero and all following commits report a failure as they have non-zero\ngeneration in a file that contains generation number zero.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 34 ++++++++++++++++++++++++++++++++++\n t/t5318-commit-graph.sh | 11 +++++++++++\n 2 files changed, 45 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 6d8d774eb0..e0f71658da 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -846,10 +846,14 @@ static void graph_report(const char *fmt, ...)\n \tva_end(ap);\n }\n \n+#define GENERATION_ZERO_EXISTS 1\n+#define GENERATION_NUMBER_EXISTS 2\n+\n int verify_commit_graph(struct repository *r, struct commit_graph *g)\n {\n \tuint32_t i, cur_fanout_pos = 0;\n \tstruct object_id prev_oid, cur_oid;\n+\tint generation_zero = 0;\n \n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n@@ -911,6 +915,7 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \tfor (i = 0; i < g->num_commits; i++) {\n \t\tstruct commit *graph_commit, *odb_commit;\n \t\tstruct commit_list *graph_parents, *odb_parents;\n+\t\tuint32_t max_generation = 0;\n \n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n@@ -945,6 +950,9 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid),\n \t\t\t\t\t oid_to_hex(&odb_parents->item->object.oid));\n \n+\t\t\tif (graph_parents->item->generation > max_generation)\n+\t\t\t\tmax_generation = graph_parents->item->generation;\n+\n \t\t\tgraph_parents = graph_parents->next;\n \t\t\todb_parents = odb_parents->next;\n \t\t}\n@@ -952,6 +960,32 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\tif (odb_parents != NULL)\n \t\t\tgraph_report(\"commit-graph parent list for commit %s terminates early\",\n \t\t\t\t oid_to_hex(&cur_oid));\n+\n+\t\tif (!graph_commit->generation) {\n+\t\t\tif (generation_zero == GENERATION_NUMBER_EXISTS)\n+\t\t\t\tgraph_report(\"commit-graph has generation number zero for commit %s, but non-zero elsewhere\",\n+\t\t\t\t\t oid_to_hex(&cur_oid));\n+\t\t\tgeneration_zero = GENERATION_ZERO_EXISTS;\n+\t\t} else if (generation_zero == GENERATION_ZERO_EXISTS)\n+\t\t\tgraph_report(\"commit-graph has non-zero generation number for commit %s, but zero elsewhere\",\n+\t\t\t\t oid_to_hex(&cur_oid));\n+\n+\t\tif (generation_zero == GENERATION_ZERO_EXISTS)\n+\t\t\tcontinue;\n+\n+\t\t/*\n+\t\t * If one of our parents has generation GENERATION_NUMBER_MAX, then\n+\t\t * our generation is also GENERATION_NUMBER_MAX. Decrement to avoid\n+\t\t * extra logic in the following condition.\n+\t\t */\n+\t\tif (max_generation == GENERATION_NUMBER_MAX)\n+\t\t\tmax_generation--;\n+\n+\t\tif (graph_commit->generation != max_generation + 1)\n+\t\t\tgraph_report(\"commit-graph generation for commit %s is %u != %u\",\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t graph_commit->generation,\n+\t\t\t\t max_generation + 1);\n \t}\n \n \treturn verify_commit_graph_error;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 9a3481c30f..5b75c4dca2 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -272,6 +272,7 @@ GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n GRAPH_BYTE_COMMIT_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN))\n GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4))\n GRAPH_BYTE_COMMIT_WRONG_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3))\n+GRAPH_BYTE_COMMIT_GENERATION=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 11))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -366,4 +367,14 @@ test_expect_success 'detect wrong parent' '\n \t\t\"commit-graph parent for\"\n '\n \n+test_expect_success 'detect incorrect generation number' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_GENERATION \"\\070\" \\\n+\t\t\"generation for commit\"\n+'\n+\n+test_expect_success 'detect incorrect generation number' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_GENERATION \"\\01\" \\\n+\t\t\"non-zero generation number\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349712","messageId":"20180608135548.216405-9-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 08/21] commit-graph: verify required chunks are present","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:19Z","receivedAt":"2018-06-08T13:57:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file requires the following three chunks:\n\n* OID Fanout\n* OID Lookup\n* Commit Data\n\nIf any of these are missing, then the 'verify' subcommand should\nreport a failure. This includes the chunk IDs malformed or the\nchunk count is truncated.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 9 +++++++++\n t/t5318-commit-graph.sh | 29 +++++++++++++++++++++++++++++\n 2 files changed, 38 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 22ef696e18..f30b4ccee9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -848,5 +848,14 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\treturn 1;\n \t}\n \n+\tverify_commit_graph_error = 0;\n+\n+\tif (!g->chunk_oid_fanout)\n+\t\tgraph_report(\"commit-graph is missing the OID Fanout chunk\");\n+\tif (!g->chunk_oid_lookup)\n+\t\tgraph_report(\"commit-graph is missing the OID Lookup chunk\");\n+\tif (!g->chunk_commit_data)\n+\t\tgraph_report(\"commit-graph is missing the Commit Data chunk\");\n+\n \treturn verify_commit_graph_error;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 8f96e2636c..c03792a8ed 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -249,6 +249,15 @@ test_expect_success 'git commit-graph verify' '\n \n GRAPH_BYTE_VERSION=4\n GRAPH_BYTE_HASH=5\n+GRAPH_BYTE_CHUNK_COUNT=6\n+GRAPH_CHUNK_LOOKUP_OFFSET=8\n+GRAPH_CHUNK_LOOKUP_WIDTH=12\n+GRAPH_CHUNK_LOOKUP_ROWS=5\n+GRAPH_BYTE_OID_FANOUT_ID=$GRAPH_CHUNK_LOOKUP_OFFSET\n+GRAPH_BYTE_OID_LOOKUP_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n+\t\t\t 1 * $GRAPH_CHUNK_LOOKUP_WIDTH))\n+GRAPH_BYTE_COMMIT_DATA_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n+\t\t\t 2 * $GRAPH_CHUNK_LOOKUP_WIDTH))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -283,4 +292,24 @@ test_expect_success 'detect bad hash version' '\n \t\t\"hash version\"\n '\n \n+test_expect_success 'detect low chunk count' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_CHUNK_COUNT \"\\02\" \\\n+\t\t\"missing the .* chunk\"\n+'\n+\n+test_expect_success 'detect missing OID fanout chunk' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_FANOUT_ID \"\\0\" \\\n+\t\t\"missing the OID Fanout chunk\"\n+'\n+\n+test_expect_success 'detect missing OID lookup chunk' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_LOOKUP_ID \"\\0\" \\\n+\t\t\"missing the OID Lookup chunk\"\n+'\n+\n+test_expect_success 'detect missing commit data chunk' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_DATA_ID \"\\0\" \\\n+\t\t\"missing the Commit Data chunk\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349713","messageId":"20180608135548.216405-8-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 07/21] commit-graph: verify catches corrupt signature","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:18Z","receivedAt":"2018-06-08T13:57:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is the first of several commits that add a test to check that\n'git commit-graph verify' catches corruption in the commit-graph\nfile. The first test checks that the command catches an error in\nthe file signature. This is a check that exists in the existing\ncommit-graph reading code.\n\nAdd a helper method 'corrupt_graph_and_verify' to the test script\nt5318-commit-graph.sh. This helper corrupts the commit-graph file\nat a certain location, runs 'git commit-graph verify', and reports\nthe output to the 'err' file. This data is filtered to remove the\nlines added by 'test_must_fail' when the test is run verbosely.\nThen, the output is checked to contain a specific error message.\n\nMost messages from 'git commit-graph verify' will not be marked\nfor translation. There will be one exception: the message that\nreports an invalid checksum will be marked for translation, as that\nis the only message that is intended for a typical user.\n\nHelped-by: Szeder Gábor \nSigned-off-by: Derrick Stolee \n---\n t/t5318-commit-graph.sh | 43 +++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 43 insertions(+)\n\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 6ca451dfd2..8f96e2636c 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -235,9 +235,52 @@ test_expect_success 'perform fast-forward merge in full repo' '\n \ttest_cmp expect output\n '\n \n+# the verify tests below expect the commit-graph to contain\n+# exactly the commits reachable from the commits/8 branch.\n+# If the file changes the set of commits in the list, then the\n+# offsets into the binary file will result in different edits\n+# and the tests will likely break.\n+\n test_expect_success 'git commit-graph verify' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit rev-parse commits/8 | git commit-graph write --stdin-commits &&\n \tgit commit-graph verify >output\n '\n \n+GRAPH_BYTE_VERSION=4\n+GRAPH_BYTE_HASH=5\n+\n+# usage: corrupt_graph_and_verify \n+# Manipulates the commit-graph file at the position\n+# by inserting the data, then runs 'git commit-graph verify'\n+# and places the output in the file 'err'. Test 'err' for\n+# the given string.\n+corrupt_graph_and_verify() {\n+\tpos=$1\n+\tdata=\"${2:-\\0}\"\n+\tgrepstr=$3\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n+\tcp $objdir/info/commit-graph commit-graph-backup &&\n+\tprintf \"$data\" | dd of=\"$objdir/info/commit-graph\" bs=1 seek=\"$pos\" conv=notrunc &&\n+\ttest_must_fail git commit-graph verify 2>test_err &&\n+\tgrep -v \"^+\" test_err >err\n+\ttest_i18ngrep \"$grepstr\" err\n+}\n+\n+test_expect_success 'detect bad signature' '\n+\tcorrupt_graph_and_verify 0 \"\\0\" \\\n+\t\t\"graph signature\"\n+'\n+\n+test_expect_success 'detect bad version' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\02\" \\\n+\t\t\"graph version\"\n+'\n+\n+test_expect_success 'detect bad hash version' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_HASH \"\\02\" \\\n+\t\t\"hash version\"\n+'\n+\n test_done\n-- \n2.18.0.rc1\n\n"},{"id":"349714","messageId":"20180608135548.216405-5-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 04/21] commit: force commit to parse from object database","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:15Z","receivedAt":"2018-06-08T13:57:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In anticipation of verifying commit-graph file contents against the\nobject database, create parse_commit_internal() to allow side-stepping\nthe commit-graph file and parse directly from the object database.\n\nDue to the use of generation numbers, this method should not be called\nunless the intention is explicit in avoiding commits from the\ncommit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 9 +++++++--\n commit.h | 1 +\n 2 files changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex d53dc16d72..720c6acddf 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -418,7 +418,7 @@ int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long s\n \treturn 0;\n }\n \n-int parse_commit_gently(struct commit *item, int quiet_on_missing)\n+int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph)\n {\n \tenum object_type type;\n \tvoid *buffer;\n@@ -429,7 +429,7 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \t\treturn -1;\n \tif (item->object.parsed)\n \t\treturn 0;\n-\tif (parse_commit_in_graph(item))\n+\tif (use_commit_graph && parse_commit_in_graph(item))\n \t\treturn 0;\n \tbuffer = read_object_file(&item->object.oid, &type, &size);\n \tif (!buffer)\n@@ -450,6 +450,11 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \treturn ret;\n }\n \n+int parse_commit_gently(struct commit *item, int quiet_on_missing)\n+{\n+\treturn parse_commit_internal(item, quiet_on_missing, 1);\n+}\n+\n void parse_commit_or_die(struct commit *item)\n {\n \tif (parse_commit(item))\ndiff --git a/commit.h b/commit.h\nindex 3ad07c2e3d..7e0f273720 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -77,6 +77,7 @@ struct commit *lookup_commit_reference_by_name(const char *name);\n struct commit *lookup_commit_or_die(const struct object_id *oid, const char *ref_name);\n \n int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph);\n+int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph);\n int parse_commit_gently(struct commit *item, int quiet_on_missing);\n static inline int parse_commit(struct commit *item)\n {\n-- \n2.18.0.rc1\n\n"},{"id":"349715","messageId":"20180608135548.216405-2-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v6 01/21] commit-graph: UNLEAK before die()","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-06-08T13:56:13Z","receivedAt":"2018-06-08T13:57:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 37420ae0fd..f0875b8bf3 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -51,8 +51,11 @@ static int graph_read(int argc, const char **argv)\n \tgraph_name = get_commit_graph_filename(opts.obj_dir);\n \tgraph = load_commit_graph_one(graph_name);\n \n-\tif (!graph)\n+\tif (!graph) {\n+\t\tUNLEAK(graph_name);\n \t\tdie(\"graph file %s does not exist\", graph_name);\n+\t}\n+\n \tFREE_AND_NULL(graph_name);\n \n \tprintf(\"header: %08x %d %d %d %d\\n\",\n-- \n2.18.0.rc1\n\n"},{"id":"349717","messageId":"CANQwDwdRUFnqYva+mvPh8fijcYNGNDRac857aHMhzHUOLeASpw@mail.gmail.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"Re: [PATCH v6 00/21] Integrate commit-graph into 'fsck' and 'gc'","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2018-06-08T15:05:50Z","receivedAt":"2018-06-08T15:06:32Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Fri, 8 Jun 2018 at 15:56, Derrick Stolee wrote:\n\n> [..], the following\n> diff occurs from the previous patch:\n[...]\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index b24e8b6689..9a0661983c 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -33,8 +33,8 @@ test_expect_success 'create commits and repack' '\n> '\n>\n> graph_git_two_modes() {\n> - git -c core.commitGraph=true $1 >output\n> - git -c core.commitGraph=false $1 >expect\n> + git -c core.graph=true $1 >output\n> + git -c core.graph=false $1 >expect\n> test_cmp output expect\n> }\n\nIt seems that you have accidentally removed the fix from previous version.\nIt needs to be core.commitGraph, not core.graph.\n\nBest,\n-- \nJakub Narębski\n"},{"id":"349718","messageId":"2d50791e-f957-077e-a5a8-b2a8882f90c5@gmail.com","threadId":"48647","inReplyTo":"CANQwDwdRUFnqYva+mvPh8fijcYNGNDRac857aHMhzHUOLeASpw@mail.gmail.com","subject":"Re: [PATCH v6 00/21] Integrate commit-graph into 'fsck' and 'gc'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-08T15:11:04Z","receivedAt":"2018-06-08T15:11:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/8/2018 11:05 AM, Jakub Narębski wrote:\n> On Fri, 8 Jun 2018 at 15:56, Derrick Stolee wrote:\n>\n>> [..], the following\n>> diff occurs from the previous patch:\n> [...]\n>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n>> index b24e8b6689..9a0661983c 100755\n>> --- a/t/t5318-commit-graph.sh\n>> +++ b/t/t5318-commit-graph.sh\n>> @@ -33,8 +33,8 @@ test_expect_success 'create commits and repack' '\n>> '\n>>\n>> graph_git_two_modes() {\n>> - git -c core.commitGraph=true $1 >output\n>> - git -c core.commitGraph=false $1 >expect\n>> + git -c core.graph=true $1 >output\n>> + git -c core.graph=false $1 >expect\n>> test_cmp output expect\n>> }\n> It seems that you have accidentally removed the fix from previous version.\n> It needs to be core.commitGraph, not core.graph.\n>\n>\n\nI didn't rebase the fix that I sent as a separate patch [1] (we want \nthat change applied to 'master' while this one targets topics in 'next' \nand 'pu'). So this specific diff is unfortunate noise.\n\nThanks!\n-Stolee\n\n[1] \nhttps://public-inbox.org/git/20180604123906.136417-1-dstolee@microsoft.com/\n     [PATCH] t5318-commit-graph.sh: use core.commitGraph\n"},{"id":"349746","messageId":"20180608222414.11306-1-szeder.dev@gmail.com","threadId":"48647","inReplyTo":"20180608135548.216405-21-dstolee@microsoft.com","subject":"Re: [PATCH v6 20/21] gc: automatically write commit-graph files","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-06-08T22:24:14Z","receivedAt":"2018-06-08T22:24:35Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"\n> The commit-graph file is a very helpful feature for speeding up git\n> operations. In order to make it more useful, make it possible to\n> write the commit-graph file during standard garbage collection\n> operations.\n> \n> Add a 'gc.commitGraph' config setting that triggers writing a\n> commit-graph file after any non-trivial 'git gc' command. Defaults to\n> false while the commit-graph feature matures. We specifically do not\n> want to have this on by default until the commit-graph feature is fully\n> integrated with history-modifying features like shallow clones.\n\nSo I played around with an earlier version of this patch series a\nwhile ago... and as I looked into my gitconfig today I was surprised\nto have both 'core.commitGraph' and 'gc.commitGraph' config variables\nset. When I looked into it I came across this email from Ævar:\n\n https://public-inbox.org/git/87fu3peni2.fsf@evledraar.gmail.com/\n\n > Other than the question if 'gc.commitGraph' and 'core.commitGraph'\n > should be independent config variables, and the exact wording of the\n > git-gc docs, it looks good to me.\n\n Sans doc errors you pointed out in other places (you need to set\n core.commitGraph so it's read at all), I think it's very useful to have\n these split up. It's simliar to pack.useBitmaps & pack.writeBitmaps.\n\nI think the comparison with pack bitmaps makes a lot of sense and I\nhave to say that I really like those 'useBitmaps' and 'writeBitmaps'\nvariable names, because it's clear right away which one is which,\nwithout consulting the documentation. I think having 'useCommitGraph'\nand 'writeCommitGraph' variables would be a lot better than the same\nvariable name in two different sections, and I'm sure that then I\nwouldn't have been caught off guard.\n\nYeah, I know, my timing sucks, with 'core.commitGraph' already out\nthere in the -rc releases... sorry.\n\n"},{"id":"349990","messageId":"xmqqtvq89c6u.fsf@gitster-ct.c.googlers.com","threadId":"48647","inReplyTo":"20180608135548.216405-13-dstolee@microsoft.com","subject":"Re: [PATCH v6 12/21] commit-graph: verify parent list","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-06-12T16:55:37Z","receivedAt":"2018-06-12T16:55:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 2b9214bc83..9a3481c30f 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -269,6 +269,9 @@ GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 8))\n> GRAPH_BYTE_OID_LOOKUP_MISSING=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 4 + 10))\n> GRAPH_COMMIT_DATA_OFFSET=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * $NUM_COMMITS))\n> GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n> +GRAPH_BYTE_COMMIT_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN))\n> +GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4))\n> +GRAPH_BYTE_COMMIT_WRONG_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3))\n\nThere seems to depend on having GRAPH_COMMIT_DATA_OFFSET in this\nfile, but I do not think 'next' or sb/object-store-alloc has one,\nand I do not think I saw an earlier patch in this series to add such\na line.\n\nIt appears that many messages in this series are sent in\nquoted-printable when they do not have to, which made me say fuzzy\nthings like \"I do not think I saw\" in the previous paragraph,\ninstead of being able to be definitive with a simple \"grep\" result.\nPlease try to see if you can find a way to tell your MUA not to do\nunnecessary mail munging, if you can do so easily.\n\nThe worst one was 07/21, which came in base64 and contained a patch\nto this file in MS-DOS line ending convention, which made it\nimpossible to apply.\n\n> \n> # usage: corrupt_graph_and_verify \n> # Manipulates the commit-graph file at the position\n> @@ -348,4 +351,19 @@ test_expect_success 'detect incorrect tree OID' '\n> \t\t\"root tree OID for commit\"\n> '\n> \n> +test_expect_success 'detect incorrect parent int-id' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_PARENT \"\\01\" \\\n> +\t\t\"invalid parent\"\n> +'\n> +\n> +test_expect_success 'detect extra parent int-id' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_EXTRA_PARENT \"\\00\" \\\n> +\t\t\"is too long\"\n> +'\n> +\n> +test_expect_success 'detect wrong parent' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_WRONG_PARENT \"\\01\" \\\n> +\t\t\"commit-graph parent for\"\n> +'\n> +\n> test_done\n"},{"id":"350832","messageId":"abe65bdb-92b1-2932-3d15-3ddfae7e75c3@gmail.com","threadId":"48647","inReplyTo":"20180608222414.11306-1-szeder.dev@gmail.com","subject":"Re: [PATCH v6 20/21] gc: automatically write commit-graph files","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-25T14:48:55Z","receivedAt":"2018-06-25T14:49:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/8/2018 6:24 PM, SZEDER Gábor wrote:\n>> The commit-graph file is a very helpful feature for speeding up git\n>> operations. In order to make it more useful, make it possible to\n>> write the commit-graph file during standard garbage collection\n>> operations.\n>>\n>> Add a 'gc.commitGraph' config setting that triggers writing a\n>> commit-graph file after any non-trivial 'git gc' command. Defaults to\n>> false while the commit-graph feature matures. We specifically do not\n>> want to have this on by default until the commit-graph feature is fully\n>> integrated with history-modifying features like shallow clones.\n> So I played around with an earlier version of this patch series a\n> while ago... and as I looked into my gitconfig today I was surprised\n> to have both 'core.commitGraph' and 'gc.commitGraph' config variables\n> set. When I looked into it I came across this email from Ævar:\n>\n> https://public-inbox.org/git/87fu3peni2.fsf@evledraar.gmail.com/\n>\n> > Other than the question if 'gc.commitGraph' and 'core.commitGraph'\n> > should be independent config variables, and the exact wording of the\n> > git-gc docs, it looks good to me.\n>\n> Sans doc errors you pointed out in other places (you need to set\n> core.commitGraph so it's read at all), I think it's very useful to have\n> these split up. It's simliar to pack.useBitmaps & pack.writeBitmaps.\n>\n> I think the comparison with pack bitmaps makes a lot of sense and I\n> have to say that I really like those 'useBitmaps' and 'writeBitmaps'\n> variable names, because it's clear right away which one is which,\n> without consulting the documentation. I think having 'useCommitGraph'\n> and 'writeCommitGraph' variables would be a lot better than the same\n> variable name in two different sections, and I'm sure that then I\n> wouldn't have been caught off guard.\n\nSorry for the late reply. Maybe 'gc.writeCommitGraph' makes it clear \nthat the setting enables writing the commit-graph during  a 'git gc'. Is \nthat more clear?\n\nThanks,\n-Stolee\n"},{"id":"351087","messageId":"20180627132447.142473-1-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180608135548.216405-1-dstolee@microsoft.com","subject":"[PATCH v7 00/22] Integrate commit-graph into 'fsck' and 'gc'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:25Z","receivedAt":"2018-06-27T13:24:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThis v7 has very little material difference from the previous version.\nThe only changes are:\n\n* Rename 'gc.commitGraph' to 'gc.writeCommitGraph'\n\n* Add 't5318-commit-graph.sh: use core.commitGraph' patch that was\n dropped.\n\n* Rebase onto latest master, which appears to fix any issues merging\n into 'next' and 'pu'.\n\nI'm sending this from my gmail account to hopefully avoid any message\nmunging that happened in v6.\n\nYou can see my commits on GitHub [1].\n\n[1] https://github.com/gitgitgadget/git/pull/6\n\nThanks,\n-Stolee\n\nDerrick Stolee (22):\n t5318-commit-graph.sh: use core.commitGraph\n commit-graph: UNLEAK before die()\n commit-graph: fix GRAPH_MIN_SIZE\n commit-graph: parse commit from chosen graph\n commit: force commit to parse from object database\n commit-graph: load a root tree from specific graph\n commit-graph: add 'verify' subcommand\n commit-graph: verify catches corrupt signature\n commit-graph: verify required chunks are present\n commit-graph: verify corrupt OID fanout and lookup\n commit-graph: verify objects exist\n commit-graph: verify root tree OIDs\n commit-graph: verify parent list\n commit-graph: verify generation number\n commit-graph: verify commit date\n commit-graph: test for corrupted octopus edge\n commit-graph: verify contents match checksum\n fsck: verify commit-graph\n commit-graph: use string-list API for input\n commit-graph: add '--reachable' option\n gc: automatically write commit-graph files\n commit-graph: update design document\n\n Documentation/config.txt | 9 +-\n Documentation/git-commit-graph.txt | 14 +-\n Documentation/git-fsck.txt | 3 +\n Documentation/git-gc.txt | 4 +\n Documentation/technical/commit-graph.txt | 22 --\n builtin/commit-graph.c | 99 ++++++---\n builtin/fsck.c | 21 ++\n builtin/gc.c | 6 +\n commit-graph.c | 249 +++++++++++++++++++++--\n commit-graph.h | 11 +-\n commit.c | 10 +-\n commit.h | 1 +\n t/t5318-commit-graph.sh | 205 ++++++++++++++++++-\n 13 files changed, 572 insertions(+), 82 deletions(-)\n\n\nbase-commit: ed843436dd4924c10669820cc73daf50f0b4dabd\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351088","messageId":"20180627132447.142473-2-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 01/22] t5318-commit-graph.sh: use core.commitGraph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:26Z","receivedAt":"2018-06-27T13:25:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph tests should be checking that normal Git operations\nsucceed and have matching output with and without the commit-graph\nfeature enabled. However, the test was toggling 'core.graph' instead\nof the correct 'core.commitGraph' variable.\n\nSigned-off-by: Derrick Stolee \n---\n\nJunio,\n\nI sent this patch as a one-off a while ago, but it seems it was dropped.\nI'm adding it back here so we don't forget it.\n\n-Stolee\n\n t/t5318-commit-graph.sh | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 77d85aefe7..59d0be2877 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -28,8 +28,8 @@ test_expect_success 'create commits and repack' '\n '\n \n graph_git_two_modes() {\n-\tgit -c core.graph=true $1 >output\n-\tgit -c core.graph=false $1 >expect\n+\tgit -c core.commitGraph=true $1 >output\n+\tgit -c core.commitGraph=false $1 >expect\n \ttest_cmp output expect\n }\n \n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351089","messageId":"20180627132447.142473-3-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 02/22] commit-graph: UNLEAK before die()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:27Z","receivedAt":"2018-06-27T13:25:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 37420ae0fd..f0875b8bf3 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -51,8 +51,11 @@ static int graph_read(int argc, const char **argv)\n \tgraph_name = get_commit_graph_filename(opts.obj_dir);\n \tgraph = load_commit_graph_one(graph_name);\n \n-\tif (!graph)\n+\tif (!graph) {\n+\t\tUNLEAK(graph_name);\n \t\tdie(\"graph file %s does not exist\", graph_name);\n+\t}\n+\n \tFREE_AND_NULL(graph_name);\n \n \tprintf(\"header: %08x %d %d %d %d\\n\",\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351090","messageId":"20180627132447.142473-4-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 03/22] commit-graph: fix GRAPH_MIN_SIZE","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:28Z","receivedAt":"2018-06-27T13:25:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The GRAPH_MIN_SIZE macro should be the smallest size of a parsable\ncommit-graph file. However, the minimum number of chunks was wrong.\nIt is possible to write a commit-graph file with zero commits, and\nthat violates this macro's value.\n\nRewrite the macro, and use extra macros to better explain the magic\nconstants.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex b63a1fc85e..f83f6d2373 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -35,10 +35,11 @@\n \n #define GRAPH_LAST_EDGE 0x80000000\n \n+#define GRAPH_HEADER_SIZE 8\n #define GRAPH_FANOUT_SIZE (4 * 256)\n #define GRAPH_CHUNKLOOKUP_WIDTH 12\n-#define GRAPH_MIN_SIZE (5 * GRAPH_CHUNKLOOKUP_WIDTH + GRAPH_FANOUT_SIZE + \\\n-\t\t\tGRAPH_OID_LEN + 8)\n+#define GRAPH_MIN_SIZE (GRAPH_HEADER_SIZE + 4 * GRAPH_CHUNKLOOKUP_WIDTH \\\n+\t\t\t+ GRAPH_FANOUT_SIZE + GRAPH_OID_LEN)\n \n char *get_commit_graph_filename(const char *obj_dir)\n {\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351091","messageId":"20180627132447.142473-6-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 05/22] commit: force commit to parse from object database","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:30Z","receivedAt":"2018-06-27T13:25:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In anticipation of verifying commit-graph file contents against the\nobject database, create parse_commit_internal() to allow side-stepping\nthe commit-graph file and parse directly from the object database.\n\nDue to the use of generation numbers, this method should not be called\nunless the intention is explicit in avoiding commits from the\ncommit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 10 ++++++++--\n commit.h | 1 +\n 2 files changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex 0c3b75aeff..598cf21cee 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -418,7 +418,7 @@ int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long s\n \treturn 0;\n }\n \n-int parse_commit_gently(struct commit *item, int quiet_on_missing)\n+int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph)\n {\n \tenum object_type type;\n \tvoid *buffer;\n@@ -429,7 +429,7 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \t\treturn -1;\n \tif (item->object.parsed)\n \t\treturn 0;\n-\tif (parse_commit_in_graph(item))\n+\tif (use_commit_graph && parse_commit_in_graph(item))\n \t\treturn 0;\n \tbuffer = read_object_file(&item->object.oid, &type, &size);\n \tif (!buffer)\n@@ -441,6 +441,7 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \t\treturn error(\"Object %s not a commit\",\n \t\t\t oid_to_hex(&item->object.oid));\n \t}\n+\n \tret = parse_commit_buffer(item, buffer, size, 0);\n \tif (save_commit_buffer && !ret) {\n \t\tset_commit_buffer(item, buffer, size);\n@@ -450,6 +451,11 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \treturn ret;\n }\n \n+int parse_commit_gently(struct commit *item, int quiet_on_missing)\n+{\n+\treturn parse_commit_internal(item, quiet_on_missing, 1);\n+}\n+\n void parse_commit_or_die(struct commit *item)\n {\n \tif (parse_commit(item))\ndiff --git a/commit.h b/commit.h\nindex 3ad07c2e3d..7e0f273720 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -77,6 +77,7 @@ struct commit *lookup_commit_reference_by_name(const char *name);\n struct commit *lookup_commit_or_die(const struct object_id *oid, const char *ref_name);\n \n int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph);\n+int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph);\n int parse_commit_gently(struct commit *item, int quiet_on_missing);\n static inline int parse_commit(struct commit *item)\n {\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351092","messageId":"20180627132447.142473-5-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 04/22] commit-graph: parse commit from chosen graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:29Z","receivedAt":"2018-06-27T13:25:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Before verifying a commit-graph file against the object database, we\nneed to parse all commits from the given commit-graph file. Create\nparse_commit_in_graph_one() to target a given struct commit_graph.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 18 +++++++++++++++---\n 1 file changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f83f6d2373..e77b19971d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -314,7 +314,7 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \t}\n }\n \n-int parse_commit_in_graph(struct commit *item)\n+static int parse_commit_in_graph_one(struct commit_graph *g, struct commit *item)\n {\n \tuint32_t pos;\n \n@@ -322,9 +322,21 @@ int parse_commit_in_graph(struct commit *item)\n \t\treturn 0;\n \tif (item->object.parsed)\n \t\treturn 1;\n+\n+\tif (find_commit_in_graph(item, g, &pos))\n+\t\treturn fill_commit_in_graph(item, g, pos);\n+\n+\treturn 0;\n+}\n+\n+int parse_commit_in_graph(struct commit *item)\n+{\n+\tif (!core_commit_graph)\n+\t\treturn 0;\n+\n \tprepare_commit_graph();\n-\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n-\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n+\tif (commit_graph)\n+\t\treturn parse_commit_in_graph_one(commit_graph, item);\n \treturn 0;\n }\n \n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351093","messageId":"20180627132447.142473-7-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 06/22] commit-graph: load a root tree from specific graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:31Z","receivedAt":"2018-06-27T13:25:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When lazy-loading a tree for a commit, it will be important to select\nthe tree from a specific struct commit_graph. Create a new method that\nspecifies the commit-graph file and use that in\nget_commit_tree_in_graph().\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 12 +++++++++---\n 1 file changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex e77b19971d..9e228d3bb5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -362,14 +362,20 @@ static struct tree *load_tree_for_commit(struct commit_graph *g, struct commit *\n \treturn c->maybe_tree;\n }\n \n-struct tree *get_commit_tree_in_graph(const struct commit *c)\n+static struct tree *get_commit_tree_in_graph_one(struct commit_graph *g,\n+\t\t\t\t\t\t const struct commit *c)\n {\n \tif (c->maybe_tree)\n \t\treturn c->maybe_tree;\n \tif (c->graph_pos == COMMIT_NOT_FROM_GRAPH)\n-\t\tBUG(\"get_commit_tree_in_graph called from non-commit-graph commit\");\n+\t\tBUG(\"get_commit_tree_in_graph_one called from non-commit-graph commit\");\n+\n+\treturn load_tree_for_commit(g, (struct commit *)c);\n+}\n \n-\treturn load_tree_for_commit(commit_graph, (struct commit *)c);\n+struct tree *get_commit_tree_in_graph(const struct commit *c)\n+{\n+\treturn get_commit_tree_in_graph_one(commit_graph, c);\n }\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351094","messageId":"20180627132447.142473-9-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 08/22] commit-graph: verify catches corrupt signature","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:33Z","receivedAt":"2018-06-27T13:25:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is the first of several commits that add a test to check that\n'git commit-graph verify' catches corruption in the commit-graph\nfile. The first test checks that the command catches an error in\nthe file signature. This is a check that exists in the existing\ncommit-graph reading code.\n\nAdd a helper method 'corrupt_graph_and_verify' to the test script\nt5318-commit-graph.sh. This helper corrupts the commit-graph file\nat a certain location, runs 'git commit-graph verify', and reports\nthe output to the 'err' file. This data is filtered to remove the\nlines added by 'test_must_fail' when the test is run verbosely.\nThen, the output is checked to contain a specific error message.\n\nMost messages from 'git commit-graph verify' will not be marked\nfor translation. There will be one exception: the message that\nreports an invalid checksum will be marked for translation, as that\nis the only message that is intended for a typical user.\n\nHelped-by: Szeder Gábor \nSigned-off-by: Derrick Stolee \n---\n t/t5318-commit-graph.sh | 43 +++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 43 insertions(+)\n\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 0830ef9fdd..c0c1ff09b9 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -235,9 +235,52 @@ test_expect_success 'perform fast-forward merge in full repo' '\n \ttest_cmp expect output\n '\n \n+# the verify tests below expect the commit-graph to contain\n+# exactly the commits reachable from the commits/8 branch.\n+# If the file changes the set of commits in the list, then the\n+# offsets into the binary file will result in different edits\n+# and the tests will likely break.\n+\n test_expect_success 'git commit-graph verify' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit rev-parse commits/8 | git commit-graph write --stdin-commits &&\n \tgit commit-graph verify >output\n '\n \n+GRAPH_BYTE_VERSION=4\n+GRAPH_BYTE_HASH=5\n+\n+# usage: corrupt_graph_and_verify \n+# Manipulates the commit-graph file at the position\n+# by inserting the data, then runs 'git commit-graph verify'\n+# and places the output in the file 'err'. Test 'err' for\n+# the given string.\n+corrupt_graph_and_verify() {\n+\tpos=$1\n+\tdata=\"${2:-\\0}\"\n+\tgrepstr=$3\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n+\tcp $objdir/info/commit-graph commit-graph-backup &&\n+\tprintf \"$data\" | dd of=\"$objdir/info/commit-graph\" bs=1 seek=\"$pos\" conv=notrunc &&\n+\ttest_must_fail git commit-graph verify 2>test_err &&\n+\tgrep -v \"^+\" test_err >err\n+\ttest_i18ngrep \"$grepstr\" err\n+}\n+\n+test_expect_success 'detect bad signature' '\n+\tcorrupt_graph_and_verify 0 \"\\0\" \\\n+\t\t\"graph signature\"\n+'\n+\n+test_expect_success 'detect bad version' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\02\" \\\n+\t\t\"graph version\"\n+'\n+\n+test_expect_success 'detect bad hash version' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_HASH \"\\02\" \\\n+\t\t\"hash version\"\n+'\n+\n test_done\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351095","messageId":"20180627132447.142473-8-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 07/22] commit-graph: add 'verify' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:32Z","receivedAt":"2018-06-27T13:25:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"If the commit-graph file becomes corrupt, we need a way to verify\nthat its contents match the object database. In the manner of\n'git fsck' we will implement a 'git commit-graph verify' subcommand\nto report all issues with the file.\n\nAdd the 'verify' subcommand to the 'commit-graph' builtin and its\ndocumentation. The subcommand is currently a no-op except for\nloading the commit-graph into memory, which may trigger run-time\nerrors that would be caught by normal use. Add a simple test that\nensures the command returns a zero error code.\n\nIf no commit-graph file exists, this is an acceptable state. Do\nnot report any errors.\n\nHelped-by: Ramsay Jones \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 6 +++++\n builtin/commit-graph.c | 39 ++++++++++++++++++++++++++++++\n commit-graph.c | 23 ++++++++++++++++++\n commit-graph.h | 3 +++\n t/t5318-commit-graph.sh | 10 ++++++++\n 5 files changed, 81 insertions(+)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 4c97b555cc..a222cfab08 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git commit-graph read' [--object-dir ]\n+'git commit-graph verify' [--object-dir ]\n 'git commit-graph write' [--object-dir ]\n \n \n@@ -52,6 +53,11 @@ existing commit-graph file.\n Read a graph file given by the commit-graph file and output basic\n details about the graph file. Used for debugging purposes.\n \n+'verify'::\n+\n+Read the commit-graph file and verify its contents against the object\n+database. Used to check for corrupted data.\n+\n \n EXAMPLES\n --------\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex f0875b8bf3..9d108f43a9 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -3,15 +3,22 @@\n #include \"dir.h\"\n #include \"lockfile.h\"\n #include \"parse-options.h\"\n+#include \"repository.h\"\n #include \"commit-graph.h\"\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n+\tN_(\"git commit-graph verify [--object-dir ]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n+static const char * const builtin_commit_graph_verify_usage[] = {\n+\tN_(\"git commit-graph verify [--object-dir ]\"),\n+\tNULL\n+};\n+\n static const char * const builtin_commit_graph_read_usage[] = {\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tNULL\n@@ -29,6 +36,36 @@ static struct opts_commit_graph {\n \tint append;\n } opts;\n \n+\n+static int graph_verify(int argc, const char **argv)\n+{\n+\tstruct commit_graph *graph = NULL;\n+\tchar *graph_name;\n+\n+\tstatic struct option builtin_commit_graph_verify_options[] = {\n+\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n+\t\t\t N_(\"dir\"),\n+\t\t\t N_(\"The object directory to store the graph\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t builtin_commit_graph_verify_options,\n+\t\t\t builtin_commit_graph_verify_usage, 0);\n+\n+\tif (!opts.obj_dir)\n+\t\topts.obj_dir = get_object_directory();\n+\n+\tgraph_name = get_commit_graph_filename(opts.obj_dir);\n+\tgraph = load_commit_graph_one(graph_name);\n+\tFREE_AND_NULL(graph_name);\n+\n+\tif (!graph)\n+\t\treturn 0;\n+\n+\treturn verify_commit_graph(the_repository, graph);\n+}\n+\n static int graph_read(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n@@ -165,6 +202,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \tif (argc > 0) {\n \t\tif (!strcmp(argv[0], \"read\"))\n \t\t\treturn graph_read(argc, argv);\n+\t\tif (!strcmp(argv[0], \"verify\"))\n+\t\t\treturn graph_verify(argc, argv);\n \t\tif (!strcmp(argv[0], \"write\"))\n \t\t\treturn graph_write(argc, argv);\n \t}\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 9e228d3bb5..22ef696e18 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -827,3 +827,26 @@ void write_commit_graph(const char *obj_dir,\n \toids.alloc = 0;\n \toids.nr = 0;\n }\n+\n+static int verify_commit_graph_error;\n+\n+static void graph_report(const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tverify_commit_graph_error = 1;\n+\n+\tva_start(ap, fmt);\n+\tvfprintf(stderr, fmt, ap);\n+\tfprintf(stderr, \"\\n\");\n+\tva_end(ap);\n+}\n+\n+int verify_commit_graph(struct repository *r, struct commit_graph *g)\n+{\n+\tif (!g) {\n+\t\tgraph_report(\"no commit-graph file loaded\");\n+\t\treturn 1;\n+\t}\n+\n+\treturn verify_commit_graph_error;\n+}\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 96cccb10f3..4359812fb4 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -2,6 +2,7 @@\n #define COMMIT_GRAPH_H\n \n #include \"git-compat-util.h\"\n+#include \"repository.h\"\n \n char *get_commit_graph_filename(const char *obj_dir);\n \n@@ -53,4 +54,6 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tint nr_commits,\n \t\t\tint append);\n \n+int verify_commit_graph(struct repository *r, struct commit_graph *g);\n+\n #endif\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 59d0be2877..0830ef9fdd 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -11,6 +11,11 @@ test_expect_success 'setup full repo' '\n \tobjdir=\".git/objects\"\n '\n \n+test_expect_success 'verify graph with no graph file' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph verify\n+'\n+\n test_expect_success 'write graph with no packs' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write --object-dir . &&\n@@ -230,4 +235,9 @@ test_expect_success 'perform fast-forward merge in full repo' '\n \ttest_cmp expect output\n '\n \n+test_expect_success 'git commit-graph verify' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph verify >output\n+'\n+\n test_done\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351096","messageId":"20180627132447.142473-13-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 12/22] commit-graph: verify root tree OIDs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:37Z","receivedAt":"2018-06-27T13:25:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The 'verify' subcommand must compare the commit content parsed from the\ncommit-graph against the content in the object database. Use\nlookup_commit() and parse_commit_in_graph_one() to parse the commits\nfrom the graph and compare against a commit that is loaded separately\nand parsed directly from the object database.\n\nAdd checks for the root tree OID.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 17 ++++++++++++++++-\n t/t5318-commit-graph.sh | 7 +++++++\n 2 files changed, 23 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 00e89b71e9..5df18394f9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -866,6 +866,8 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\treturn verify_commit_graph_error;\n \n \tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct commit *graph_commit;\n+\n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n \t\tif (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n@@ -883,6 +885,11 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \n \t\t\tcur_fanout_pos++;\n \t\t}\n+\n+\t\tgraph_commit = lookup_commit(&cur_oid);\n+\t\tif (!parse_commit_in_graph_one(g, graph_commit))\n+\t\t\tgraph_report(\"failed to parse %s from commit-graph\",\n+\t\t\t\t oid_to_hex(&cur_oid));\n \t}\n \n \twhile (cur_fanout_pos < 256) {\n@@ -899,16 +906,24 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\treturn verify_commit_graph_error;\n \n \tfor (i = 0; i < g->num_commits; i++) {\n-\t\tstruct commit *odb_commit;\n+\t\tstruct commit *graph_commit, *odb_commit;\n \n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n+\t\tgraph_commit = lookup_commit(&cur_oid);\n \t\todb_commit = (struct commit *)create_object(r, cur_oid.hash, alloc_commit_node(r));\n \t\tif (parse_commit_internal(odb_commit, 0, 0)) {\n \t\t\tgraph_report(\"failed to parse %s from object database\",\n \t\t\t\t oid_to_hex(&cur_oid));\n \t\t\tcontinue;\n \t\t}\n+\n+\t\tif (oidcmp(&get_commit_tree_in_graph_one(g, graph_commit)->object.oid,\n+\t\t\t get_commit_tree_oid(odb_commit)))\n+\t\t\tgraph_report(\"root tree OID for commit %s in commit-graph is %s != %s\",\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t oid_to_hex(get_commit_tree_oid(graph_commit)),\n+\t\t\t\t oid_to_hex(get_commit_tree_oid(odb_commit)));\n \t}\n \n \treturn verify_commit_graph_error;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex a27ec97269..f258c6d5d0 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -267,6 +267,8 @@ GRAPH_BYTE_FANOUT2=$(($GRAPH_FANOUT_OFFSET + 4 * 255))\n GRAPH_OID_LOOKUP_OFFSET=$(($GRAPH_FANOUT_OFFSET + 4 * 256))\n GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 8))\n GRAPH_BYTE_OID_LOOKUP_MISSING=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 4 + 10))\n+GRAPH_COMMIT_DATA_OFFSET=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * $NUM_COMMITS))\n+GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -341,4 +343,9 @@ test_expect_success 'detect OID not in object database' '\n \t\t\"from object database\"\n '\n \n+test_expect_success 'detect incorrect tree OID' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_TREE \"\\01\" \\\n+\t\t\"root tree OID for commit\"\n+'\n+\n test_done\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351097","messageId":"20180627132447.142473-14-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 13/22] commit-graph: verify parent list","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:38Z","receivedAt":"2018-06-27T13:25:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file stores parents in a two-column portion of the\ncommit data chunk. If there is only one parent, then the second column\nstores 0xFFFFFFFF to indicate no second parent.\n\nThe 'verify' subcommand checks the parent list for the commit loaded\nfrom the commit-graph and the one parsed from the object database. Test\nthese checks for corrupt parents, too many parents, and wrong parents.\n\nAdd a boundary check to insert_parent_or_die() for when the parent\nposition value is out of range.\n\nThe octopus merge will be tested in a later commit.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 28 ++++++++++++++++++++++++++++\n t/t5318-commit-graph.sh | 18 ++++++++++++++++++\n 2 files changed, 46 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 5df18394f9..6d8d774eb0 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -244,6 +244,9 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n \tstruct commit *c;\n \tstruct object_id oid;\n \n+\tif (pos >= g->num_commits)\n+\t\tdie(\"invalid parent position %\"PRIu64, pos);\n+\n \thashcpy(oid.hash, g->chunk_oid_lookup + g->hash_len * pos);\n \tc = lookup_commit(&oid);\n \tif (!c)\n@@ -907,6 +910,7 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \n \tfor (i = 0; i < g->num_commits; i++) {\n \t\tstruct commit *graph_commit, *odb_commit;\n+\t\tstruct commit_list *graph_parents, *odb_parents;\n \n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n@@ -924,6 +928,30 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\t\t\t oid_to_hex(&cur_oid),\n \t\t\t\t oid_to_hex(get_commit_tree_oid(graph_commit)),\n \t\t\t\t oid_to_hex(get_commit_tree_oid(odb_commit)));\n+\n+\t\tgraph_parents = graph_commit->parents;\n+\t\todb_parents = odb_commit->parents;\n+\n+\t\twhile (graph_parents) {\n+\t\t\tif (odb_parents == NULL) {\n+\t\t\t\tgraph_report(\"commit-graph parent list for commit %s is too long\",\n+\t\t\t\t\t oid_to_hex(&cur_oid));\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (oidcmp(&graph_parents->item->object.oid, &odb_parents->item->object.oid))\n+\t\t\t\tgraph_report(\"commit-graph parent for %s is %s != %s\",\n+\t\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid),\n+\t\t\t\t\t oid_to_hex(&odb_parents->item->object.oid));\n+\n+\t\t\tgraph_parents = graph_parents->next;\n+\t\t\todb_parents = odb_parents->next;\n+\t\t}\n+\n+\t\tif (odb_parents != NULL)\n+\t\t\tgraph_report(\"commit-graph parent list for commit %s terminates early\",\n+\t\t\t\t oid_to_hex(&cur_oid));\n \t}\n \n \treturn verify_commit_graph_error;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex f258c6d5d0..b41c8f4d9b 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -269,6 +269,9 @@ GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 8))\n GRAPH_BYTE_OID_LOOKUP_MISSING=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 4 + 10))\n GRAPH_COMMIT_DATA_OFFSET=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * $NUM_COMMITS))\n GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n+GRAPH_BYTE_COMMIT_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN))\n+GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4))\n+GRAPH_BYTE_COMMIT_WRONG_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -348,4 +351,19 @@ test_expect_success 'detect incorrect tree OID' '\n \t\t\"root tree OID for commit\"\n '\n \n+test_expect_success 'detect incorrect parent int-id' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_PARENT \"\\01\" \\\n+\t\t\"invalid parent\"\n+'\n+\n+test_expect_success 'detect extra parent int-id' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_EXTRA_PARENT \"\\00\" \\\n+\t\t\"is too long\"\n+'\n+\n+test_expect_success 'detect wrong parent' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_WRONG_PARENT \"\\01\" \\\n+\t\t\"commit-graph parent for\"\n+'\n+\n test_done\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351098","messageId":"20180627132447.142473-15-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 14/22] commit-graph: verify generation number","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:39Z","receivedAt":"2018-06-27T13:25:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"While iterating through the commit parents, perform the generation\nnumber calculation and compare against the value stored in the\ncommit-graph.\n\nThe tests demonstrate that having a different set of parents affects\nthe generation number calculation, and this value propagates to\ndescendants. Hence, we drop the single-line condition on the output.\n\nSince Git will ship with the commit-graph feature without generation\nnumbers, we need to accept commit-graphs with all generation numbers\nequal to zero. In this case, ignore the generation number calculation.\n\nHowever, verify that we should never have a mix of zero and non-zero\ngeneration numbers. Create a test that sets one commit to generation\nzero and all following commits report a failure as they have non-zero\ngeneration in a file that contains generation number zero.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 34 ++++++++++++++++++++++++++++++++++\n t/t5318-commit-graph.sh | 11 +++++++++++\n 2 files changed, 45 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 6d8d774eb0..e0f71658da 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -846,10 +846,14 @@ static void graph_report(const char *fmt, ...)\n \tva_end(ap);\n }\n \n+#define GENERATION_ZERO_EXISTS 1\n+#define GENERATION_NUMBER_EXISTS 2\n+\n int verify_commit_graph(struct repository *r, struct commit_graph *g)\n {\n \tuint32_t i, cur_fanout_pos = 0;\n \tstruct object_id prev_oid, cur_oid;\n+\tint generation_zero = 0;\n \n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n@@ -911,6 +915,7 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \tfor (i = 0; i < g->num_commits; i++) {\n \t\tstruct commit *graph_commit, *odb_commit;\n \t\tstruct commit_list *graph_parents, *odb_parents;\n+\t\tuint32_t max_generation = 0;\n \n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n@@ -945,6 +950,9 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid),\n \t\t\t\t\t oid_to_hex(&odb_parents->item->object.oid));\n \n+\t\t\tif (graph_parents->item->generation > max_generation)\n+\t\t\t\tmax_generation = graph_parents->item->generation;\n+\n \t\t\tgraph_parents = graph_parents->next;\n \t\t\todb_parents = odb_parents->next;\n \t\t}\n@@ -952,6 +960,32 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\tif (odb_parents != NULL)\n \t\t\tgraph_report(\"commit-graph parent list for commit %s terminates early\",\n \t\t\t\t oid_to_hex(&cur_oid));\n+\n+\t\tif (!graph_commit->generation) {\n+\t\t\tif (generation_zero == GENERATION_NUMBER_EXISTS)\n+\t\t\t\tgraph_report(\"commit-graph has generation number zero for commit %s, but non-zero elsewhere\",\n+\t\t\t\t\t oid_to_hex(&cur_oid));\n+\t\t\tgeneration_zero = GENERATION_ZERO_EXISTS;\n+\t\t} else if (generation_zero == GENERATION_ZERO_EXISTS)\n+\t\t\tgraph_report(\"commit-graph has non-zero generation number for commit %s, but zero elsewhere\",\n+\t\t\t\t oid_to_hex(&cur_oid));\n+\n+\t\tif (generation_zero == GENERATION_ZERO_EXISTS)\n+\t\t\tcontinue;\n+\n+\t\t/*\n+\t\t * If one of our parents has generation GENERATION_NUMBER_MAX, then\n+\t\t * our generation is also GENERATION_NUMBER_MAX. Decrement to avoid\n+\t\t * extra logic in the following condition.\n+\t\t */\n+\t\tif (max_generation == GENERATION_NUMBER_MAX)\n+\t\t\tmax_generation--;\n+\n+\t\tif (graph_commit->generation != max_generation + 1)\n+\t\t\tgraph_report(\"commit-graph generation for commit %s is %u != %u\",\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t graph_commit->generation,\n+\t\t\t\t max_generation + 1);\n \t}\n \n \treturn verify_commit_graph_error;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex b41c8f4d9b..3128e19aef 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -272,6 +272,7 @@ GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n GRAPH_BYTE_COMMIT_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN))\n GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4))\n GRAPH_BYTE_COMMIT_WRONG_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3))\n+GRAPH_BYTE_COMMIT_GENERATION=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 11))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -366,4 +367,14 @@ test_expect_success 'detect wrong parent' '\n \t\t\"commit-graph parent for\"\n '\n \n+test_expect_success 'detect incorrect generation number' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_GENERATION \"\\070\" \\\n+\t\t\"generation for commit\"\n+'\n+\n+test_expect_success 'detect incorrect generation number' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_GENERATION \"\\01\" \\\n+\t\t\"non-zero generation number\"\n+'\n+\n test_done\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351099","messageId":"20180627132447.142473-17-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 16/22] commit-graph: test for corrupted octopus edge","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:41Z","receivedAt":"2018-06-27T13:25:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file has an extra chunk to store the parent int-ids for\nparents beyond the first parent for octopus merges. Our test repo has a\nsingle octopus merge that we can manipulate to demonstrate the 'verify'\nsubcommand detects incorrect values in that chunk.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5318-commit-graph.sh | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 2c65e6a95c..a0cf1f66de 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -248,6 +248,7 @@ test_expect_success 'git commit-graph verify' '\n '\n \n NUM_COMMITS=9\n+NUM_OCTOPUS_EDGES=2\n HASH_LEN=20\n GRAPH_BYTE_VERSION=4\n GRAPH_BYTE_HASH=5\n@@ -274,6 +275,10 @@ GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4))\n GRAPH_BYTE_COMMIT_WRONG_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3))\n GRAPH_BYTE_COMMIT_GENERATION=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 11))\n GRAPH_BYTE_COMMIT_DATE=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 12))\n+GRAPH_COMMIT_DATA_WIDTH=$(($HASH_LEN + 16))\n+GRAPH_OCTOPUS_DATA_OFFSET=$(($GRAPH_COMMIT_DATA_OFFSET + \\\n+\t\t\t $GRAPH_COMMIT_DATA_WIDTH * $NUM_COMMITS))\n+GRAPH_BYTE_OCTOPUS=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -383,4 +388,9 @@ test_expect_success 'detect incorrect commit date' '\n \t\t\"commit date\"\n '\n \n+test_expect_success 'detect incorrect parent for octopus merge' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OCTOPUS \"\\01\" \\\n+\t\t\"invalid parent\"\n+'\n+\n test_done\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351100","messageId":"20180627132447.142473-19-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 18/22] fsck: verify commit-graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:43Z","receivedAt":"2018-06-27T13:25:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"If core.commitGraph is true, verify the contents of the commit-graph\nduring 'git fsck' using the 'git commit-graph verify' subcommand. Run\nthis check on all alternates, as well.\n\nWe use a new process for two reasons:\n\n1. The subcommand decouples the details of loading and verifying a\n commit-graph file from the other fsck details.\n\n2. The commit-graph verification requires the commits to be loaded\n in a specific order to guarantee we parse from the commit-graph\n file for some objects and from the object database for others.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-fsck.txt | 3 +++\n builtin/fsck.c | 21 +++++++++++++++++++++\n t/t5318-commit-graph.sh | 8 ++++++++\n 3 files changed, 32 insertions(+)\n\ndiff --git a/Documentation/git-fsck.txt b/Documentation/git-fsck.txt\nindex b9f060e3b2..ab9a93fb9b 100644\n--- a/Documentation/git-fsck.txt\n+++ b/Documentation/git-fsck.txt\n@@ -110,6 +110,9 @@ Any corrupt objects you will have to find in backups or other archives\n (i.e., you can just remove them and do an 'rsync' with some other site in\n the hopes that somebody else has the object you have corrupted).\n \n+If core.commitGraph is true, the commit-graph file will also be inspected\n+using 'git commit-graph verify'. See linkgit:git-commit-graph[1].\n+\n Extracted Diagnostics\n ---------------------\n \ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 3ad4f160f9..9fb2edc69f 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -18,6 +18,7 @@\n #include \"decorate.h\"\n #include \"packfile.h\"\n #include \"object-store.h\"\n+#include \"run-command.h\"\n \n #define REACHABLE 0x0001\n #define SEEN 0x0002\n@@ -47,6 +48,7 @@ static int name_objects;\n #define ERROR_REACHABLE 02\n #define ERROR_PACK 04\n #define ERROR_REFS 010\n+#define ERROR_COMMIT_GRAPH 020\n \n static const char *describe_object(struct object *obj)\n {\n@@ -822,5 +824,24 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t}\n \n \tcheck_connectivity();\n+\n+\tif (core_commit_graph) {\n+\t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n+\t\tconst char *verify_argv[] = { \"commit-graph\", \"verify\", NULL, NULL, NULL };\n+\t\tcommit_graph_verify.argv = verify_argv;\n+\t\tcommit_graph_verify.git_cmd = 1;\n+\n+\t\tif (run_command(&commit_graph_verify))\n+\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n+\n+\t\tprepare_alt_odb(the_repository);\n+\t\tfor (alt = the_repository->objects->alt_odb_list; alt; alt = alt->next) {\n+\t\t\tverify_argv[2] = \"--object-dir\";\n+\t\t\tverify_argv[3] = alt->path;\n+\t\t\tif (run_command(&commit_graph_verify))\n+\t\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n+\t\t}\n+\t}\n+\n \treturn errors_found;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex fed05e2f12..a9e8c774d5 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -399,4 +399,12 @@ test_expect_success 'detect invalid checksum hash' '\n \t\t\"incorrect checksum\"\n '\n \n+test_expect_success 'git fsck (checks commit-graph)' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit fsck &&\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_FOOTER \"\\00\" \\\n+\t\t\"incorrect checksum\" &&\n+\ttest_must_fail git fsck\n+'\n+\n test_done\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351101","messageId":"20180627132447.142473-18-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 17/22] commit-graph: verify contents match checksum","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:42Z","receivedAt":"2018-06-27T13:25:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file ends with a SHA1 hash of the previous contents. If\na commit-graph file has errors but the checksum hash is correct, then we\nknow that the problem is a bug in Git and not simply file corruption\nafter-the-fact.\n\nCompute the checksum right away so it is the first error that appears,\nand make the message translatable since this error can be \"corrected\" by\na user by simply deleting the file and recomputing. The rest of the\nerrors are useful only to developers.\n\nBe sure to continue checking the rest of the file data if the checksum\nis wrong. This is important for our tests, as we break the checksum as\nwe modify bytes of the commit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 16 ++++++++++++++--\n t/t5318-commit-graph.sh | 6 ++++++\n 2 files changed, 20 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 6d6c6beff9..d926c4b59f 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -833,6 +833,7 @@ void write_commit_graph(const char *obj_dir,\n \toids.nr = 0;\n }\n \n+#define VERIFY_COMMIT_GRAPH_ERROR_HASH 2\n static int verify_commit_graph_error;\n \n static void graph_report(const char *fmt, ...)\n@@ -852,8 +853,10 @@ static void graph_report(const char *fmt, ...)\n int verify_commit_graph(struct repository *r, struct commit_graph *g)\n {\n \tuint32_t i, cur_fanout_pos = 0;\n-\tstruct object_id prev_oid, cur_oid;\n+\tstruct object_id prev_oid, cur_oid, checksum;\n \tint generation_zero = 0;\n+\tstruct hashfile *f;\n+\tint devnull;\n \n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n@@ -872,6 +875,15 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \tif (verify_commit_graph_error)\n \t\treturn verify_commit_graph_error;\n \n+\tdevnull = open(\"/dev/null\", O_WRONLY);\n+\tf = hashfd(devnull, NULL);\n+\thashwrite(f, g->data, g->data_len - g->hash_len);\n+\tfinalize_hashfile(f, checksum.hash, CSUM_CLOSE);\n+\tif (hashcmp(checksum.hash, g->data + g->data_len - g->hash_len)) {\n+\t\tgraph_report(_(\"the commit-graph file has incorrect checksum and is likely corrupt\"));\n+\t\tverify_commit_graph_error = VERIFY_COMMIT_GRAPH_ERROR_HASH;\n+\t}\n+\n \tfor (i = 0; i < g->num_commits; i++) {\n \t\tstruct commit *graph_commit;\n \n@@ -909,7 +921,7 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\tcur_fanout_pos++;\n \t}\n \n-\tif (verify_commit_graph_error)\n+\tif (verify_commit_graph_error & ~VERIFY_COMMIT_GRAPH_ERROR_HASH)\n \t\treturn verify_commit_graph_error;\n \n \tfor (i = 0; i < g->num_commits; i++) {\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex a0cf1f66de..fed05e2f12 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -279,6 +279,7 @@ GRAPH_COMMIT_DATA_WIDTH=$(($HASH_LEN + 16))\n GRAPH_OCTOPUS_DATA_OFFSET=$(($GRAPH_COMMIT_DATA_OFFSET + \\\n \t\t\t $GRAPH_COMMIT_DATA_WIDTH * $NUM_COMMITS))\n GRAPH_BYTE_OCTOPUS=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4))\n+GRAPH_BYTE_FOOTER=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4 * $NUM_OCTOPUS_EDGES))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -393,4 +394,9 @@ test_expect_success 'detect incorrect parent for octopus merge' '\n \t\t\"invalid parent\"\n '\n \n+test_expect_success 'detect invalid checksum hash' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_FOOTER \"\\00\" \\\n+\t\t\"incorrect checksum\"\n+'\n+\n test_done\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351102","messageId":"20180627132447.142473-12-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 11/22] commit-graph: verify objects exist","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:36Z","receivedAt":"2018-06-27T13:25:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In the 'verify' subcommand, load commits directly from the object\ndatabase to ensure they exist. Parse by skipping the commit-graph.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 18 ++++++++++++++++++\n t/t5318-commit-graph.sh | 7 +++++++\n 2 files changed, 25 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 866a9e7e41..00e89b71e9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -11,6 +11,7 @@\n #include \"sha1-lookup.h\"\n #include \"commit-graph.h\"\n #include \"object-store.h\"\n+#include \"alloc.h\"\n \n #define GRAPH_SIGNATURE 0x43475048 /* \"CGPH\" */\n #define GRAPH_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n@@ -242,6 +243,7 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n {\n \tstruct commit *c;\n \tstruct object_id oid;\n+\n \thashcpy(oid.hash, g->chunk_oid_lookup + g->hash_len * pos);\n \tc = lookup_commit(&oid);\n \tif (!c)\n@@ -893,5 +895,21 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\tcur_fanout_pos++;\n \t}\n \n+\tif (verify_commit_graph_error)\n+\t\treturn verify_commit_graph_error;\n+\n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct commit *odb_commit;\n+\n+\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n+\n+\t\todb_commit = (struct commit *)create_object(r, cur_oid.hash, alloc_commit_node(r));\n+\t\tif (parse_commit_internal(odb_commit, 0, 0)) {\n+\t\t\tgraph_report(\"failed to parse %s from object database\",\n+\t\t\t\t oid_to_hex(&cur_oid));\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+\n \treturn verify_commit_graph_error;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 8ae2a49cfa..a27ec97269 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -247,6 +247,7 @@ test_expect_success 'git commit-graph verify' '\n \tgit commit-graph verify >output\n '\n \n+NUM_COMMITS=9\n HASH_LEN=20\n GRAPH_BYTE_VERSION=4\n GRAPH_BYTE_HASH=5\n@@ -265,6 +266,7 @@ GRAPH_BYTE_FANOUT1=$(($GRAPH_FANOUT_OFFSET + 4 * 4))\n GRAPH_BYTE_FANOUT2=$(($GRAPH_FANOUT_OFFSET + 4 * 255))\n GRAPH_OID_LOOKUP_OFFSET=$(($GRAPH_FANOUT_OFFSET + 4 * 256))\n GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 8))\n+GRAPH_BYTE_OID_LOOKUP_MISSING=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 4 + 10))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -334,4 +336,9 @@ test_expect_success 'detect incorrect OID order' '\n \t\t\"incorrect OID order\"\n '\n \n+test_expect_success 'detect OID not in object database' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_LOOKUP_MISSING \"\\01\" \\\n+\t\t\"from object database\"\n+'\n+\n test_done\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351103","messageId":"20180627132447.142473-11-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 10/22] commit-graph: verify corrupt OID fanout and lookup","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:35Z","receivedAt":"2018-06-27T13:25:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In the commit-graph file, the OID fanout chunk provides an index into\nthe OID lookup. The 'verify' subcommand should find incorrect values\nin the fanout.\n\nSimilarly, the 'verify' subcommand should find out-of-order values in\nthe OID lookup.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 36 ++++++++++++++++++++++++++++++++++++\n t/t5318-commit-graph.sh | 22 ++++++++++++++++++++++\n 2 files changed, 58 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f30b4ccee9..866a9e7e41 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -843,6 +843,9 @@ static void graph_report(const char *fmt, ...)\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g)\n {\n+\tuint32_t i, cur_fanout_pos = 0;\n+\tstruct object_id prev_oid, cur_oid;\n+\n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n \t\treturn 1;\n@@ -857,5 +860,38 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \tif (!g->chunk_commit_data)\n \t\tgraph_report(\"commit-graph is missing the Commit Data chunk\");\n \n+\tif (verify_commit_graph_error)\n+\t\treturn verify_commit_graph_error;\n+\n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n+\n+\t\tif (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n+\t\t\tgraph_report(\"commit-graph has incorrect OID order: %s then %s\",\n+\t\t\t\t oid_to_hex(&prev_oid),\n+\t\t\t\t oid_to_hex(&cur_oid));\n+\n+\t\toidcpy(&prev_oid, &cur_oid);\n+\n+\t\twhile (cur_oid.hash[0] > cur_fanout_pos) {\n+\t\t\tuint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n+\t\t\tif (i != fanout_value)\n+\t\t\t\tgraph_report(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\",\n+\t\t\t\t\t cur_fanout_pos, fanout_value, i);\n+\n+\t\t\tcur_fanout_pos++;\n+\t\t}\n+\t}\n+\n+\twhile (cur_fanout_pos < 256) {\n+\t\tuint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n+\n+\t\tif (g->num_commits != fanout_value)\n+\t\t\tgraph_report(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\",\n+\t\t\t\t cur_fanout_pos, fanout_value, i);\n+\n+\t\tcur_fanout_pos++;\n+\t}\n+\n \treturn verify_commit_graph_error;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex dc16849ddd..8ae2a49cfa 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -247,6 +247,7 @@ test_expect_success 'git commit-graph verify' '\n \tgit commit-graph verify >output\n '\n \n+HASH_LEN=20\n GRAPH_BYTE_VERSION=4\n GRAPH_BYTE_HASH=5\n GRAPH_BYTE_CHUNK_COUNT=6\n@@ -258,6 +259,12 @@ GRAPH_BYTE_OID_LOOKUP_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n \t\t\t 1 * $GRAPH_CHUNK_LOOKUP_WIDTH))\n GRAPH_BYTE_COMMIT_DATA_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n \t\t\t 2 * $GRAPH_CHUNK_LOOKUP_WIDTH))\n+GRAPH_FANOUT_OFFSET=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n+\t\t $GRAPH_CHUNK_LOOKUP_WIDTH * $GRAPH_CHUNK_LOOKUP_ROWS))\n+GRAPH_BYTE_FANOUT1=$(($GRAPH_FANOUT_OFFSET + 4 * 4))\n+GRAPH_BYTE_FANOUT2=$(($GRAPH_FANOUT_OFFSET + 4 * 255))\n+GRAPH_OID_LOOKUP_OFFSET=$(($GRAPH_FANOUT_OFFSET + 4 * 256))\n+GRAPH_BYTE_OID_LOOKUP_ORDER=$(($GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN * 8))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -312,4 +319,19 @@ test_expect_success 'detect missing commit data chunk' '\n \t\t\"missing the Commit Data chunk\"\n '\n \n+test_expect_success 'detect incorrect fanout' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_FANOUT1 \"\\01\" \\\n+\t\t\"fanout value\"\n+'\n+\n+test_expect_success 'detect incorrect fanout final value' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_FANOUT2 \"\\01\" \\\n+\t\t\"fanout value\"\n+'\n+\n+test_expect_success 'detect incorrect OID order' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_LOOKUP_ORDER \"\\01\" \\\n+\t\t\"incorrect OID order\"\n+'\n+\n test_done\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351104","messageId":"20180627132447.142473-23-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 22/22] commit-graph: update design document","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:47Z","receivedAt":"2018-06-27T13:25:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph feature is now integrated with 'fsck' and 'gc',\nso remove those items from the \"Future Work\" section of the\ncommit-graph design document.\n\nAlso remove the section on lazy-loading trees, as that was completed\nin an earlier patch series.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 22 ----------------------\n 1 file changed, 22 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex e1a883eb46..c664acbd76 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -118,9 +118,6 @@ Future Work\n - The commit graph feature currently does not honor commit grafts. This can\n be remedied by duplicating or refactoring the current graft logic.\n \n-- The 'commit-graph' subcommand does not have a \"verify\" mode that is\n- necessary for integration with fsck.\n-\n - After computing and storing generation numbers, we must make graph\n walks aware of generation numbers to gain the performance benefits they\n enable. This will mostly be accomplished by swapping a commit-date-ordered\n@@ -130,25 +127,6 @@ Future Work\n - 'log --topo-order'\n - 'tag --merged'\n \n-- Currently, parse_commit_gently() requires filling in the root tree\n- object for a commit. This passes through lookup_tree() and consequently\n- lookup_object(). Also, it calls lookup_commit() when loading the parents.\n- These method calls check the ODB for object existence, even if the\n- consumer does not need the content. For example, we do not need the\n- tree contents when computing merge bases. Now that commit parsing is\n- removed from the computation time, these lookup operations are the\n- slowest operations keeping graph walks from being fast. Consider\n- loading these objects without verifying their existence in the ODB and\n- only loading them fully when consumers need them. Consider a method\n- such as \"ensure_tree_loaded(commit)\" that fully loads a tree before\n- using commit->tree.\n-\n-- The current design uses the 'commit-graph' subcommand to generate the graph.\n- When this feature stabilizes enough to recommend to most users, we should\n- add automatic graph writes to common operations that create many commits.\n- For example, one could compute a graph on 'clone', 'fetch', or 'repack'\n- commands.\n-\n - A server could provide a commit graph file as part of the network protocol\n to avoid extra calculations by clients. This feature is only of benefit if\n the user is willing to trust the file, because verifying the file is correct\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351105","messageId":"20180627132447.142473-16-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 15/22] commit-graph: verify commit date","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:40Z","receivedAt":"2018-06-27T13:25:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n commit-graph.c | 6 ++++++\n t/t5318-commit-graph.sh | 6 ++++++\n 2 files changed, 12 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex e0f71658da..6d6c6beff9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -986,6 +986,12 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\t\t\t oid_to_hex(&cur_oid),\n \t\t\t\t graph_commit->generation,\n \t\t\t\t max_generation + 1);\n+\n+\t\tif (graph_commit->date != odb_commit->date)\n+\t\t\tgraph_report(\"commit date for commit %s in commit-graph is %\"PRItime\" != %\"PRItime,\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t graph_commit->date,\n+\t\t\t\t odb_commit->date);\n \t}\n \n \treturn verify_commit_graph_error;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 3128e19aef..2c65e6a95c 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -273,6 +273,7 @@ GRAPH_BYTE_COMMIT_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN))\n GRAPH_BYTE_COMMIT_EXTRA_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4))\n GRAPH_BYTE_COMMIT_WRONG_PARENT=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3))\n GRAPH_BYTE_COMMIT_GENERATION=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 11))\n+GRAPH_BYTE_COMMIT_DATE=$(($GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 12))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -377,4 +378,9 @@ test_expect_success 'detect incorrect generation number' '\n \t\t\"non-zero generation number\"\n '\n \n+test_expect_success 'detect incorrect commit date' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_DATE \"\\01\" \\\n+\t\t\"commit date\"\n+'\n+\n test_done\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351106","messageId":"20180627132447.142473-22-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 21/22] gc: automatically write commit-graph files","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:46Z","receivedAt":"2018-06-27T13:25:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file is a very helpful feature for speeding up git\noperations. In order to make it more useful, make it possible to\nwrite the commit-graph file during standard garbage collection\noperations.\n\nAdd a 'gc.commitGraph' config setting that triggers writing a\ncommit-graph file after any non-trivial 'git gc' command. Defaults to\nfalse while the commit-graph feature matures. We specifically do not\nwant to have this on by default until the commit-graph feature is fully\nintegrated with history-modifying features like shallow clones.\n\nHelped-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 9 ++++++---\n Documentation/git-gc.txt | 4 ++++\n builtin/gc.c | 6 ++++++\n t/t5318-commit-graph.sh | 14 ++++++++++++++\n 4 files changed, 30 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 1cc18a828c..978deecfee 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -904,9 +904,12 @@ core.notesRef::\n This setting defaults to \"refs/notes/commits\", and it can be overridden by\n the `GIT_NOTES_REF` environment variable. See linkgit:git-notes[1].\n \n-core.commitGraph::\n-\tEnable git commit graph feature. Allows reading from the\n-\tcommit-graph file.\n+gc.commitGraph::\n+\tIf true, then gc will rewrite the commit-graph file when\n+\tlinkgit:git-gc[1] is run. When using linkgit:git-gc[1]\n+\t'--auto' the commit-graph will be updated if housekeeping is\n+\trequired. Default is false. See linkgit:git-commit-graph[1]\n+\tfor details.\n \n core.sparseCheckout::\n \tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 24b2dd44fe..f5bc98ccb3 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -136,6 +136,10 @@ The optional configuration variable `gc.packRefs` determines if\n it within all non-bare repos or it can be set to a boolean value.\n This defaults to true.\n \n+The optional configuration variable `gc.commitGraph` determines if\n+'git gc' should run 'git commit-graph write'. This can be set to a\n+boolean value. This defaults to false.\n+\n The optional configuration variable `gc.aggressiveWindow` controls how\n much time is spent optimizing the delta compression of the objects in\n the repository when the --aggressive option is specified. The larger\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex ccfb1ceaeb..b7dd206f41 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -20,6 +20,7 @@\n #include \"sigchain.h\"\n #include \"argv-array.h\"\n #include \"commit.h\"\n+#include \"commit-graph.h\"\n #include \"packfile.h\"\n #include \"object-store.h\"\n #include \"pack.h\"\n@@ -40,6 +41,7 @@ static int aggressive_depth = 50;\n static int aggressive_window = 250;\n static int gc_auto_threshold = 6700;\n static int gc_auto_pack_limit = 50;\n+static int gc_write_commit_graph = 0;\n static int detach_auto = 1;\n static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n@@ -129,6 +131,7 @@ static void gc_config(void)\n \tgit_config_get_int(\"gc.aggressivedepth\", &aggressive_depth);\n \tgit_config_get_int(\"gc.auto\", &gc_auto_threshold);\n \tgit_config_get_int(\"gc.autopacklimit\", &gc_auto_pack_limit);\n+\tgit_config_get_bool(\"gc.writecommitgraph\", &gc_write_commit_graph);\n \tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n \tgit_config_get_expiry(\"gc.pruneexpire\", &prune_expire);\n \tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n@@ -641,6 +644,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tif (pack_garbage.nr > 0)\n \t\tclean_pack_garbage();\n \n+\tif (gc_write_commit_graph)\n+\t\twrite_commit_graph_reachable(get_object_directory(), 0);\n+\n \tif (auto_gc && too_many_loose_objects())\n \t\twarning(_(\"There are too many unreachable loose objects; \"\n \t\t\t\"run 'git prune' to remove them.\"));\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 2208c266b5..5947de3d24 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -245,6 +245,20 @@ test_expect_success 'perform fast-forward merge in full repo' '\n \ttest_cmp expect output\n '\n \n+test_expect_success 'check that gc computes commit-graph' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit --allow-empty -m \"blank\" &&\n+\tgit commit-graph write --reachable &&\n+\tcp $objdir/info/commit-graph commit-graph-before-gc &&\n+\tgit reset --hard HEAD~1 &&\n+\tgit config gc.writeCommitGraph true &&\n+\tgit gc &&\n+\tcp $objdir/info/commit-graph commit-graph-after-gc &&\n+\t! test_cmp commit-graph-before-gc commit-graph-after-gc &&\n+\tgit commit-graph write --reachable &&\n+\ttest_cmp commit-graph-after-gc $objdir/info/commit-graph\n+'\n+\n # the verify tests below expect the commit-graph to contain\n # exactly the commits reachable from the commits/8 branch.\n # If the file changes the set of commits in the list, then the\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351107","messageId":"20180627132447.142473-21-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 20/22] commit-graph: add '--reachable' option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:45Z","receivedAt":"2018-06-27T13:25:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When writing commit-graph files, it can be convenient to ask for all\nreachable commits (starting at the ref set) in the resulting file. This\nis particularly helpful when writing to stdin is complicated, such as a\nfuture integration with 'git gc'.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 8 ++++++--\n builtin/commit-graph.c | 16 ++++++++++++----\n commit-graph.c | 18 ++++++++++++++++++\n commit-graph.h | 1 +\n t/t5318-commit-graph.sh | 10 ++++++++++\n 5 files changed, 47 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex a222cfab08..dececb79d7 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -38,12 +38,16 @@ Write a commit graph file based on the commits found in packfiles.\n +\n With the `--stdin-packs` option, generate the new commit graph by\n walking objects only in the specified pack-indexes. (Cannot be combined\n-with --stdin-commits.)\n+with `--stdin-commits` or `--reachable`.)\n +\n With the `--stdin-commits` option, generate the new commit graph by\n walking commits starting at the commits specified in stdin as a list\n of OIDs in hex, one OID per line. (Cannot be combined with\n---stdin-packs.)\n+`--stdin-packs` or `--reachable`.)\n++\n+With the `--reachable` option, generate the new commit graph by walking\n+commits starting at all refs. (Cannot be combined with `--stdin-commits`\n+or `--stdin-packs`.)\n +\n With the `--append` option, include all commits that are present in the\n existing commit-graph file.\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex ea28bc311a..c7d0db5ab4 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -10,7 +10,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph verify [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -25,12 +25,13 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n static struct opts_commit_graph {\n \tconst char *obj_dir;\n+\tint reachable;\n \tint stdin_packs;\n \tint stdin_commits;\n \tint append;\n@@ -127,6 +128,8 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n \t\t\tN_(\"dir\"),\n \t\t\tN_(\"The object directory to store the graph\")),\n+\t\tOPT_BOOL(0, \"reachable\", &opts.reachable,\n+\t\t\tN_(\"start walk at all refs\")),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\tN_(\"scan pack-indexes listed by stdin for commits\")),\n \t\tOPT_BOOL(0, \"stdin-commits\", &opts.stdin_commits,\n@@ -140,11 +143,16 @@ static int graph_write(int argc, const char **argv)\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\n \n-\tif (opts.stdin_packs && opts.stdin_commits)\n-\t\tdie(_(\"cannot use both --stdin-commits and --stdin-packs\"));\n+\tif (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)\n+\t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n \n+\tif (opts.reachable) {\n+\t\twrite_commit_graph_reachable(opts.obj_dir, opts.append);\n+\t\treturn 0;\n+\t}\n+\n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n \t\tstruct strbuf buf = STRBUF_INIT;\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a06e7e9549..adf54e3fe7 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -7,6 +7,7 @@\n #include \"packfile.h\"\n #include \"commit.h\"\n #include \"object.h\"\n+#include \"refs.h\"\n #include \"revision.h\"\n #include \"sha1-lookup.h\"\n #include \"commit-graph.h\"\n@@ -656,6 +657,23 @@ static void compute_generation_numbers(struct packed_commit_list* commits)\n \t}\n }\n \n+static int add_ref_to_list(const char *refname,\n+\t\t\t const struct object_id *oid,\n+\t\t\t int flags, void *cb_data)\n+{\n+\tstruct string_list *list = (struct string_list*)cb_data;\n+\tstring_list_append(list, oid_to_hex(oid));\n+\treturn 0;\n+}\n+\n+void write_commit_graph_reachable(const char *obj_dir, int append)\n+{\n+\tstruct string_list list;\n+\tstring_list_init(&list, 1);\n+\tfor_each_ref(add_ref_to_list, &list);\n+\twrite_commit_graph(obj_dir, NULL, &list, append);\n+}\n+\n void write_commit_graph(const char *obj_dir,\n \t\t\tstruct string_list *pack_indexes,\n \t\t\tstruct string_list *commit_hex,\ndiff --git a/commit-graph.h b/commit-graph.h\nindex a79b854470..506cb45fb1 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -48,6 +48,7 @@ struct commit_graph {\n \n struct commit_graph *load_commit_graph_one(const char *graph_file);\n \n+void write_commit_graph_reachable(const char *obj_dir, int append);\n void write_commit_graph(const char *obj_dir,\n \t\t\tstruct string_list *pack_indexes,\n \t\t\tstruct string_list *commit_hex,\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex a9e8c774d5..2208c266b5 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -205,6 +205,16 @@ test_expect_success 'build graph from commits with append' '\n graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n \n+test_expect_success 'build graph using --reachable' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph write --reachable &&\n+\ttest_path_is_file $objdir/info/commit-graph &&\n+\tgraph_read_expect \"11\" \"large_edges\"\n+'\n+\n+graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n+graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n+\n test_expect_success 'setup bare repo' '\n \tcd \"$TRASH_DIRECTORY\" &&\n \tgit clone --bare --no-local full bare &&\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351108","messageId":"20180627132447.142473-10-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 09/22] commit-graph: verify required chunks are present","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:34Z","receivedAt":"2018-06-27T13:26:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file requires the following three chunks:\n\n* OID Fanout\n* OID Lookup\n* Commit Data\n\nIf any of these are missing, then the 'verify' subcommand should\nreport a failure. This includes the chunk IDs malformed or the\nchunk count is truncated.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 9 +++++++++\n t/t5318-commit-graph.sh | 29 +++++++++++++++++++++++++++++\n 2 files changed, 38 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 22ef696e18..f30b4ccee9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -848,5 +848,14 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\treturn 1;\n \t}\n \n+\tverify_commit_graph_error = 0;\n+\n+\tif (!g->chunk_oid_fanout)\n+\t\tgraph_report(\"commit-graph is missing the OID Fanout chunk\");\n+\tif (!g->chunk_oid_lookup)\n+\t\tgraph_report(\"commit-graph is missing the OID Lookup chunk\");\n+\tif (!g->chunk_commit_data)\n+\t\tgraph_report(\"commit-graph is missing the Commit Data chunk\");\n+\n \treturn verify_commit_graph_error;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex c0c1ff09b9..dc16849ddd 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -249,6 +249,15 @@ test_expect_success 'git commit-graph verify' '\n \n GRAPH_BYTE_VERSION=4\n GRAPH_BYTE_HASH=5\n+GRAPH_BYTE_CHUNK_COUNT=6\n+GRAPH_CHUNK_LOOKUP_OFFSET=8\n+GRAPH_CHUNK_LOOKUP_WIDTH=12\n+GRAPH_CHUNK_LOOKUP_ROWS=5\n+GRAPH_BYTE_OID_FANOUT_ID=$GRAPH_CHUNK_LOOKUP_OFFSET\n+GRAPH_BYTE_OID_LOOKUP_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n+\t\t\t 1 * $GRAPH_CHUNK_LOOKUP_WIDTH))\n+GRAPH_BYTE_COMMIT_DATA_ID=$(($GRAPH_CHUNK_LOOKUP_OFFSET + \\\n+\t\t\t 2 * $GRAPH_CHUNK_LOOKUP_WIDTH))\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -283,4 +292,24 @@ test_expect_success 'detect bad hash version' '\n \t\t\"hash version\"\n '\n \n+test_expect_success 'detect low chunk count' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_CHUNK_COUNT \"\\02\" \\\n+\t\t\"missing the .* chunk\"\n+'\n+\n+test_expect_success 'detect missing OID fanout chunk' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_FANOUT_ID \"\\0\" \\\n+\t\t\"missing the OID Fanout chunk\"\n+'\n+\n+test_expect_success 'detect missing OID lookup chunk' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_LOOKUP_ID \"\\0\" \\\n+\t\t\"missing the OID Lookup chunk\"\n+'\n+\n+test_expect_success 'detect missing commit data chunk' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_DATA_ID \"\\0\" \\\n+\t\t\"missing the Commit Data chunk\"\n+'\n+\n test_done\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351109","messageId":"20180627132447.142473-20-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180627132447.142473-1-dstolee@microsoft.com","subject":"[PATCH v7 19/22] commit-graph: use string-list API for input","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T13:24:44Z","receivedAt":"2018-06-27T13:26:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 39 +++++++++++++--------------------------\n commit-graph.c | 15 +++++++--------\n commit-graph.h | 7 +++----\n 3 files changed, 23 insertions(+), 38 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 9d108f43a9..ea28bc311a 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -119,13 +119,9 @@ static int graph_read(int argc, const char **argv)\n \n static int graph_write(int argc, const char **argv)\n {\n-\tconst char **pack_indexes = NULL;\n-\tint packs_nr = 0;\n-\tconst char **commit_hex = NULL;\n-\tint commits_nr = 0;\n-\tconst char **lines = NULL;\n-\tint lines_nr = 0;\n-\tint lines_alloc = 0;\n+\tstruct string_list *pack_indexes = NULL;\n+\tstruct string_list *commit_hex = NULL;\n+\tstruct string_list lines;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n@@ -149,34 +145,25 @@ static int graph_write(int argc, const char **argv)\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n \n+\tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n \t\tstruct strbuf buf = STRBUF_INIT;\n-\t\tlines_nr = 0;\n-\t\tlines_alloc = 128;\n-\t\tALLOC_ARRAY(lines, lines_alloc);\n-\n-\t\twhile (strbuf_getline(&buf, stdin) != EOF) {\n-\t\t\tALLOC_GROW(lines, lines_nr + 1, lines_alloc);\n-\t\t\tlines[lines_nr++] = strbuf_detach(&buf, NULL);\n-\t\t}\n-\n-\t\tif (opts.stdin_packs) {\n-\t\t\tpack_indexes = lines;\n-\t\t\tpacks_nr = lines_nr;\n-\t\t}\n-\t\tif (opts.stdin_commits) {\n-\t\t\tcommit_hex = lines;\n-\t\t\tcommits_nr = lines_nr;\n-\t\t}\n+\n+\t\twhile (strbuf_getline(&buf, stdin) != EOF)\n+\t\t\tstring_list_append(&lines, strbuf_detach(&buf, NULL));\n+\n+\t\tif (opts.stdin_packs)\n+\t\t\tpack_indexes = &lines;\n+\t\tif (opts.stdin_commits)\n+\t\t\tcommit_hex = &lines;\n \t}\n \n \twrite_commit_graph(opts.obj_dir,\n \t\t\t pack_indexes,\n-\t\t\t packs_nr,\n \t\t\t commit_hex,\n-\t\t\t commits_nr,\n \t\t\t opts.append);\n \n+\tstring_list_clear(&lines, 0);\n \treturn 0;\n }\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex d926c4b59f..a06e7e9549 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -657,10 +657,8 @@ static void compute_generation_numbers(struct packed_commit_list* commits)\n }\n \n void write_commit_graph(const char *obj_dir,\n-\t\t\tconst char **pack_indexes,\n-\t\t\tint nr_packs,\n-\t\t\tconst char **commit_hex,\n-\t\t\tint nr_commits,\n+\t\t\tstruct string_list *pack_indexes,\n+\t\t\tstruct string_list *commit_hex,\n \t\t\tint append)\n {\n \tstruct packed_oid_list oids;\n@@ -701,10 +699,10 @@ void write_commit_graph(const char *obj_dir,\n \t\tint dirlen;\n \t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n \t\tdirlen = packname.len;\n-\t\tfor (i = 0; i < nr_packs; i++) {\n+\t\tfor (i = 0; i < pack_indexes->nr; i++) {\n \t\t\tstruct packed_git *p;\n \t\t\tstrbuf_setlen(&packname, dirlen);\n-\t\t\tstrbuf_addstr(&packname, pack_indexes[i]);\n+\t\t\tstrbuf_addstr(&packname, pack_indexes->items[i].string);\n \t\t\tp = add_packed_git(packname.buf, packname.len, 1);\n \t\t\tif (!p)\n \t\t\t\tdie(\"error adding pack %s\", packname.buf);\n@@ -717,12 +715,13 @@ void write_commit_graph(const char *obj_dir,\n \t}\n \n \tif (commit_hex) {\n-\t\tfor (i = 0; i < nr_commits; i++) {\n+\t\tfor (i = 0; i < commit_hex->nr; i++) {\n \t\t\tconst char *end;\n \t\t\tstruct object_id oid;\n \t\t\tstruct commit *result;\n \n-\t\t\tif (commit_hex[i] && parse_oid_hex(commit_hex[i], &oid, &end))\n+\t\t\tif (commit_hex->items[i].string &&\n+\t\t\t parse_oid_hex(commit_hex->items[i].string, &oid, &end))\n \t\t\t\tcontinue;\n \n \t\t\tresult = lookup_commit_reference_gently(&oid, 1);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 4359812fb4..a79b854470 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -3,6 +3,7 @@\n \n #include \"git-compat-util.h\"\n #include \"repository.h\"\n+#include \"string-list.h\"\n \n char *get_commit_graph_filename(const char *obj_dir);\n \n@@ -48,10 +49,8 @@ struct commit_graph {\n struct commit_graph *load_commit_graph_one(const char *graph_file);\n \n void write_commit_graph(const char *obj_dir,\n-\t\t\tconst char **pack_indexes,\n-\t\t\tint nr_packs,\n-\t\t\tconst char **commit_hex,\n-\t\t\tint nr_commits,\n+\t\t\tstruct string_list *pack_indexes,\n+\t\t\tstruct string_list *commit_hex,\n \t\t\tint append);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n-- \n2.18.0.24.g1b579a2ee9\n\n"},{"id":"351126","messageId":"xmqqa7rggmer.fsf@gitster-ct.c.googlers.com","threadId":"48647","inReplyTo":"20180627132447.142473-2-dstolee@microsoft.com","subject":"Re: [PATCH v7 01/22] t5318-commit-graph.sh: use core.commitGraph","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-06-27T17:38:20Z","receivedAt":"2018-06-27T17:38:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> The commit-graph tests should be checking that normal Git operations\n> succeed and have matching output with and without the commit-graph\n> feature enabled. However, the test was toggling 'core.graph' instead\n> of the correct 'core.commitGraph' variable.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n>\n> Junio,\n>\n> I sent this patch as a one-off a while ago, but it seems it was dropped.\n> I'm adding it back here so we don't forget it.\n\nThanks, will queue.\n\n\n>\n> -Stolee\n>\n> t/t5318-commit-graph.sh | 4 ++--\n> 1 file changed, 2 insertions(+), 2 deletions(-)\n>\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 77d85aefe7..59d0be2877 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -28,8 +28,8 @@ test_expect_success 'create commits and repack' '\n> '\n> \n> graph_git_two_modes() {\n> -\tgit -c core.graph=true $1 >output\n> -\tgit -c core.graph=false $1 >expect\n> +\tgit -c core.commitGraph=true $1 >output\n> +\tgit -c core.commitGraph=false $1 >expect\n> \ttest_cmp output expect\n> }\n"},{"id":"351128","messageId":"xmqq36x8glt0.fsf@gitster-ct.c.googlers.com","threadId":"48647","inReplyTo":"20180627132447.142473-18-dstolee@microsoft.com","subject":"Re: [PATCH v7 17/22] commit-graph: verify contents match checksum","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-06-27T17:51:23Z","receivedAt":"2018-06-27T17:51:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> +\tdevnull = open(\"/dev/null\", O_WRONLY);\n> +\tf = hashfd(devnull, NULL);\n> +\thashwrite(f, g->data, g->data_len - g->hash_len);\n\nI wondered if we can hide the knowledge of \"/dev/null\" by reusing\nhashfd_check() from csum-file.c (which is used by the verification\nof pack idx file), which also gives us the hash comparison of an\nexisting file on disk for free. But unfortunately this codepath\nonly verifies the checksum and not contents, so we cannot avoid\nsetting up the /dev/null sink manually like this patch does, I\nguess.\n\n> +\tfinalize_hashfile(f, checksum.hash, CSUM_CLOSE);\n> +\tif (hashcmp(checksum.hash, g->data + g->data_len - g->hash_len)) {\n> +\t\tgraph_report(_(\"the commit-graph file has incorrect checksum and is likely corrupt\"));\n> +\t\tverify_commit_graph_error = VERIFY_COMMIT_GRAPH_ERROR_HASH;\n> +\t}\n> +\n> \tfor (i = 0; i < g->num_commits; i++) {\n> \t\tstruct commit *graph_commit;\n> \n> @@ -909,7 +921,7 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n> \t\tcur_fanout_pos++;\n> \t}\n> \n> -\tif (verify_commit_graph_error)\n> +\tif (verify_commit_graph_error & ~VERIFY_COMMIT_GRAPH_ERROR_HASH)\n> \t\treturn verify_commit_graph_error;\n> \n> \tfor (i = 0; i < g->num_commits; i++) {\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index a0cf1f66de..fed05e2f12 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -279,6 +279,7 @@ GRAPH_COMMIT_DATA_WIDTH=$(($HASH_LEN + 16))\n> GRAPH_OCTOPUS_DATA_OFFSET=$(($GRAPH_COMMIT_DATA_OFFSET + \\\n> \t\t\t $GRAPH_COMMIT_DATA_WIDTH * $NUM_COMMITS))\n> GRAPH_BYTE_OCTOPUS=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4))\n> +GRAPH_BYTE_FOOTER=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4 * $NUM_OCTOPUS_EDGES))\n> \n> # usage: corrupt_graph_and_verify \n> # Manipulates the commit-graph file at the position\n> @@ -393,4 +394,9 @@ test_expect_success 'detect incorrect parent for octopus merge' '\n> \t\t\"invalid parent\"\n> '\n> \n> +test_expect_success 'detect invalid checksum hash' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_FOOTER \"\\00\" \\\n> +\t\t\"incorrect checksum\"\n> +'\n> +\n> test_done\n"},{"id":"351129","messageId":"xmqqy3f0f74u.fsf@gitster-ct.c.googlers.com","threadId":"48647","inReplyTo":"20180627132447.142473-21-dstolee@microsoft.com","subject":"Re: [PATCH v7 20/22] commit-graph: add '--reachable' option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-06-27T17:53:37Z","receivedAt":"2018-06-27T17:53:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> diff --git a/commit-graph.c b/commit-graph.c\n> index a06e7e9549..adf54e3fe7 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -7,6 +7,7 @@\n> #include \"packfile.h\"\n> #include \"commit.h\"\n> #include \"object.h\"\n> +#include \"refs.h\"\n> #include \"revision.h\"\n> #include \"sha1-lookup.h\"\n> #include \"commit-graph.h\"\n> @@ -656,6 +657,23 @@ static void compute_generation_numbers(struct packed_commit_list* commits)\n> \t}\n> }\n> \n> +static int add_ref_to_list(const char *refname,\n> +\t\t\t const struct object_id *oid,\n> +\t\t\t int flags, void *cb_data)\n> +{\n> +\tstruct string_list *list = (struct string_list*)cb_data;\n\nStyle: \"(struct string_list *)cb_data\"\n\nAlso please have a blank line here between the decls and the first\nstatement.\n\n> +\tstring_list_append(list, oid_to_hex(oid));\n> +\treturn 0;\n> +}\n"},{"id":"351135","messageId":"xmqqr2ksf6ed.fsf@gitster-ct.c.googlers.com","threadId":"48647","inReplyTo":"20180627132447.142473-22-dstolee@microsoft.com","subject":"Re: [PATCH v7 21/22] gc: automatically write commit-graph files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-06-27T18:09:30Z","receivedAt":"2018-06-27T18:09:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> @@ -40,6 +41,7 @@ static int aggressive_depth = 50;\n> static int aggressive_window = 250;\n> static int gc_auto_threshold = 6700;\n> static int gc_auto_pack_limit = 50;\n> +static int gc_write_commit_graph = 0;\n\nPlease avoid unnecessary (and undesirable) explicit initialization\nto 0. Instead, let BSS to handle it by leaving \" = 0\" out.\n\n> +test_expect_success 'check that gc computes commit-graph' '\n> +\tcd \"$TRASH_DIRECTORY/full\" &&\n> +\tgit commit --allow-empty -m \"blank\" &&\n> +\tgit commit-graph write --reachable &&\n> +\tcp $objdir/info/commit-graph commit-graph-before-gc &&\n> +\tgit reset --hard HEAD~1 &&\n> +\tgit config gc.writeCommitGraph true &&\n> +\tgit gc &&\n> +\tcp $objdir/info/commit-graph commit-graph-after-gc &&\n> +\t! test_cmp commit-graph-before-gc commit-graph-after-gc &&\n\nThe set of commits in the commit graph will chanbe by discarding the\n(old) tip commit, so the fact that the contents of the file changed\nacross gc proves that \"commit-graph write\" was triggered during gc.\n\nCome to think of it, do we promise to end-users (in docs etc.) that\ncommit-graph covers *ONLY* commits reachable from refs and HEAD? I\nam wondering what should happen if \"git gc\" here does not prune the\nreflog for HEAD---wouldn't we want to reuse the properties of the\ncommit we are discarding when it comes back (e.g. you push, then\nreset back, and the next pull makes it reappear in your repository)?\n\nI guess what I am really questioning is if it is sensible to define\n\"--reachable\" as \"starting at all refs\", unlike the usual connectivity\nrules \"gc\" uses, especially when this is run from inside \"gc\".\n\n> +\tgit commit-graph write --reachable &&\n> +\ttest_cmp commit-graph-after-gc $objdir/info/commit-graph\n\nThis says that running \"commit-graph write\" twice without changing\nthe topology MUST yield byte-for-byte identical commit-graph file.\n\nIs that a reasonable requirement on the future implementation? I am\nwondering if there will arise a situation where you need to store\nrecords in \"some\" fixed order but two records compare \"the same\" and\ntie-breaking them to give stable sort is expensive, or something\nlike that, which would benefit if you leave an escape hatch to allow\ntwo logically identical graphs expressed bitwise differently.\n\n> +'\n> +\n> # the verify tests below expect the commit-graph to contain\n> # exactly the commits reachable from the commits/8 branch.\n> # If the file changes the set of commits in the list, then the\n"},{"id":"351137","messageId":"xmqqmuvgf6ds.fsf@gitster-ct.c.googlers.com","threadId":"48647","inReplyTo":"20180627132447.142473-23-dstolee@microsoft.com","subject":"Re: [PATCH v7 22/22] commit-graph: update design document","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-06-27T18:09:51Z","receivedAt":"2018-06-27T18:09:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> The commit-graph feature is now integrated with 'fsck' and 'gc',\n> so remove those items from the \"Future Work\" section of the\n> commit-graph design document.\n\nNice ;-)\n"},{"id":"351141","messageId":"7ac29eb4-6df7-02a2-e34b-e8622dc0a012@gmail.com","threadId":"48647","inReplyTo":"xmqqr2ksf6ed.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v7 21/22] gc: automatically write commit-graph files","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-27T18:24:09Z","receivedAt":"2018-06-27T18:24:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/27/2018 2:09 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> @@ -40,6 +41,7 @@ static int aggressive_depth = 50;\n>> static int aggressive_window = 250;\n>> static int gc_auto_threshold = 6700;\n>> static int gc_auto_pack_limit = 50;\n>> +static int gc_write_commit_graph = 0;\n> Please avoid unnecessary (and undesirable) explicit initialization\n> to 0. Instead, let BSS to handle it by leaving \" = 0\" out.\n>\n>> +test_expect_success 'check that gc computes commit-graph' '\n>> +\tcd \"$TRASH_DIRECTORY/full\" &&\n>> +\tgit commit --allow-empty -m \"blank\" &&\n>> +\tgit commit-graph write --reachable &&\n>> +\tcp $objdir/info/commit-graph commit-graph-before-gc &&\n>> +\tgit reset --hard HEAD~1 &&\n>> +\tgit config gc.writeCommitGraph true &&\n>> +\tgit gc &&\n>> +\tcp $objdir/info/commit-graph commit-graph-after-gc &&\n>> +\t! test_cmp commit-graph-before-gc commit-graph-after-gc &&\n> The set of commits in the commit graph will chanbe by discarding the\n> (old) tip commit, so the fact that the contents of the file changed\n> across gc proves that \"commit-graph write\" was triggered during gc.\n>\n> Come to think of it, do we promise to end-users (in docs etc.) that\n> commit-graph covers *ONLY* commits reachable from refs and HEAD? I\n> am wondering what should happen if \"git gc\" here does not prune the\n> reflog for HEAD---wouldn't we want to reuse the properties of the\n> commit we are discarding when it comes back (e.g. you push, then\n> reset back, and the next pull makes it reappear in your repository)?\n\nToday I learned that 'gc' keeps some of the reflog around. That makes \nsense, but I wouldn't optimize the commit-graph file for this scenario.\n\n> I guess what I am really questioning is if it is sensible to define\n> \"--reachable\" as \"starting at all refs\", unlike the usual connectivity\n> rules \"gc\" uses, especially when this is run from inside \"gc\".\n\nIt is sensible to me, especially because we only lose performance if we \nvisit those other commits that are still in the object database. By \nwriting the commit-graph on 'gc' and not during 'fetch', we are already \nassuming the commit-graph will usually be behind the set of commits that \nthe user cares about, by design.\n\nAn alternate view on the decision will need help answering from others \nwho know more than me: In fetch negotiation, does the client report \ncommits in the reflog as 'have's or do they get re-downloaded on a \nresulting 'git pull'?\n\n>\n>> +\tgit commit-graph write --reachable &&\n>> +\ttest_cmp commit-graph-after-gc $objdir/info/commit-graph\n> This says that running \"commit-graph write\" twice without changing\n> the topology MUST yield byte-for-byte identical commit-graph file.\n>\n> Is that a reasonable requirement on the future implementation? I am\n> wondering if there will arise a situation where you need to store\n> records in \"some\" fixed order but two records compare \"the same\" and\n> tie-breaking them to give stable sort is expensive, or something\n> like that, which would benefit if you leave an escape hatch to allow\n> two logically identical graphs expressed bitwise differently.\n\nSince the file format allows flexibility in the order of the chunks, it \nis possible to have bitwise-different files that represent the same set \nof data. However, I would not want git to provide inconsistent output \ngiven the same set of commits covered by the file.\n\nThanks,\n-Stolee\n"},{"id":"351166","messageId":"20180627215926.119376-1-jonathantanmy@google.com","threadId":"48647","inReplyTo":"20180627132447.142473-8-dstolee@microsoft.com","subject":"Re: [PATCH v7 07/22] commit-graph: add 'verify' subcommand","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-06-27T21:59:26Z","receivedAt":"2018-06-27T21:59:33Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> +int verify_commit_graph(struct repository *r, struct commit_graph *g)\n\nI haven't had the time to review this patch set, but I did rebase my\nobject store refactoring [1] on this and wrote a test:\n\n static void test_verify_commit_graph(const char *gitdir, const char *worktree)\n {\n \tstruct repository r;\n \tchar *graph_name;\n \tstruct commit_graph *graph;\n \n \trepo_init(&r, gitdir, worktree);\n \n \tgraph_name = get_commit_graph_filename(r.objects->objectdir);\n \tgraph = load_commit_graph_one(graph_name);\n \n \tprintf(\"verification returned %d\\n\", verify_commit_graph(&r, graph));\n \n \trepo_clear(&r);\n }\n\nHowever, it doesn't work because verify_commit_graph() invokes\nparse_commit_internal(), which tries to look up replace refs in\nthe_repository.\n\nI think that verify_commit_graph() should not take a repository argument\nfor now. To minimize churn on the review of this patch set, and to\nminimize diffs when we migrate parse_commit_internal() (and likely other\nfunctions) to take in a repository argument, I would be OK with\nsomething like the following instead:\n\n int verify_commit_graph(struct commit_graph *g)\n {\n\t /*\n\t * NEEDSWORK: Make r into a parameter when all functions\n\t * invoked by this function are not hardcoded to operate on\n\t * the_repository.\n\t */\n\t struct repository *r = the_repository;\n\t /* ... */\n\nAs for my rebased refactoring, I'll send the patches to the mailing list\nonce Junio updates ds/commit-graph-fsck with these latest changes, so\nthat I can rebase once again on that and ensure that everything still\nworks.\n\n[1] https://public-inbox.org/git/cover.1529616356.git.jonathantanmy@google.com/\n"},{"id":"351332","messageId":"f6116a1b-8e5a-9cb9-9c8f-53f56a9b79da@gmail.com","threadId":"48647","inReplyTo":"20180627215926.119376-1-jonathantanmy@google.com","subject":"Re: [PATCH v7 07/22] commit-graph: add 'verify' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-29T12:47:55Z","receivedAt":"2018-06-29T12:48:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/27/2018 5:59 PM, Jonathan Tan wrote:\n>> +int verify_commit_graph(struct repository *r, struct commit_graph *g)\n> I haven't had the time to review this patch set, but I did rebase my\n> object store refactoring [1] on this and wrote a test:\n>\n> static void test_verify_commit_graph(const char *gitdir, const char *worktree)\n> {\n> \tstruct repository r;\n> \tchar *graph_name;\n> \tstruct commit_graph *graph;\n> \n> \trepo_init(&r, gitdir, worktree);\n> \n> \tgraph_name = get_commit_graph_filename(r.objects->objectdir);\n> \tgraph = load_commit_graph_one(graph_name);\n> \n> \tprintf(\"verification returned %d\\n\", verify_commit_graph(&r, graph));\n> \n> \trepo_clear(&r);\n> }\n>\n> However, it doesn't work because verify_commit_graph() invokes\n> parse_commit_internal(), which tries to look up replace refs in\n> the_repository.\n>\n> I think that verify_commit_graph() should not take a repository argument\n> for now. To minimize churn on the review of this patch set, and to\n> minimize diffs when we migrate parse_commit_internal() (and likely other\n> functions) to take in a repository argument, I would be OK with\n> something like the following instead:\n>\n> int verify_commit_graph(struct commit_graph *g)\n> {\n> \t /*\n> \t * NEEDSWORK: Make r into a parameter when all functions\n> \t * invoked by this function are not hardcoded to operate on\n> \t * the_repository.\n> \t */\n> \t struct repository *r = the_repository;\n> \t /* ... */\n>\n> As for my rebased refactoring, I'll send the patches to the mailing list\n> once Junio updates ds/commit-graph-fsck with these latest changes, so\n> that I can rebase once again on that and ensure that everything still\n> works.\n>\n> [1] https://public-inbox.org/git/cover.1529616356.git.jonathantanmy@google.com/\n\nThanks for looking into this, Jonathan. At some point I took my series \nand moved it on top of Stefan's lookup_object() series, but then moved \noff of it since those commits were not in 'pu'. This 'struct repository \n*r' didn't get removed in that process.\n\nThanks,\n\n-Stolee\n\n"},{"id":"352223","messageId":"20180711093822.2123-1-szeder.dev@gmail.com","threadId":"48647","inReplyTo":"20180627132447.142473-7-dstolee@microsoft.com","subject":"Re: [PATCH v7 06/22] commit-graph: load a root tree from specific graph","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-07-11T09:38:22Z","receivedAt":"2018-07-11T09:38:36Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"\n> When lazy-loading a tree for a commit, it will be important to select\n> the tree from a specific struct commit_graph. Create a new method that\n> specifies the commit-graph file and use that in\n> get_commit_tree_in_graph().\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 12 +++++++++---\n> 1 file changed, 9 insertions(+), 3 deletions(-)\n> \n> diff --git a/commit-graph.c b/commit-graph.c\n> index e77b19971d..9e228d3bb5 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -362,14 +362,20 @@ static struct tree *load_tree_for_commit(struct commit_graph *g, struct commit *\n> \treturn c->maybe_tree;\n> }\n> \n> -struct tree *get_commit_tree_in_graph(const struct commit *c)\n> +static struct tree *get_commit_tree_in_graph_one(struct commit_graph *g,\n> +\t\t\t\t\t\t const struct commit *c)\n> {\n> \tif (c->maybe_tree)\n> \t\treturn c->maybe_tree;\n\nCoccinelle and 'commit.cocci' complain about this change, because it\nrenames a function directly accessing struct commit's maybe_tree\nmember.\n\nIn 'commit.cocci' the regex listing the names of functions that are\nallowed to access c->maybe_tree directly should be updated accordingly\nas well.\n\n> \tif (c->graph_pos == COMMIT_NOT_FROM_GRAPH)\n> -\t\tBUG(\"get_commit_tree_in_graph called from non-commit-graph commit\");\n> +\t\tBUG(\"get_commit_tree_in_graph_one called from non-commit-graph commit\");\n> +\n> +\treturn load_tree_for_commit(g, (struct commit *)c);\n> +}\n> \n> -\treturn load_tree_for_commit(commit_graph, (struct commit *)c);\n> +struct tree *get_commit_tree_in_graph(const struct commit *c)\n> +{\n> +\treturn get_commit_tree_in_graph_one(commit_graph, c);\n> }\n> \n> static void write_graph_chunk_fanout(struct hashfile *f,\n> -- \n> 2.18.0.24.g1b579a2ee9\n> \n> \n"},{"id":"352472","messageId":"20180713163033.110497-1-dstolee@microsoft.com","threadId":"48647","inReplyTo":"20180711093822.2123-1-szeder.dev@gmail.com","subject":"[PATCH] coccinelle: update commit.cocci","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-07-13T16:30:46Z","receivedAt":"2018-07-13T16:30:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"A recent patch series renamed the get_commit_tree_from_graph method but\nforgot to update the coccinelle script that excempted it from rules\nregarding accesses to 'maybe_tree'. This fixes that oversight to bring\nthe coccinelle scripts back to a good state.\n\nSigned-off-by: Derrick Stolee \n---\n contrib/coccinelle/commit.cocci | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/contrib/coccinelle/commit.cocci b/contrib/coccinelle/commit.cocci\nindex a7e9215ffc..aec3345adb 100644\n--- a/contrib/coccinelle/commit.cocci\n+++ b/contrib/coccinelle/commit.cocci\n@@ -12,7 +12,7 @@ expression c;\n \n // These excluded functions must access c->maybe_tree direcly.\n @@\n-identifier f !~ \"^(get_commit_tree|get_commit_tree_in_graph|load_tree_for_commit)$\";\n+identifier f !~ \"^(get_commit_tree|get_commit_tree_in_graph_one|load_tree_for_commit)$\";\n expression c;\n @@\n f(...) {...\n\nbase-commit: d4f65b8d141e041eb5e558cd9e763873e29863b9\n-- \n2.18.0.118.gd4f65b8d14\n\n"},{"id":"352507","messageId":"CAPig+cQezN26oebER5wno0ENHdZ3+nKKW6avhW5ZsPo50bJ+ew@mail.gmail.com","threadId":"48647","inReplyTo":"20180713163033.110497-1-dstolee@microsoft.com","subject":"Re: [PATCH] coccinelle: update commit.cocci","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-07-13T17:45:09Z","receivedAt":"2018-07-13T17:45:22Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, Jul 13, 2018 at 12:30 PM Derrick Stolee wrote:\n> A recent patch series renamed the get_commit_tree_from_graph method but\n> forgot to update the coccinelle script that excempted it from rules\n\ns/excempted/exempted/\n\n> regarding accesses to 'maybe_tree'. This fixes that oversight to bring\n> the coccinelle scripts back to a good state.\n>\n> Signed-off-by: Derrick Stolee \n"},{"id":"354367","messageId":"20180803093909.2853-1-szeder.dev@gmail.com","threadId":"48647","inReplyTo":"20180627132447.142473-22-dstolee@microsoft.com","subject":"Re: [PATCH v7 21/22] gc: automatically write commit-graph files","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-08-03T09:39:09Z","receivedAt":"2018-08-03T09:39:41Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"\nHi Derrick,\n\n> The commit-graph file is a very helpful feature for speeding up git\n> operations. In order to make it more useful, make it possible to\n> write the commit-graph file during standard garbage collection\n> operations.\n> \n> Add a 'gc.commitGraph' config setting that triggers writing a\n> commit-graph file after any non-trivial 'git gc' command. Defaults to\n> false while the commit-graph feature matures. We specifically do not\n> want to have this on by default until the commit-graph feature is fully\n> integrated with history-modifying features like shallow clones.\n> \n> Helped-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/config.txt | 9 ++++++---\n> Documentation/git-gc.txt | 4 ++++\n> builtin/gc.c | 6 ++++++\n> t/t5318-commit-graph.sh | 14 ++++++++++++++\n> 4 files changed, 30 insertions(+), 3 deletions(-)\n> \n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index 1cc18a828c..978deecfee 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -904,9 +904,12 @@ core.notesRef::\n> This setting defaults to \"refs/notes/commits\", and it can be overridden by\n> the `GIT_NOTES_REF` environment variable. See linkgit:git-notes[1].\n> \n> -core.commitGraph::\n> -\tEnable git commit graph feature. Allows reading from the\n> -\tcommit-graph file.\n> +gc.commitGraph::\n> +\tIf true, then gc will rewrite the commit-graph file when\n> +\tlinkgit:git-gc[1] is run. When using linkgit:git-gc[1]\n> +\t'--auto' the commit-graph will be updated if housekeeping is\n> +\trequired. Default is false. See linkgit:git-commit-graph[1]\n> +\tfor details.\n\nSorry, I've lost track of this patch series, unfortunately, but after\nit hit master yesterday a merge conflict with one of my (unrelated)\nWIP patches drew my attention to this hunk above, which got me\nconfused:\n\n - Why remove the description of 'core.commitGraph'? 'git grep -i\n core.commitGraph' shows that it's still used supported.\n\n - Why insert the description of 'gc.commitGraph' here instead of next\n to the other 'gc.*' config variables, breaking lexicographical\n order of confvars?\n\n - This hunk, the other documentation hunk right below, and indeed the\n commit message all speak about 'gc.commitGraph', but the new code\n and test added in this patch use 'gc.writeCommitGraph' (following\n my suggestion from a couple of weeks ago, I think?).\n\nWhat's going on?\n\n> \n> core.sparseCheckout::\n> \tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n> diff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\n> index 24b2dd44fe..f5bc98ccb3 100644\n> --- a/Documentation/git-gc.txt\n> +++ b/Documentation/git-gc.txt\n> @@ -136,6 +136,10 @@ The optional configuration variable `gc.packRefs` determines if\n> it within all non-bare repos or it can be set to a boolean value.\n> This defaults to true.\n> \n> +The optional configuration variable `gc.commitGraph` determines if\n> +'git gc' should run 'git commit-graph write'. This can be set to a\n> +boolean value. This defaults to false.\n> +\n> The optional configuration variable `gc.aggressiveWindow` controls how\n> much time is spent optimizing the delta compression of the objects in\n> the repository when the --aggressive option is specified. The larger\n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index ccfb1ceaeb..b7dd206f41 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -20,6 +20,7 @@\n> #include \"sigchain.h\"\n> #include \"argv-array.h\"\n> #include \"commit.h\"\n> +#include \"commit-graph.h\"\n> #include \"packfile.h\"\n> #include \"object-store.h\"\n> #include \"pack.h\"\n> @@ -40,6 +41,7 @@ static int aggressive_depth = 50;\n> static int aggressive_window = 250;\n> static int gc_auto_threshold = 6700;\n> static int gc_auto_pack_limit = 50;\n> +static int gc_write_commit_graph = 0;\n> static int detach_auto = 1;\n> static timestamp_t gc_log_expire_time;\n> static const char *gc_log_expire = \"1.day.ago\";\n> @@ -129,6 +131,7 @@ static void gc_config(void)\n> \tgit_config_get_int(\"gc.aggressivedepth\", &aggressive_depth);\n> \tgit_config_get_int(\"gc.auto\", &gc_auto_threshold);\n> \tgit_config_get_int(\"gc.autopacklimit\", &gc_auto_pack_limit);\n> +\tgit_config_get_bool(\"gc.writecommitgraph\", &gc_write_commit_graph);\n> \tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n> \tgit_config_get_expiry(\"gc.pruneexpire\", &prune_expire);\n> \tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n> @@ -641,6 +644,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n> \tif (pack_garbage.nr > 0)\n> \t\tclean_pack_garbage();\n> \n> +\tif (gc_write_commit_graph)\n> +\t\twrite_commit_graph_reachable(get_object_directory(), 0);\n> +\n> \tif (auto_gc && too_many_loose_objects())\n> \t\twarning(_(\"There are too many unreachable loose objects; \"\n> \t\t\t\"run 'git prune' to remove them.\"));\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 2208c266b5..5947de3d24 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -245,6 +245,20 @@ test_expect_success 'perform fast-forward merge in full repo' '\n> \ttest_cmp expect output\n> '\n> \n> +test_expect_success 'check that gc computes commit-graph' '\n> +\tcd \"$TRASH_DIRECTORY/full\" &&\n> +\tgit commit --allow-empty -m \"blank\" &&\n> +\tgit commit-graph write --reachable &&\n> +\tcp $objdir/info/commit-graph commit-graph-before-gc &&\n> +\tgit reset --hard HEAD~1 &&\n> +\tgit config gc.writeCommitGraph true &&\n> +\tgit gc &&\n> +\tcp $objdir/info/commit-graph commit-graph-after-gc &&\n> +\t! test_cmp commit-graph-before-gc commit-graph-after-gc &&\n> +\tgit commit-graph write --reachable &&\n> +\ttest_cmp commit-graph-after-gc $objdir/info/commit-graph\n> +'\n> +\n> # the verify tests below expect the commit-graph to contain\n> # exactly the commits reachable from the commits/8 branch.\n> # If the file changes the set of commits in the list, then the\n> -- \n> 2.18.0.24.g1b579a2ee9\n> \n> \n"},{"id":"355300","messageId":"20180812201845.2923-1-szeder.dev@gmail.com","threadId":"48647","inReplyTo":"20180627132447.142473-22-dstolee@microsoft.com","subject":"[PATCH] t5318: use 'test_cmp_bin' to compare commit-graph files","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-08-12T20:18:45Z","receivedAt":"2018-08-12T20:18:59Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"The commit-graph files are binary files, so they should not be\ncompared with 'test_cmp', because that might cause issues on Windows,\nwhere 'test_cmp' is a shell function to deal with random LF-CRLF\nconversions.\n\nUse 'test_cmp_bin' instead.\n\nSigned-off-by: SZEDER Gábor \n---\n t/t5318-commit-graph.sh | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 4f17d7701e..1c148ebf21 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -254,9 +254,9 @@ test_expect_success 'check that gc computes commit-graph' '\n \tgit config gc.writeCommitGraph true &&\n \tgit gc &&\n \tcp $objdir/info/commit-graph commit-graph-after-gc &&\n-\t! test_cmp commit-graph-before-gc commit-graph-after-gc &&\n+\t! test_cmp_bin commit-graph-before-gc commit-graph-after-gc &&\n \tgit commit-graph write --reachable &&\n-\ttest_cmp commit-graph-after-gc $objdir/info/commit-graph\n+\ttest_cmp_bin commit-graph-after-gc $objdir/info/commit-graph\n '\n \n # the verify tests below expect the commit-graph to contain\n-- \n2.18.0.408.g42635c01bc\n\n"},{"id":"355322","messageId":"f07a0a8f-bf19-0895-e48c-4daa9246df98@gmail.com","threadId":"48647","inReplyTo":"20180812201845.2923-1-szeder.dev@gmail.com","subject":"Re: [PATCH] t5318: use 'test_cmp_bin' to compare commit-graph files","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-08-13T11:24:34Z","receivedAt":"2018-08-13T11:24:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/12/2018 4:18 PM, SZEDER Gábor wrote:\n> The commit-graph files are binary files, so they should not be\n> compared with 'test_cmp', because that might cause issues on Windows,\n> where 'test_cmp' is a shell function to deal with random LF-CRLF\n> conversions.\n>\n> Use 'test_cmp_bin' instead.\n>\n> Signed-off-by: SZEDER Gábor \n> ---\n> t/t5318-commit-graph.sh | 4 ++--\n> 1 file changed, 2 insertions(+), 2 deletions(-)\n>\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 4f17d7701e..1c148ebf21 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -254,9 +254,9 @@ test_expect_success 'check that gc computes commit-graph' '\n> \tgit config gc.writeCommitGraph true &&\n> \tgit gc &&\n> \tcp $objdir/info/commit-graph commit-graph-after-gc &&\n> -\t! test_cmp commit-graph-before-gc commit-graph-after-gc &&\n> +\t! test_cmp_bin commit-graph-before-gc commit-graph-after-gc &&\n> \tgit commit-graph write --reachable &&\n> -\ttest_cmp commit-graph-after-gc $objdir/info/commit-graph\n> +\ttest_cmp_bin commit-graph-after-gc $objdir/info/commit-graph\n> '\n> \n> # the verify tests below expect the commit-graph to contain\n\nThanks, Szeder! I didn't know about test_cmp_bin, and I appreciate you \nkeeping up test hygiene.\n\nReviewed-by: Derrick Stolee \n"},{"id":"355346","messageId":"CAM0VKjmhnMfND8HwAvLChgZ6iFYuHUpD=C=3sqjkF9GUsLjpVQ@mail.gmail.com","threadId":"48647","inReplyTo":"f07a0a8f-bf19-0895-e48c-4daa9246df98@gmail.com","subject":"Re: [PATCH] t5318: use 'test_cmp_bin' to compare commit-graph files","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-08-13T11:41:14Z","receivedAt":"2018-08-13T11:41:29Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Aug 13, 2018 at 1:24 PM Derrick Stolee wrote:\n>\n> On 8/12/2018 4:18 PM, SZEDER Gábor wrote:\n> > The commit-graph files are binary files, so they should not be\n> > compared with 'test_cmp', because that might cause issues on Windows,\n> > where 'test_cmp' is a shell function to deal with random LF-CRLF\n> > conversions.\n> >\n> > Use 'test_cmp_bin' instead.\n\nHang on a minute, this is not the commit message I wanted to send\nout...\n"},{"id":"355350","messageId":"20180813115243.28295-1-szeder.dev@gmail.com","threadId":"48647","inReplyTo":"CAM0VKjmhnMfND8HwAvLChgZ6iFYuHUpD=C=3sqjkF9GUsLjpVQ@mail.gmail.com","subject":"[PATCH v2] t5318: use 'test_cmp_bin' to compare commit-graph files","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-08-13T11:52:43Z","receivedAt":"2018-08-13T11:53:17Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"The commit-graph files are binary files, so they should not be\ncompared with 'test_cmp', because that might cause issues like\ncrashing[1] or infinite loop[2] on Windows, where 'test_cmp' is a\nshell function to deal with random LF-CRLF conversions[3].\n\nUse 'test_cmp_bin' instead.\n\n1 - b93e6e3663 (t5000, t5003: do not use test_cmp to compare binary\n files, 2014-06-04)\n2 - f9f3851b4d (t9300: use test_cmp_bin instead of test_cmp to compare\n binary files, 2014-09-12)\n3 - 4d715ac05c (Windows: a test_cmp that is agnostic to random LF <>\n CRLF conversions, 2013-10-26)\n\nReviewed-by: Derrick Stolee \nSigned-off-by: SZEDER Gábor \n---\n\nSame diff, updated commit message, including Derrick's Reviewed-by.\n\n t/t5318-commit-graph.sh | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 4f17d7701e..1c148ebf21 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -254,9 +254,9 @@ test_expect_success 'check that gc computes commit-graph' '\n \tgit config gc.writeCommitGraph true &&\n \tgit gc &&\n \tcp $objdir/info/commit-graph commit-graph-after-gc &&\n-\t! test_cmp commit-graph-before-gc commit-graph-after-gc &&\n+\t! test_cmp_bin commit-graph-before-gc commit-graph-after-gc &&\n \tgit commit-graph write --reachable &&\n-\ttest_cmp commit-graph-after-gc $objdir/info/commit-graph\n+\ttest_cmp_bin commit-graph-after-gc $objdir/info/commit-graph\n '\n \n # the verify tests below expect the commit-graph to contain\n-- \n2.18.0.408.g42635c01bc\n\n"},{"id":"357853","messageId":"CAP8UFD3kzZo8qQqj=8T-TXbvb9b-KD=Ok7TgdV--Cb8MT4NCiA@mail.gmail.com","threadId":"48647","inReplyTo":"20180627132447.142473-21-dstolee@microsoft.com","subject":"Re: [PATCH v7 20/22] commit-graph: add '--reachable' option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-09-11T05:22:10Z","receivedAt":"2018-09-11T05:22:13Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Jun 27, 2018 at 3:24 PM, Derrick Stolee wrote:\n> When writing commit-graph files, it can be convenient to ask for all\n> reachable commits (starting at the ref set) in the resulting file. This\n> is particularly helpful when writing to stdin is complicated, such as a\n> future integration with 'git gc'.\n\nIt would be nice if the \"Future Work\" section of\nDocumentation/technical/commit-graph.txt had something about\nintegration with 'git gc'.\n\n> With the `--stdin-commits` option, generate the new commit graph by\n> walking commits starting at the commits specified in stdin as a list\n> of OIDs in hex, one OID per line. (Cannot be combined with\n> ---stdin-packs.)\n> +`--stdin-packs` or `--reachable`.)\n> ++\n> +With the `--reachable` option, generate the new commit graph by walking\n> +commits starting at all refs. (Cannot be combined with `--stdin-commits`\n> +or `--stdin-packs`.)\n> +\n> With the `--append` option, include all commits that are present in the\n> existing commit-graph file.\n\nThe \"EXAMPLES\" section still contains:\n\n* Write a graph file containing all reachable commits.\n+\n------------------------------------------------\n$ git show-ref -s | git commit-graph write --stdin-commits\n------------------------------------------------\n\nI wonder if this should have been changed to use `--reachable`.\n\nThanks!\n"},{"id":"357859","messageId":"69879118-2753-971e-9818-3ebd4a927ffc@gmail.com","threadId":"48647","inReplyTo":"CAP8UFD3kzZo8qQqj=8T-TXbvb9b-KD=Ok7TgdV--Cb8MT4NCiA@mail.gmail.com","subject":"Re: [PATCH v7 20/22] commit-graph: add '--reachable' option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-09-11T11:19:46Z","receivedAt":"2018-09-11T11:19:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/11/2018 1:22 AM, Christian Couder wrote:\n> On Wed, Jun 27, 2018 at 3:24 PM, Derrick Stolee wrote:\n>> When writing commit-graph files, it can be convenient to ask for all\n>> reachable commits (starting at the ref set) in the resulting file. This\n>> is particularly helpful when writing to stdin is complicated, such as a\n>> future integration with 'git gc'.\n> It would be nice if the \"Future Work\" section of\n> Documentation/technical/commit-graph.txt had something about\n> integration with 'git gc'.\n\nHi Christian,\n\nI'm a bit confused about this statement, because at the point in time of \nthis patch we had a spot in the \"Future Work\" section about \nautomatically updating the graph _somewhere_. The \"future integration \nwith 'git gc'\" I refer to in this patch is implemented in PATCH 21/22. \nIn PATCH 22/22 we removed this section from the technical doc:\n\n-\n-- The current design uses the 'commit-graph' subcommand to generate the graph.\n- When this feature stabilizes enough to recommend to most users, we should\n- add automatic graph writes to common operations that create many commits.\n- For example, one could compute a graph on 'clone', 'fetch', or 'repack'\n- commands.\n-\n\nNow that this feature is shipped in Git 2.19.0, this no longer belongs \nin \"Future Work\".\n\n>> With the `--stdin-commits` option, generate the new commit graph by\n>> walking commits starting at the commits specified in stdin as a list\n>> of OIDs in hex, one OID per line. (Cannot be combined with\n>> ---stdin-packs.)\n>> +`--stdin-packs` or `--reachable`.)\n>> ++\n>> +With the `--reachable` option, generate the new commit graph by walking\n>> +commits starting at all refs. (Cannot be combined with `--stdin-commits`\n>> +or `--stdin-packs`.)\n>> +\n>> With the `--append` option, include all commits that are present in the\n>> existing commit-graph file.\n> The \"EXAMPLES\" section still contains:\n>\n> * Write a graph file containing all reachable commits.\n> +\n> ------------------------------------------------\n> $ git show-ref -s | git commit-graph write --stdin-commits\n> ------------------------------------------------\n>\n> I wonder if this should have been changed to use `--reachable`.\n>\n> Thanks!\nThis is a good idea. I can work on that.\n\nThanks,\n-Stolee\n"},{"id":"357862","messageId":"CAP8UFD3Fm2ZHH8TtmpZNnWn2_LvFWKaB+HzAcrWK0ea1HyOE+Q@mail.gmail.com","threadId":"48647","inReplyTo":"69879118-2753-971e-9818-3ebd4a927ffc@gmail.com","subject":"Re: [PATCH v7 20/22] commit-graph: add '--reachable' option","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-09-11T15:13:59Z","receivedAt":"2018-09-11T15:14:02Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Tue, Sep 11, 2018 at 1:19 PM, Derrick Stolee wrote:\n> On 9/11/2018 1:22 AM, Christian Couder wrote:\n\n>> It would be nice if the \"Future Work\" section of\n>> Documentation/technical/commit-graph.txt had something about\n>> integration with 'git gc'.\n>\n> I'm a bit confused about this statement, because at the point in time of\n> this patch we had a spot in the \"Future Work\" section about automatically\n> updating the graph _somewhere_. The \"future integration with 'git gc'\" I\n> refer to in this patch is implemented in PATCH 21/22. In PATCH 22/22 we\n> removed this section from the technical doc:\n\nYeah, sorry I had not seen PATCH 21/22. I was interested in an update\nabout the commit graph feature and especially to know if the way to\nuse it had been simplified in v2.19.0, so I took a look at the\nDocumentation/technical/commit-graph.txt and\nDocumentation/git-commit-graph.txt, but I didn't find anything there\nexcept the `--reachable` option. So I thought that integrating with gc\nwas still something that needed to be worked on.\n\n> Now that this feature is shipped in Git 2.19.0, this no longer belongs in\n> \"Future Work\".\n\nThis is great!\n\nIt would be nice though to have something in\nDocumentation/technical/commit-graph.txt and perhaps also\nDocumentation/git-commit-graph.txt about it, as it is easy to overlook\nthe gc.writeCommitGraph documentation in Documentation/config.txt.\n\n>> The \"EXAMPLES\" section still contains:\n>>\n>> * Write a graph file containing all reachable commits.\n>> +\n>> ------------------------------------------------\n>> $ git show-ref -s | git commit-graph write --stdin-commits\n>> ------------------------------------------------\n>>\n>> I wonder if this should have been changed to use `--reachable`.\n>>\n>> Thanks!\n>\n> This is a good idea. I can work on that.\n\nThanks,\nChristian.\n"}]}