{"thread":{"id":"48311","subject":"[RFC PATCH 01/12] fixup! commit-graph: always load commit-graph information","startedAt":"2018-04-17T18:10:46Z","lastAt":"2018-06-04T14:42:43Z","messageCount":149,"participants":["Derrick Stolee","Jakub Narebski","Ævar Arnfjörð Bjarmason","Martin Ågren","Stefan Beller","Junio C Hamano","brian m. carlson","SZEDER Gábor","Duy Nguyen"],"isPatch":true,"patchVersion":1,"patchTotal":12},"messages":[{"id":"344904","messageId":"20180417181028.198397-2-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180417181028.198397-1-dstolee@microsoft.com","subject":"[RFC PATCH 01/12] fixup! commit-graph: always load commit-graph information","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T18:10:38Z","receivedAt":"2018-04-17T18:10:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"---\n commit-graph.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 21e853c21a..3f0c142603 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -304,7 +304,7 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \t\t*pos = item->graph_pos;\n \t\treturn 1;\n \t} else {\n-\t\treturn bsearch_graph(commit_graph, &(item->object.oid), pos);\n+\t\treturn bsearch_graph(g, &(item->object.oid), pos);\n \t}\n }\n \n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344905","messageId":"20180417181028.198397-5-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180417181028.198397-1-dstolee@microsoft.com","subject":"[RFC PATCH 04/12] commit-graph: parse commit from chosen graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T18:10:40Z","receivedAt":"2018-04-17T18:10:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Before checking a commit-graph file against the object database, we\nneed to parse all commits from the given commit-graph file. Create\nparse_commit_in_graph_one() to target a given struct commit_graph.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 18 ++++++++++++++----\n 1 file changed, 14 insertions(+), 4 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex c5e5a0f860..6d0d303a7a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -308,17 +308,27 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \t}\n }\n \n-int parse_commit_in_graph(struct commit *item)\n+int parse_commit_in_graph_one(struct commit_graph *g, struct commit *item)\n {\n \tuint32_t pos;\n \n \tif (item->object.parsed)\n-\t\treturn 0;\n+\t\treturn 1;\n+\n+\tif (find_commit_in_graph(item, g, &pos))\n+\t\treturn fill_commit_in_graph(item, g, pos);\n+\n+\treturn 0;\n+}\n+\n+int parse_commit_in_graph(struct commit *item)\n+{\n \tif (!core_commit_graph)\n \t\treturn 0;\n+\n \tprepare_commit_graph();\n-\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n-\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n+\tif (commit_graph)\n+\t\treturn parse_commit_in_graph_one(commit_graph, item);\n \treturn 0;\n }\n \n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344906","messageId":"20180417181028.198397-7-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180417181028.198397-1-dstolee@microsoft.com","subject":"[RFC PATCH 06/12] commit: force commit to parse from object database","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T18:10:41Z","receivedAt":"2018-04-17T18:10:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In anticipation of checking commit-graph file contents against the\nobject database, create parse_commit_internal() to allow side-stepping\nthe commit-graph file and parse directly from the object database.\n\nDue to the use of generation numbers, this method should not be called\nunless the intention is explicit in avoiding commits from the\ncommit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 14 ++++++++++----\n commit.h | 1 +\n 2 files changed, 11 insertions(+), 4 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex 9ef6f699bd..07752d8503 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -392,7 +392,8 @@ int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long s\n \treturn 0;\n }\n \n-int parse_commit_gently(struct commit *item, int quiet_on_missing)\n+\n+int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph)\n {\n \tenum object_type type;\n \tvoid *buffer;\n@@ -403,17 +404,17 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \t\treturn -1;\n \tif (item->object.parsed)\n \t\treturn 0;\n-\tif (parse_commit_in_graph(item))\n+\tif (use_commit_graph && parse_commit_in_graph(item))\n \t\treturn 0;\n \tbuffer = read_sha1_file(item->object.oid.hash, &type, &size);\n \tif (!buffer)\n \t\treturn quiet_on_missing ? -1 :\n \t\t\terror(\"Could not read %s\",\n-\t\t\t oid_to_hex(&item->object.oid));\n+\t\t\t\t\toid_to_hex(&item->object.oid));\n \tif (type != OBJ_COMMIT) {\n \t\tfree(buffer);\n \t\treturn error(\"Object %s not a commit\",\n-\t\t\t oid_to_hex(&item->object.oid));\n+\t\t\t\toid_to_hex(&item->object.oid));\n \t}\n \tret = parse_commit_buffer(item, buffer, size, 0);\n \tif (save_commit_buffer && !ret) {\n@@ -424,6 +425,11 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \treturn ret;\n }\n \n+int parse_commit_gently(struct commit *item, int quiet_on_missing)\n+{\n+\treturn parse_commit_internal(item, quiet_on_missing, 1);\n+}\n+\n void parse_commit_or_die(struct commit *item)\n {\n \tif (parse_commit(item))\ndiff --git a/commit.h b/commit.h\nindex b5afde1ae9..5fde74fcd7 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -73,6 +73,7 @@ struct commit *lookup_commit_reference_by_name(const char *name);\n struct commit *lookup_commit_or_die(const struct object_id *oid, const char *ref_name);\n \n int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph);\n+int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph);\n int parse_commit_gently(struct commit *item, int quiet_on_missing);\n static inline int parse_commit(struct commit *item)\n {\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344907","messageId":"20180417181028.198397-8-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180417181028.198397-1-dstolee@microsoft.com","subject":"[RFC PATCH 07/12] commit-graph: load a root tree from specific graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T18:10:42Z","receivedAt":"2018-04-17T18:10:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When lazy-loading a tree for a commit, it will be important to select\nthe tree from a specific struct commit_graph. Create a new method that\nspecifies the commit-graph file and use that in\nget_commit_tree_in_graph().\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 10 ++++++++--\n 1 file changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 6e3c08cd5c..80a2ac2a6a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -354,14 +354,20 @@ static struct tree *load_tree_for_commit(struct commit_graph *g, struct commit *\n \treturn c->maybe_tree;\n }\n \n-struct tree *get_commit_tree_in_graph(const struct commit *c)\n+static struct tree *get_commit_tree_in_graph_one(struct commit_graph *g,\n+\t\t\t\t\t\t const struct commit *c)\n {\n \tif (c->maybe_tree)\n \t\treturn c->maybe_tree;\n \tif (c->graph_pos == COMMIT_NOT_FROM_GRAPH)\n \t\tBUG(\"get_commit_tree_in_graph called from non-commit-graph commit\");\n \n-\treturn load_tree_for_commit(commit_graph, (struct commit *)c);\n+\treturn load_tree_for_commit(g, (struct commit *)c);\n+}\n+\n+struct tree *get_commit_tree_in_graph(const struct commit *c)\n+{\n+\treturn get_commit_tree_in_graph_one(commit_graph, c);\n }\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344908","messageId":"20180417181028.198397-11-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180417181028.198397-1-dstolee@microsoft.com","subject":"[RFC PATCH 10/12] commit-graph: add '--reachable' option","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T18:10:44Z","receivedAt":"2018-04-17T18:10:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When writing commit-graph files, it can be convenient to ask for all\nreachable commits (starting at the ref set) in the resulting file. This\nis particularly helpful when writing to stdin is complicated, such as a\nfuture integration with 'git gc' which will call\n'git commit-graph write --reachable' after performing cleanup of the\nobject database.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 8 ++++--\n builtin/commit-graph.c | 41 +++++++++++++++++++++++++++---\n t/t5318-commit-graph.sh | 10 ++++++++\n 3 files changed, 53 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 93c7841ba2..1b14d40590 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -37,12 +37,16 @@ Write a commit graph file based on the commits found in packfiles.\n +\n With the `--stdin-packs` option, generate the new commit graph by\n walking objects only in the specified pack-indexes. (Cannot be combined\n-with --stdin-commits.)\n+with --stdin-commits or --reachable.)\n +\n With the `--stdin-commits` option, generate the new commit graph by\n walking commits starting at the commits specified in stdin as a list\n of OIDs in hex, one OID per line. (Cannot be combined with\n---stdin-packs.)\n+--stdin-packs or --reachable.)\n++\n+With the `--reachable` option, generate the new commit graph by walking\n+commits starting at all refs. (Cannot be combined with --stdin-commits\n+or --stind-packs.)\n +\n With the `--append` option, include all commits that are present in the\n existing commit-graph file.\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 77c1a04932..a89285ada8 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -3,13 +3,14 @@\n #include \"dir.h\"\n #include \"lockfile.h\"\n #include \"parse-options.h\"\n+#include \"refs.h\"\n #include \"commit-graph.h\"\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph check [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -24,12 +25,13 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n static struct opts_commit_graph {\n \tconst char *obj_dir;\n+\tint reachable;\n \tint stdin_packs;\n \tint stdin_commits;\n \tint append;\n@@ -113,6 +115,25 @@ static int graph_read(int argc, const char **argv)\n \treturn 0;\n }\n \n+struct hex_list {\n+\tchar **hex_strs;\n+\tint hex_nr;\n+\tint hex_alloc;\n+};\n+\n+static int add_ref_to_list(const char *refname,\n+\t\t\t const struct object_id *oid,\n+\t\t\t int flags, void *cb_data)\n+{\n+\tstruct hex_list *list = (struct hex_list*)cb_data;\n+\n+\tALLOC_GROW(list->hex_strs, list->hex_nr + 1, list->hex_alloc);\n+\tlist->hex_strs[list->hex_nr] = xcalloc(GIT_MAX_HEXSZ + 1, 1);\n+\tstrcpy(list->hex_strs[list->hex_nr], oid_to_hex(oid));\n+\tlist->hex_nr++;\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tconst char **pack_indexes = NULL;\n@@ -127,6 +148,8 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n \t\t\tN_(\"dir\"),\n \t\t\tN_(\"The object directory to store the graph\")),\n+\t\tOPT_BOOL(0, \"reachable\", &opts.reachable,\n+\t\t\tN_(\"start walk at all refs\")),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\tN_(\"scan pack-indexes listed by stdin for commits\")),\n \t\tOPT_BOOL(0, \"stdin-commits\", &opts.stdin_commits,\n@@ -140,8 +163,8 @@ static int graph_write(int argc, const char **argv)\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\n \n-\tif (opts.stdin_packs && opts.stdin_commits)\n-\t\tdie(_(\"cannot use both --stdin-commits and --stdin-packs\"));\n+\tif (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)\n+\t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n \n@@ -164,6 +187,16 @@ static int graph_write(int argc, const char **argv)\n \t\t\tcommit_hex = lines;\n \t\t\tcommits_nr = lines_nr;\n \t\t}\n+\t} else if (opts.reachable) {\n+\t\tstruct hex_list list;\n+\t\tlist.hex_nr = 0;\n+\t\tlist.hex_alloc = 128;\n+\t\tALLOC_ARRAY(list.hex_strs, list.hex_alloc);\n+\n+\t\tfor_each_ref(add_ref_to_list, &list);\n+\n+\t\tcommit_hex = (const char **)list.hex_strs;\n+\t\tcommits_nr = list.hex_nr;\n \t}\n \n \twrite_commit_graph(opts.obj_dir,\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex e91053271a..ccadd22f57 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -200,6 +200,16 @@ test_expect_success 'build graph from commits with append' '\n graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n \n+test_expect_success 'build graph using --reachable' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph write --reachable &&\n+\ttest_path_is_file $objdir/info/commit-graph &&\n+\tgraph_read_expect \"11\" \"large_edges\"\n+'\n+\n+graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n+graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n+\n test_expect_success 'setup bare repo' '\n \tcd \"$TRASH_DIRECTORY\" &&\n \tgit clone --bare --no-local full bare &&\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344909","messageId":"20180417181028.198397-13-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180417181028.198397-1-dstolee@microsoft.com","subject":"[RFC PATCH 12/12] commit-graph: update design document","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T18:10:45Z","receivedAt":"2018-04-17T18:10:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph feature is now integrated with 'fsck' and 'gc',\nso remove those items from the \"Future Work\" section of the\ncommit-graph design document.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 9 ---------\n 1 file changed, 9 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex d9f2713efa..d04657b781 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -118,9 +118,6 @@ Future Work\n - The commit graph feature currently does not honor commit grafts. This can\n be remedied by duplicating or refactoring the current graft logic.\n \n-- The 'commit-graph' subcommand does not have a \"verify\" mode that is\n- necessary for integration with fsck.\n-\n - After computing and storing generation numbers, we must make graph\n walks aware of generation numbers to gain the performance benefits they\n enable. This will mostly be accomplished by swapping a commit-date-ordered\n@@ -142,12 +139,6 @@ Future Work\n such as \"ensure_tree_loaded(commit)\" that fully loads a tree before\n using commit->tree.\n \n-- The current design uses the 'commit-graph' subcommand to generate the graph.\n- When this feature stabilizes enough to recommend to most users, we should\n- add automatic graph writes to common operations that create many commits.\n- For example, one could compute a graph on 'clone', 'fetch', or 'repack'\n- commands.\n-\n - A server could provide a commit graph file as part of the network protocol\n to avoid extra calculations by clients. This feature is only of benefit if\n the user is willing to trust the file, because verifying the file is correct\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344910","messageId":"20180417181028.198397-12-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180417181028.198397-1-dstolee@microsoft.com","subject":"[RFC PATCH 11/12] gc: automatically write commit-graph files","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T18:10:45Z","receivedAt":"2018-04-17T18:11:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file is a very helpful feature for speeding up git\noperations. In order to make it more useful, write the commit-graph file\nby default during standard garbage collection operations.\n\nAdd a 'gc.commitGraph' config setting that triggers writing a\ncommit-graph file after any non-trivial 'git gc' command.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-gc.txt | 4 ++++\n builtin/gc.c | 8 ++++++++\n 2 files changed, 12 insertions(+)\n\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 571b5a7e3c..17dd654a59 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -119,6 +119,10 @@ The optional configuration variable `gc.packRefs` determines if\n it within all non-bare repos or it can be set to a boolean value.\n This defaults to true.\n \n+The optional configuration variable 'gc.commitGraph' determines if\n+'git gc' runs 'git commit-graph write'. This can be set to a boolean\n+value. This defaults to false.\n+\n The optional configuration variable `gc.aggressiveWindow` controls how\n much time is spent optimizing the delta compression of the objects in\n the repository when the --aggressive option is specified. The larger\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 77fa720bd0..070f2a7a3d 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -34,6 +34,7 @@ static int aggressive_depth = 50;\n static int aggressive_window = 250;\n static int gc_auto_threshold = 6700;\n static int gc_auto_pack_limit = 50;\n+static int gc_commit_graph = 1;\n static int detach_auto = 1;\n static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n@@ -46,6 +47,7 @@ static struct argv_array repack = ARGV_ARRAY_INIT;\n static struct argv_array prune = ARGV_ARRAY_INIT;\n static struct argv_array prune_worktrees = ARGV_ARRAY_INIT;\n static struct argv_array rerere = ARGV_ARRAY_INIT;\n+static struct argv_array commit_graph = ARGV_ARRAY_INIT;\n \n static struct tempfile *pidfile;\n static struct lock_file log_lock;\n@@ -121,6 +123,7 @@ static void gc_config(void)\n \tgit_config_get_int(\"gc.aggressivedepth\", &aggressive_depth);\n \tgit_config_get_int(\"gc.auto\", &gc_auto_threshold);\n \tgit_config_get_int(\"gc.autopacklimit\", &gc_auto_pack_limit);\n+\tgit_config_get_bool(\"gc.commitgraph\", &gc_commit_graph);\n \tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n \tgit_config_get_expiry(\"gc.pruneexpire\", &prune_expire);\n \tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n@@ -374,6 +377,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \targv_array_pushl(&prune, \"prune\", \"--expire\", NULL);\n \targv_array_pushl(&prune_worktrees, \"worktree\", \"prune\", \"--expire\", NULL);\n \targv_array_pushl(&rerere, \"rerere\", \"gc\", NULL);\n+\targv_array_pushl(&commit_graph, \"commit-graph\", \"write\", \"--reachable\", NULL);\n \n \t/* default expiry time, overwritten in gc_config */\n \tgc_config();\n@@ -480,6 +484,10 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tif (pack_garbage.nr > 0)\n \t\tclean_pack_garbage();\n \n+\tif (gc_commit_graph)\n+\t\tif (run_command_v_opt(commit_graph.argv, RUN_GIT_CMD))\n+\t\t\treturn error(FAILED_RUN, commit_graph.argv[0]);\n+\n \tif (auto_gc && too_many_loose_objects())\n \t\twarning(_(\"There are too many unreachable loose objects; \"\n \t\t\t\"run 'git prune' to remove them.\"));\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344911","messageId":"20180417181028.198397-10-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180417181028.198397-1-dstolee@microsoft.com","subject":"[RFC PATCH 09/12] fsck: check commit-graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T18:10:44Z","receivedAt":"2018-04-17T18:11:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"If a commit-graph file exists, check its contents during 'git fsck'.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/fsck.c | 13 +++++++++++++\n 1 file changed, 13 insertions(+)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex ef78c6c00c..9712f230ba 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -16,6 +16,7 @@\n #include \"streaming.h\"\n #include \"decorate.h\"\n #include \"packfile.h\"\n+#include \"run-command.h\"\n \n #define REACHABLE 0x0001\n #define SEEN 0x0002\n@@ -45,6 +46,7 @@ static int name_objects;\n #define ERROR_REACHABLE 02\n #define ERROR_PACK 04\n #define ERROR_REFS 010\n+#define ERROR_COMMIT_GRAPH 020\n \n static const char *describe_object(struct object *obj)\n {\n@@ -815,5 +817,16 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t}\n \n \tcheck_connectivity();\n+\n+\tif (core_commit_graph) {\n+\t\tstruct child_process commit_graph_check = CHILD_PROCESS_INIT;\n+\t\tconst char *check_argv[] = { \"commit-graph\", \"check\", NULL, NULL };\n+\t\tcommit_graph_check.argv = check_argv;\n+\t\tcommit_graph_check.git_cmd = 1;\n+\n+\t\tif (run_command(&commit_graph_check))\n+\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n+\t}\n+\n \treturn errors_found;\n }\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344912","messageId":"20180417181028.198397-9-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180417181028.198397-1-dstolee@microsoft.com","subject":"[RFC PATCH 08/12] commit-graph: verify commit contents against odb","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T18:10:43Z","receivedAt":"2018-04-17T18:11:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When running 'git commit-graph check', compare the contents of the\ncommits that are loaded from the commit-graph file with commits that are\nloaded directly from the object database. This includes checking the\nroot tree object ID, commit date, and parents.\n\nIn addition, verify the generation number calculation is correct for all\ncommits in the commit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 82 ++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 82 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 80a2ac2a6a..b5bce2bac4 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -899,5 +899,87 @@ int check_commit_graph(struct commit_graph *g)\n \t\t\t\t graph_commit->graph_pos, i);\n \t}\n \n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct commit *graph_commit, *odb_commit;\n+\t\tstruct commit_list *graph_parents, *odb_parents;\n+\t\tint num_parents = 0;\n+\n+\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n+\n+\t\tgraph_commit = lookup_commit(&cur_oid);\n+\t\todb_commit = (struct commit *)create_object(cur_oid.hash, alloc_commit_node());\n+\t\tif (parse_commit_internal(odb_commit, 0, 0))\n+\t\t\tgraph_report(_(\"failed to parse %s from object database\"), oid_to_hex(&cur_oid));\n+\n+\t\tif (oidcmp(&get_commit_tree_in_graph_one(g, graph_commit)->object.oid,\n+\t\t\t get_commit_tree_oid(odb_commit)))\n+\t\t\tgraph_report(_(\"root tree object ID for commit %s in commit-graph is %s != %s\"),\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t oid_to_hex(get_commit_tree_oid(graph_commit)),\n+\t\t\t\t oid_to_hex(get_commit_tree_oid(odb_commit)));\n+\n+\t\tif (graph_commit->date != odb_commit->date)\n+\t\t\tgraph_report(_(\"commit date for commit %s in commit-graph is %\"PRItime\" != %\"PRItime\"\"),\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t graph_commit->date,\n+\t\t\t\t odb_commit->date);\n+\n+\n+\t\tgraph_parents = graph_commit->parents;\n+\t\todb_parents = odb_commit->parents;\n+\n+\t\twhile (graph_parents) {\n+\t\t\tnum_parents++;\n+\n+\t\t\tif (odb_parents == NULL)\n+\t\t\t\tgraph_report(_(\"commit-graph parent list for commit %s is too long (%d)\"),\n+\t\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t\t num_parents);\n+\n+\t\t\tif (oidcmp(&graph_parents->item->object.oid, &odb_parents->item->object.oid))\n+\t\t\t\tgraph_report(_(\"commit-graph parent for %s is %s != %s\"),\n+\t\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid),\n+\t\t\t\t\t oid_to_hex(&odb_parents->item->object.oid));\n+\n+\t\t\tgraph_parents = graph_parents->next;\n+\t\t\todb_parents = odb_parents->next;\n+\t\t}\n+\n+\t\tif (odb_parents != NULL)\n+\t\t\tgraph_report(_(\"commit-graph parent list for commit %s terminates early\"),\n+\t\t\t\t oid_to_hex(&cur_oid));\n+\n+\t\tif (graph_commit->generation) {\n+\t\t\tuint32_t max_generation = 0;\n+\t\t\tgraph_parents = graph_commit->parents;\n+\n+\t\t\twhile (graph_parents) {\n+\t\t\t\tif (graph_parents->item->generation == GENERATION_NUMBER_ZERO ||\n+\t\t\t\t graph_parents->item->generation == GENERATION_NUMBER_INFINITY)\n+\t\t\t\t\tgraph_report(_(\"commit-graph has valid generation for %s but not its parent, %s\"),\n+\t\t\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid));\n+\t\t\t\tif (graph_parents->item->generation > max_generation)\n+\t\t\t\t\tmax_generation = graph_parents->item->generation;\n+\t\t\t\tgraph_parents = graph_parents->next;\n+\t\t\t}\n+\n+\t\t\tif (graph_commit->generation != max_generation + 1)\n+\t\t\t\tgraph_report(_(\"commit-graph has incorrect generation for %s\"),\n+\t\t\t\t\t oid_to_hex(&cur_oid));\n+\t\t} else {\n+\t\t\tgraph_parents = graph_commit->parents;\n+\n+\t\t\twhile (graph_parents) {\n+\t\t\t\tif (graph_parents->item->generation)\n+\t\t\t\t\tgraph_report(_(\"commit-graph has generation ZERO for %s but not its parent, %s\"),\n+\t\t\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid));\n+\t\t\t\tgraph_parents = graph_parents->next;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n \treturn check_commit_graph_error;\n }\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344913","messageId":"20180417181028.198397-1-dstolee@microsoft.com","threadId":"48311","inReplyTo":null,"subject":"[RFC PATCH 00/12] Integrate commit-graph into 'fsck' and 'gc'","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T18:10:37Z","receivedAt":"2018-04-17T18:11:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph feature is not useful to end users until the\ncommit-graph file is maintained automatically by Git during normal\nupkeep operations. One natural place to trigger this write is during\n'git gc'.\n\nBefore automatically generating a commit-graph, we need to be able to\nverify the contents of a commit-graph file. Integrate commit-graph\nchecks into 'fsck' that check the commit-graph contents against commits\nin the object database.\n\nThings to think about:\n\n* Are these the right integration points?\n\n* gc.commitGraph defaults to true right now for the purpose of testing,\n but may not be required to start. The goal is to have this default to\n true eventually, but we may want to delay that until the feature is\n stable.\n\n* I implement a \"--reachable\" option to 'git commit-graph write' that\n iterates over all refs. This does the same as \n\n\tgit show-ref -s | git commit-graph write --stdin-commits\n\n but I don't know how to pipe two child processes together inside of Git.\n Perhaps this is a better solution, anyway.\n\nWhat other things should I be considering in this case? I'm unfamiliar\nwith the inner-workings of 'fsck' and 'gc', so this is a new space for me.\n\nThis RFC is based on v3 of ds/generation-numbers, and the first commit\nis a fixup! based on a bug in that version that I caught while prepping\nthis series.\n\nThanks,\n-Stolee\n\nDerrick Stolee (12):\n fixup! commit-graph: always load commit-graph information\n commit-graph: add 'check' subcommand\n commit-graph: check file header information\n commit-graph: parse commit from chosen graph\n commit-graph: check fanout and lookup table\n commit: force commit to parse from object database\n commit-graph: load a root tree from specific graph\n commit-graph: verify commit contents against odb\n fsck: check commit-graph\n commit-graph: add '--reachable' option\n gc: automatically write commit-graph files\n commit-graph: update design document\n\n Documentation/git-commit-graph.txt | 15 +-\n Documentation/git-gc.txt | 4 +\n Documentation/technical/commit-graph.txt | 9 --\n builtin/commit-graph.c | 79 +++++++++-\n builtin/fsck.c | 13 ++\n builtin/gc.c | 8 +\n commit-graph.c | 178 ++++++++++++++++++++++-\n commit-graph.h | 2 +\n commit.c | 14 +-\n commit.h | 1 +\n t/t5318-commit-graph.sh | 15 ++\n 11 files changed, 311 insertions(+), 27 deletions(-)\n\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344914","messageId":"20180417181028.198397-3-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180417181028.198397-1-dstolee@microsoft.com","subject":"[RFC PATCH 02/12] commit-graph: add 'check' subcommand","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T18:10:39Z","receivedAt":"2018-04-17T18:11:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"If the commit-graph file becomes corrupt, we need a way to verify\nits contents match the object database. In the manner of 'git fsck'\nwe will implement a 'git commit-graph check' subcommand to report\nall issues with the file.\n\nAdd the 'check' subcommand to the 'commit-graph' builtin and its\ndocumentation. Add a simple test that ensures the command returns\na zero error code.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 7 +++++-\n builtin/commit-graph.c | 38 ++++++++++++++++++++++++++++++\n commit-graph.c | 5 ++++\n commit-graph.h | 2 ++\n t/t5318-commit-graph.sh | 5 ++++\n 5 files changed, 56 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 4c97b555cc..93c7841ba2 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -9,10 +9,10 @@ git-commit-graph - Write and verify Git commit graph files\n SYNOPSIS\n --------\n [verse]\n+'git commit-graph check' [--object-dir ]\n 'git commit-graph read' [--object-dir ]\n 'git commit-graph write' [--object-dir ]\n \n-\n DESCRIPTION\n -----------\n \n@@ -52,6 +52,11 @@ existing commit-graph file.\n Read a graph file given by the commit-graph file and output basic\n details about the graph file. Used for debugging purposes.\n \n+'check'::\n+\n+Read the commit-graph file and verify its contents against the object\n+database. Used to check for corrupted data.\n+\n \n EXAMPLES\n --------\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 37420ae0fd..77c1a04932 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -7,11 +7,17 @@\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n+\tN_(\"git commit-graph check [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n+static const char * const builtin_commit_graph_check_usage[] = {\n+\tN_(\"git commit-graph check [--object-dir ]\"),\n+\tNULL\n+};\n+\n static const char * const builtin_commit_graph_read_usage[] = {\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tNULL\n@@ -29,6 +35,36 @@ static struct opts_commit_graph {\n \tint append;\n } opts;\n \n+\n+static int graph_check(int argc, const char **argv)\n+{\n+\tstruct commit_graph *graph = 0;\n+\tchar *graph_name;\n+\n+\tstatic struct option builtin_commit_graph_check_options[] = {\n+\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n+\t\t\t N_(\"dir\"),\n+\t\t\t N_(\"The object directory to store the graph\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t builtin_commit_graph_check_options,\n+\t\t\t builtin_commit_graph_check_usage, 0);\n+\n+\tif (!opts.obj_dir)\n+\t\topts.obj_dir = get_object_directory();\n+\n+\tgraph_name = get_commit_graph_filename(opts.obj_dir);\n+\tgraph = load_commit_graph_one(graph_name);\n+\n+\tif (!graph)\n+\t\tdie(\"graph file %s does not exist\", graph_name);\n+\tFREE_AND_NULL(graph_name);\n+\n+\treturn check_commit_graph(graph);\n+}\n+\n static int graph_read(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n@@ -160,6 +196,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n \tif (argc > 0) {\n+\t\tif (!strcmp(argv[0], \"check\"))\n+\t\t\treturn graph_check(argc, argv);\n \t\tif (!strcmp(argv[0], \"read\"))\n \t\t\treturn graph_read(argc, argv);\n \t\tif (!strcmp(argv[0], \"write\"))\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 3f0c142603..cd0634bba0 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -819,3 +819,8 @@ void write_commit_graph(const char *obj_dir,\n \toids.alloc = 0;\n \toids.nr = 0;\n }\n+\n+int check_commit_graph(struct commit_graph *g)\n+{\n+\treturn !g;\n+}\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 96cccb10f3..e8c8d99dff 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -53,4 +53,6 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tint nr_commits,\n \t\t\tint append);\n \n+int check_commit_graph(struct commit_graph *g);\n+\n #endif\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 77d85aefe7..e91053271a 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -230,4 +230,9 @@ test_expect_success 'perform fast-forward merge in full repo' '\n \ttest_cmp expect output\n '\n \n+test_expect_success 'git commit-graph check' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph check >output\n+'\n+\n test_done\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344915","messageId":"20180417181028.198397-4-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180417181028.198397-1-dstolee@microsoft.com","subject":"[RFC PATCH 03/12] commit-graph: check file header information","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T18:10:39Z","receivedAt":"2018-04-17T18:11:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"During a run of 'git commit-graph check', list the issues with the\nheader information in the commit-graph file. Some of this information\nis inferred from the loaded 'struct commit_graph'.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 29 ++++++++++++++++++++++++++++-\n 1 file changed, 28 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex cd0634bba0..c5e5a0f860 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -820,7 +820,34 @@ void write_commit_graph(const char *obj_dir,\n \toids.nr = 0;\n }\n \n+static int check_commit_graph_error;\n+#define graph_report(...) { check_commit_graph_error = 1; printf(__VA_ARGS__); }\n+\n int check_commit_graph(struct commit_graph *g)\n {\n-\treturn !g;\n+\tif (!g) {\n+\t\tgraph_report(_(\"no commit-graph file loaded\"));\n+\t\treturn 1;\n+\t}\n+\n+\tcheck_commit_graph_error = 0;\n+\n+\tif (get_be32(g->data) != GRAPH_SIGNATURE)\n+\t\tgraph_report(_(\"commit-graph file has incorrect header\"));\n+\n+\tif (*(g->data + 4) != 1)\n+\t\tgraph_report(_(\"commit-graph file version is not 1\"));\n+\tif (*(g->data + 5) != GRAPH_OID_VERSION)\n+\t\tgraph_report(_(\"commit-graph OID version is not 1 (SHA1)\"));\n+\n+\tif (!g->chunk_oid_fanout)\n+\t\tgraph_report(_(\"commit-graph is missing the OID Fanout chunk\"));\n+\tif (!g->chunk_oid_lookup)\n+\t\tgraph_report(_(\"commit-graph is missing the OID Lookup chunk\"));\n+\tif (!g->chunk_commit_data)\n+\t\tgraph_report(_(\"commit-graph is missing the Commit Data chunk\"));\n+\tif (g->hash_len != GRAPH_OID_LEN)\n+\t\tgraph_report(_(\"commit-graph has incorrect hash length: %d\"), g->hash_len);\n+\n+\treturn check_commit_graph_error;\n }\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344916","messageId":"20180417181028.198397-6-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180417181028.198397-1-dstolee@microsoft.com","subject":"[RFC PATCH 05/12] commit-graph: check fanout and lookup table","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-04-17T18:10:41Z","receivedAt":"2018-04-17T18:11:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"While running 'git commit-graph check', verify that the object IDs\nare listed in lexicographic order and that the fanout table correctly\nnavigates into that list of object IDs.\n\nIn anticipation of checking the commits in the commit-graph file\nagainst the object database, parse the commits from that file in\nadvance. We perform this parse now to ensure the object cache contains\nonly commits from this commit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 34 ++++++++++++++++++++++++++++++++++\n 1 file changed, 34 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 6d0d303a7a..6e3c08cd5c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -835,6 +835,9 @@ static int check_commit_graph_error;\n \n int check_commit_graph(struct commit_graph *g)\n {\n+\tuint32_t i, cur_fanout_pos = 0;\n+\tstruct object_id prev_oid, cur_oid;\n+\n \tif (!g) {\n \t\tgraph_report(_(\"no commit-graph file loaded\"));\n \t\treturn 1;\n@@ -859,5 +862,36 @@ int check_commit_graph(struct commit_graph *g)\n \tif (g->hash_len != GRAPH_OID_LEN)\n \t\tgraph_report(_(\"commit-graph has incorrect hash length: %d\"), g->hash_len);\n \n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct commit *graph_commit;\n+\n+\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n+\n+\t\tif (i > 0 && oidcmp(&prev_oid, &cur_oid) >= 0)\n+\t\t\tgraph_report(_(\"commit-graph has incorrect oid order: %s then %s\"),\n+\n+\t\toid_to_hex(&prev_oid),\n+\t\toid_to_hex(&cur_oid));\n+\t\toidcpy(&prev_oid, &cur_oid);\n+\n+\t\twhile (cur_oid.hash[0] > cur_fanout_pos) {\n+\t\t\tuint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n+\t\t\tif (i != fanout_value)\n+\t\t\t\tgraph_report(_(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\"),\n+\t\t\t\t\t cur_fanout_pos, fanout_value, i);\n+\n+\t\t\tcur_fanout_pos++;\n+\t\t}\n+\n+\t\tgraph_commit = lookup_commit(&cur_oid);\n+\n+\t\tif (!parse_commit_in_graph_one(g, graph_commit))\n+\t\t\tgraph_report(_(\"failed to parse %s from commit-graph\"), oid_to_hex(&cur_oid));\n+\n+\t\tif (graph_commit->graph_pos != i)\n+\t\t\tgraph_report(_(\"graph_pos for commit %s is %u != %u\"), oid_to_hex(&cur_oid),\n+\t\t\t\t graph_commit->graph_pos, i);\n+\t}\n+\n \treturn check_commit_graph_error;\n }\n-- \n2.17.0.39.g685157f7fb\n\n"},{"id":"344923","messageId":"ab5777ac-f0e3-e983-185a-f1f8c3f0798a@gmail.com","threadId":"48311","inReplyTo":"20180417181028.198397-1-dstolee@microsoft.com","subject":"Re: [RFC PATCH 00/12] Integrate commit-graph into 'fsck' and 'gc'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-17T18:50:47Z","receivedAt":"2018-04-17T18:50:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/17/2018 2:10 PM, Derrick Stolee wrote:\n> The commit-graph feature is not useful to end users until the\n> commit-graph file is maintained automatically by Git during normal\n> upkeep operations. One natural place to trigger this write is during\n> 'git gc'.\n>\n> Before automatically generating a commit-graph, we need to be able to\n> verify the contents of a commit-graph file. Integrate commit-graph\n> checks into 'fsck' that check the commit-graph contents against commits\n> in the object database.\n>\n> Things to think about:\n>\n> * Are these the right integration points?\n>\n> * gc.commitGraph defaults to true right now for the purpose of testing,\n> but may not be required to start. The goal is to have this default to\n> true eventually, but we may want to delay that until the feature is\n> stable.\n>\n> * I implement a \"--reachable\" option to 'git commit-graph write' that\n> iterates over all refs. This does the same as\n>\n> \tgit show-ref -s | git commit-graph write --stdin-commits\n>\n> but I don't know how to pipe two child processes together inside of Git.\n> Perhaps this is a better solution, anyway.\n>\n> What other things should I be considering in this case? I'm unfamiliar\n> with the inner-workings of 'fsck' and 'gc', so this is a new space for me.\n>\n> This RFC is based on v3 of ds/generation-numbers, and the first commit\n> is a fixup! based on a bug in that version that I caught while prepping\n> this series.\n>\n> Thanks,\n> -Stolee\n>\n> Derrick Stolee (12):\n> fixup! commit-graph: always load commit-graph information\n> commit-graph: add 'check' subcommand\n> commit-graph: check file header information\n> commit-graph: parse commit from chosen graph\n> commit-graph: check fanout and lookup table\n> commit: force commit to parse from object database\n> commit-graph: load a root tree from specific graph\n> commit-graph: verify commit contents against odb\n> fsck: check commit-graph\n> commit-graph: add '--reachable' option\n> gc: automatically write commit-graph files\n> commit-graph: update design document\n>\n> Documentation/git-commit-graph.txt | 15 +-\n> Documentation/git-gc.txt | 4 +\n> Documentation/technical/commit-graph.txt | 9 --\n> builtin/commit-graph.c | 79 +++++++++-\n> builtin/fsck.c | 13 ++\n> builtin/gc.c | 8 +\n> commit-graph.c | 178 ++++++++++++++++++++++-\n> commit-graph.h | 2 +\n> commit.c | 14 +-\n> commit.h | 1 +\n> t/t5318-commit-graph.sh | 15 ++\n> 11 files changed, 311 insertions(+), 27 deletions(-)\n\nThis RFC is also available as a GitHub pull request [1]\n\n[1] https://github.com/derrickstolee/git/pull/6\n\n"},{"id":"345110","messageId":"86o9iffhxl.fsf@gmail.com","threadId":"48311","inReplyTo":"20180417181028.198397-3-dstolee@microsoft.com","subject":"Re: [RFC PATCH 02/12] commit-graph: add 'check' subcommand","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-19T13:24:06Z","receivedAt":"2018-04-19T13:24:12Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> If the commit-graph file becomes corrupt, we need a way to verify\n> its contents match the object database. In the manner of 'git fsck'\n> we will implement a 'git commit-graph check' subcommand to report\n> all issues with the file.\n\nBikeshed: should the subcommand be called 'check' or 'verify'?\n\n>\n> Add the 'check' subcommand to the 'commit-graph' builtin and its\n> documentation. Add a simple test that ensures the command returns\n> a zero error code.\n\nIt would be nice to have the information that the 'check' subcommand is\ncurrently a [almost no-op] stub in the subject... but that might not\nhave been possible to fit.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-commit-graph.txt | 7 +++++-\n> builtin/commit-graph.c | 38 ++++++++++++++++++++++++++++++\n> commit-graph.c | 5 ++++\n> commit-graph.h | 2 ++\n> t/t5318-commit-graph.sh | 5 ++++\n> 5 files changed, 56 insertions(+), 1 deletion(-)\n>\n> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> index 4c97b555cc..93c7841ba2 100644\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -9,10 +9,10 @@ git-commit-graph - Write and verify Git commit graph files\n> SYNOPSIS\n> --------\n> [verse]\n> +'git commit-graph check' [--object-dir ]\n> 'git commit-graph read' [--object-dir ]\n> 'git commit-graph write' [--object-dir ]\n\nI still think that [--object-dir ] should be the optional parameter\nto the \"git\" wrapper, not to the \"git commit-graph\" command, i.e.\n\n 'git [--object-dir=] commit-graph '\n\nBut this can be done later, in a separate patch series.\n\n> \n> -\n\nStray change.\n\n> DESCRIPTION\n> -----------\n> \n> @@ -52,6 +52,11 @@ existing commit-graph file.\n> Read a graph file given by the commit-graph file and output basic\n> details about the graph file. Used for debugging purposes.\n> \n> +'check'::\n> +\n> +Read the commit-graph file and verify its contents against the object\n> +database. Used to check for corrupted data.\n> +\n\nI wonder if we should offer to verify file without checking against the\nobject database (which is the costly part, I think). But this too can\nbe added later if needed.\n\n> \n> EXAMPLES\n> --------\n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index 37420ae0fd..77c1a04932 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -7,11 +7,17 @@\n> \n> static char const * const builtin_commit_graph_usage[] = {\n> \tN_(\"git commit-graph [--object-dir ]\"),\n> +\tN_(\"git commit-graph check [--object-dir ]\"),\n> \tN_(\"git commit-graph read [--object-dir ]\"),\n> \tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n> \tNULL\n\nIsn't the case that each command would support the\n[--object-dir ] parameter?\n\n> };\n> \n> +static const char * const builtin_commit_graph_check_usage[] = {\n> +\tN_(\"git commit-graph check [--object-dir ]\"),\n> +\tNULL\n> +};\n> +\n\nLooks good to me.\n\n> static const char * const builtin_commit_graph_read_usage[] = {\n> \tN_(\"git commit-graph read [--object-dir ]\"),\n> \tNULL\n> @@ -29,6 +35,36 @@ static struct opts_commit_graph {\n> \tint append;\n> } opts;\n> \n> +\n> +static int graph_check(int argc, const char **argv)\n> +{\n> +\tstruct commit_graph *graph = 0;\n\nThis is NULL, isn't it? Shouldn't it be stated as such?\n\n> +\tchar *graph_name;\n> +\n> +\tstatic struct option builtin_commit_graph_check_options[] = {\n> +\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n> +\t\t\t N_(\"dir\"),\n> +\t\t\t N_(\"The object directory to store the graph\")),\n\nThis again is not the directory to _store_ the graph; this is the\ndirectory to _read_ graph from, or directory where the commit graph is\n_stored_.\n\n> +\t\tOPT_END(),\n> +\t};\n> +\n> +\targc = parse_options(argc, argv, NULL,\n> +\t\t\t builtin_commit_graph_check_options,\n> +\t\t\t builtin_commit_graph_check_usage, 0);\n> +\n> +\tif (!opts.obj_dir)\n> +\t\topts.obj_dir = get_object_directory();\n> +\n> +\tgraph_name = get_commit_graph_filename(opts.obj_dir);\n> +\tgraph = load_commit_graph_one(graph_name);\n> +\n> +\tif (!graph)\n> +\t\tdie(\"graph file %s does not exist\", graph_name);\n\nShouldn't we quote pathname? Shouldn't this error message be marked for\ntranslation? Shouldn't we use \"commit graph file\" explicitly?\n\n> +\tFREE_AND_NULL(graph_name);\n> +\n> +\treturn check_commit_graph(graph);\n> +}\n> +\n> static int graph_read(int argc, const char **argv)\n> {\n> \tstruct commit_graph *graph = NULL;\n> @@ -160,6 +196,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n> \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n> \n> \tif (argc > 0) {\n> +\t\tif (!strcmp(argv[0], \"check\"))\n> +\t\t\treturn graph_check(argc, argv);\n> \t\tif (!strcmp(argv[0], \"read\"))\n> \t\t\treturn graph_read(argc, argv);\n> \t\tif (!strcmp(argv[0], \"write\"))\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 3f0c142603..cd0634bba0 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -819,3 +819,8 @@ void write_commit_graph(const char *obj_dir,\n> \toids.alloc = 0;\n> \toids.nr = 0;\n> }\n> +\n> +int check_commit_graph(struct commit_graph *g)\n> +{\n> +\treturn !g;\n> +}\n\nI understand that it is just a start of implementing this feature, but\nit looks a bit strange that 'read' command does more sanity checks that\nthe 'check' command...\n\n> diff --git a/commit-graph.h b/commit-graph.h\n> index 96cccb10f3..e8c8d99dff 100644\n> --- a/commit-graph.h\n> +++ b/commit-graph.h\n> @@ -53,4 +53,6 @@ void write_commit_graph(const char *obj_dir,\n> \t\t\tint nr_commits,\n> \t\t\tint append);\n> \n> +int check_commit_graph(struct commit_graph *g);\n> +\n> #endif\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 77d85aefe7..e91053271a 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -230,4 +230,9 @@ test_expect_success 'perform fast-forward merge in full repo' '\n> \ttest_cmp expect output\n> '\n> \n> +test_expect_success 'git commit-graph check' '\n> +\tcd \"$TRASH_DIRECTORY/full\" &&\n> +\tgit commit-graph check >output\n> +'\n\nThere should also be negative check, that 'git commit-graph check' fails\nif there is no commit-graph file, isn't it?\n\n> +\n> test_done\n\nBest,\n-- \nJakub Narębski\n"},{"id":"345113","messageId":"86fu3rfas5.fsf@gmail.com","threadId":"48311","inReplyTo":"20180417181028.198397-4-dstolee@microsoft.com","subject":"Re: [RFC PATCH 03/12] commit-graph: check file header information","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-19T15:58:34Z","receivedAt":"2018-04-19T15:58:41Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> During a run of 'git commit-graph check', list the issues with the\n> header information in the commit-graph file. Some of this information\n> is inferred from the loaded 'struct commit_graph'.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 29 ++++++++++++++++++++++++++++-\n> 1 file changed, 28 insertions(+), 1 deletion(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index cd0634bba0..c5e5a0f860 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -820,7 +820,34 @@ void write_commit_graph(const char *obj_dir,\n> \toids.nr = 0;\n> }\n> \n> +static int check_commit_graph_error;\n> +#define graph_report(...) { check_commit_graph_error = 1; printf(__VA_ARGS__); }\n\nShouldn't 'do { ... } while(0);' trick be used here, like e.g. for\ntrace_performance macro?\n\n> +\n> int check_commit_graph(struct commit_graph *g)\n> {\n> -\treturn !g;\n> +\tif (!g) {\n> +\t\tgraph_report(_(\"no commit-graph file loaded\"));\n> +\t\treturn 1;\n> +\t}\n> +\n> +\tcheck_commit_graph_error = 0;\n> +\n\nThe load_commit_graph_one() function does its own checks, some of whose\nare present below, and some of whose are missing.\n\nIf it is used, then why duplicate tests - you would not get here as you\nwould die earlier.\n\nIf it is not used, then some tests are missing.\n\n> +\tif (get_be32(g->data) != GRAPH_SIGNATURE)\n> +\t\tgraph_report(_(\"commit-graph file has incorrect header\"));\n\nThe load_commit_graph_one() shows more detailed information:\n\n \"graph signature %X does not match signature %X\",\n\t\t graph_signature, GRAPH_SIGNATURE)\n\nAlso, load_commit_graph_one() checks that the file is not too short, and\nwe actually can access whole header.\n\n> +\n> +\tif (*(g->data + 4) != 1)\n> +\t\tgraph_report(_(\"commit-graph file version is not 1\"));\n\nAgain:\n\n \"graph version %X does not match version %X\",\n\t\t graph_version, GRAPH_VERSION\n\nAlso, here we hardcode the commit-graph file version to 1.\n\nAccidentally, don't we offer backward compatibility, in that if git can\nread commit-graph file version 2, it can also read commit-graph file\nversion 1?\n\n> +\tif (*(g->data + 5) != GRAPH_OID_VERSION)\n> +\t\tgraph_report(_(\"commit-graph OID version is not 1 (SHA1)\"));\n\nIn one part we use symbolic constant, on the other hardcoded values. If\nGRAPH_OID_VERSION changes, what then?\n\nAlso:\n\n \"hash version %X does not match version %X\",\n\t\t hash_version, GRAPH_OID_VERSION\n\n> +\n> +\tif (!g->chunk_oid_fanout)\n> +\t\tgraph_report(_(\"commit-graph is missing the OID Fanout chunk\"));\n> +\tif (!g->chunk_oid_lookup)\n> +\t\tgraph_report(_(\"commit-graph is missing the OID Lookup chunk\"));\n> +\tif (!g->chunk_commit_data)\n> +\t\tgraph_report(_(\"commit-graph is missing the Commit Data chunk\"));\n\nAll right.\n\n> +\tif (g->hash_len != GRAPH_OID_LEN)\n> +\t\tgraph_report(_(\"commit-graph has incorrect hash length: %d\"), g->hash_len);\n\nWe could be more detailed in error report: what hash length should be,\nthen?\n\n> +\n> +\treturn check_commit_graph_error;\n> }\n\nNo tests of malformed commit-graph file?\n"},{"id":"345114","messageId":"86bmeff6yf.fsf@gmail.com","threadId":"48311","inReplyTo":"20180417181028.198397-5-dstolee@microsoft.com","subject":"Re: [RFC PATCH 04/12] commit-graph: parse commit from chosen graph","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-19T17:21:12Z","receivedAt":"2018-04-19T17:21:18Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> Before checking a commit-graph file against the object database, we\n\nActually there is quite a few checks more that can be done without\naccessing the object database... I'll take a look at later commits why\nthis one is that relatively early in the series.\n\n> need to parse all commits from the given commit-graph file. Create\n> parse_commit_in_graph_one() to target a given struct commit_graph.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 18 ++++++++++++++----\n> 1 file changed, 14 insertions(+), 4 deletions(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index c5e5a0f860..6d0d303a7a 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -308,17 +308,27 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n> \t}\n> }\n> \n> -int parse_commit_in_graph(struct commit *item)\n> +int parse_commit_in_graph_one(struct commit_graph *g, struct commit *item)\n> {\n> \tuint32_t pos;\n> \n> \tif (item->object.parsed)\n> -\t\treturn 0;\n> +\t\treturn 1;\n\nI am confused and befuddled by those apparent changes between returning\n0 or returning 1 when object was parsed.\n\n> +\n> +\tif (find_commit_in_graph(item, g, &pos))\n> +\t\treturn fill_commit_in_graph(item, g, pos);\n> +\n> +\treturn 0;\n> +}\n> +\n> +int parse_commit_in_graph(struct commit *item)\n> +{\n> \tif (!core_commit_graph)\n> \t\treturn 0;\n> +\n> \tprepare_commit_graph();\n> -\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n> -\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n> +\tif (commit_graph)\n> +\t\treturn parse_commit_in_graph_one(commit_graph, item);\n> \treturn 0;\n> }\n\nSeems all right.\n"},{"id":"345192","messageId":"864lk6fics.fsf@gmail.com","threadId":"48311","inReplyTo":"20180417181028.198397-6-dstolee@microsoft.com","subject":"Re: [RFC PATCH 05/12] commit-graph: check fanout and lookup table","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-20T07:27:15Z","receivedAt":"2018-04-20T07:27:23Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> While running 'git commit-graph check', verify that the object IDs\n> are listed in lexicographic order and that the fanout table correctly\n> navigates into that list of object IDs.\n\nAll right. I think we can also sanity check the fanout table (for\nexample that it has 256 elements), see below.\n\n>\n> In anticipation of checking the commits in the commit-graph file\n> against the object database, parse the commits from that file in\n> advance. We perform this parse now to ensure the object cache contains\n> only commits from this commit-graph file.\n\nI guess this part could be a separate commit (a separate patch), because\nit is not connected to the earlier part.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 34 ++++++++++++++++++++++++++++++++++\n> 1 file changed, 34 insertions(+)\n\nNo tests that it detects broken commit-graph file (e.g. one that is\ntruncated)?\n\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 6d0d303a7a..6e3c08cd5c 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -835,6 +835,9 @@ static int check_commit_graph_error;\n> \n> int check_commit_graph(struct commit_graph *g)\n> {\n> +\tuint32_t i, cur_fanout_pos = 0;\n> +\tstruct object_id prev_oid, cur_oid;\n> +\n> \tif (!g) {\n> \t\tgraph_report(_(\"no commit-graph file loaded\"));\n> \t\treturn 1;\n> @@ -859,5 +862,36 @@ int check_commit_graph(struct commit_graph *g)\n> \tif (g->hash_len != GRAPH_OID_LEN)\n> \t\tgraph_report(_(\"commit-graph has incorrect hash length: %d\"), g->hash_len);\n> \n> +\tfor (i = 0; i < g->num_commits; i++) {\n> +\t\tstruct commit *graph_commit;\n> +\n> +\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n> +\n> +\t\tif (i > 0 && oidcmp(&prev_oid, &cur_oid) >= 0)\n> +\t\t\tgraph_report(_(\"commit-graph has incorrect oid order: %s then %s\"),\n\nGood. Reporting what problem is; we could have also reported the\nposition at which there is this problem.\n\n> +\n> +\t\toid_to_hex(&prev_oid),\n> +\t\toid_to_hex(&cur_oid));\n> +\t\toidcpy(&prev_oid, &cur_oid);\n> +\n> +\t\twhile (cur_oid.hash[0] > cur_fanout_pos) {\n> +\t\t\tuint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n> +\t\t\tif (i != fanout_value)\n> +\t\t\t\tgraph_report(_(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\"),\n\nGood. Reporting details of the problem.\n\n> +\t\t\t\t\t cur_fanout_pos, fanout_value, i);\n> +\n> +\t\t\tcur_fanout_pos++;\n> +\t\t}\n\nOne thing you don't check here is that fanout is closed, that is all the\nrest of fanout data up to 256th element (if they are any) all points at\nthe same position past the last element of OID Lookup chunk.\n\n> +\n> +\t\tgraph_commit = lookup_commit(&cur_oid);\n> +\n> +\t\tif (!parse_commit_in_graph_one(g, graph_commit))\n> +\t\t\tgraph_report(_(\"failed to parse %s from commit-graph\"), oid_to_hex(&cur_oid));\n\nDoesn't whis check Commit Data (CDAT) chunk, and therefore should better\nbe in a separate commit?\n\n> +\n> +\t\tif (graph_commit->graph_pos != i)\n> +\t\t\tgraph_report(_(\"graph_pos for commit %s is %u != %u\"), oid_to_hex(&cur_oid),\n> +\t\t\t\t graph_commit->graph_pos, i);\n\nHmmm... it seems to me that the above does not check that commit-graph\nfile is correct, but that the parsing code is correct.\n\n> +\t}\n> +\n> \treturn check_commit_graph_error;\n> }\n"},{"id":"345213","messageId":"86tvs6dqj7.fsf@gmail.com","threadId":"48311","inReplyTo":"20180417181028.198397-7-dstolee@microsoft.com","subject":"Re: [RFC PATCH 06/12] commit: force commit to parse from object database","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-20T12:13:32Z","receivedAt":"2018-04-20T12:13:39Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> In anticipation of checking commit-graph file contents against the\n> object database, create parse_commit_internal() to allow side-stepping\n> the commit-graph file and parse directly from the object database.\n\nNitpick/Bikeshed painting: do we have any naming convention for such\nfunctions (*_internal() here)?\n\n>\n> Due to the use of generation numbers, this method should not be called\n> unless the intention is explicit in avoiding commits from the\n> commit-graph file.\n\nLooks good to me, except for some stray whitespace changes in the\npatch.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit.c | 14 ++++++++++----\n> commit.h | 1 +\n> 2 files changed, 11 insertions(+), 4 deletions(-)\n>\n> diff --git a/commit.c b/commit.c\n> index 9ef6f699bd..07752d8503 100644\n> --- a/commit.c\n> +++ b/commit.c\n> @@ -392,7 +392,8 @@ int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long s\n> \treturn 0;\n> }\n> \n> -int parse_commit_gently(struct commit *item, int quiet_on_missing)\n> +\n\nStray empty line, though I think it may improve readability of the code\nby using two empty lines between separate functions.\n\nBut to be consistent with the rest of the file, there shouldn't be this\nextra empty line.\n\n> +int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph)\n> {\n> \tenum object_type type;\n> \tvoid *buffer;\n> @@ -403,17 +404,17 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n> \t\treturn -1;\n> \tif (item->object.parsed)\n> \t\treturn 0;\n> -\tif (parse_commit_in_graph(item))\n> +\tif (use_commit_graph && parse_commit_in_graph(item))\n> \t\treturn 0;\n\nAll right.\n\n> \tbuffer = read_sha1_file(item->object.oid.hash, &type, &size);\n> \tif (!buffer)\n> \t\treturn quiet_on_missing ? -1 :\n> \t\t\terror(\"Could not read %s\",\n> -\t\t\t oid_to_hex(&item->object.oid));\n> +\t\t\t\t\toid_to_hex(&item->object.oid));\n\nStray whitespace change (looks like spaces to tabs conversion).\n\n> \tif (type != OBJ_COMMIT) {\n> \t\tfree(buffer);\n> \t\treturn error(\"Object %s not a commit\",\n> -\t\t\t oid_to_hex(&item->object.oid));\n> +\t\t\t\toid_to_hex(&item->object.oid));\n\nStray whitespace change (looks like spaces to tabs conversion).\n\n> \t}\n> \tret = parse_commit_buffer(item, buffer, size, 0);\n> \tif (save_commit_buffer && !ret) {\n> @@ -424,6 +425,11 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n> \treturn ret;\n> }\n> \n> +int parse_commit_gently(struct commit *item, int quiet_on_missing)\n> +{\n> +\treturn parse_commit_internal(item, quiet_on_missing, 1);\n> +}\n\nAll right; if it is internal details of implementations, I don't mind\nthis slightly cryptic \"1\" as the value of last parameters.\n\n> +\n> void parse_commit_or_die(struct commit *item)\n> {\n> \tif (parse_commit(item))\n> diff --git a/commit.h b/commit.h\n> index b5afde1ae9..5fde74fcd7 100644\n> --- a/commit.h\n> +++ b/commit.h\n> @@ -73,6 +73,7 @@ struct commit *lookup_commit_reference_by_name(const char *name);\n> struct commit *lookup_commit_or_die(const struct object_id *oid, const char *ref_name);\n> \n> int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph);\n> +int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph);\n> int parse_commit_gently(struct commit *item, int quiet_on_missing);\n> static inline int parse_commit(struct commit *item)\n> {\n\nAll right.\n"},{"id":"345216","messageId":"86o9iedqbr.fsf@gmail.com","threadId":"48311","inReplyTo":"20180417181028.198397-8-dstolee@microsoft.com","subject":"Re: [RFC PATCH 07/12] commit-graph: load a root tree from specific graph","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-20T12:18:00Z","receivedAt":"2018-04-20T12:18:05Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> When lazy-loading a tree for a commit, it will be important to select\n> the tree from a specific struct commit_graph. Create a new method that\n> specifies the commit-graph file and use that in\n> get_commit_tree_in_graph().\n>\n> Signed-off-by: Derrick Stolee \n\nLooks good to me.\n\n> ---\n> commit-graph.c | 10 ++++++++--\n> 1 file changed, 8 insertions(+), 2 deletions(-)\n[...]\n"},{"id":"345233","messageId":"86h8o5eset.fsf@gmail.com","threadId":"48311","inReplyTo":"20180417181028.198397-9-dstolee@microsoft.com","subject":"Re: [RFC PATCH 08/12] commit-graph: verify commit contents against odb","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-20T16:47:38Z","receivedAt":"2018-04-20T16:47:44Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\nOne more check that could been done, and which do not require accessing\nthe object database, would be testing correctness of the Large Edge List\n(EDGE) chunk.\n\nFor each commit in the commit-graph (in the Commit Data (CDAT) chunk),\ncheck if it has more than two parents (if the value for second parent is\ndifferent from 0xffffffff but has the most significant bit set). If\nthere is any such commit, then.\n\n1. Check that EDGE chunk exists\n2. For each octopus merge:\n - check that the index into EDGE array is less than number of its\n elements (sanity check the index)\n - for each parent in the EDGE list, check if the position is valid:\n is less than number of commits in the commit-graph\n - check that list of parents in EDGE terminates\n3. If feasible, also check that\n - all entries in EDGE chunk are referenced directly or indirectly\n - number of parent list terminators (with most significant bit set)\n is equal to the number of octopus merges (no overlap of parent\n lists) -- if it is considered an error\n\n> When running 'git commit-graph check', compare the contents of the\n> commits that are loaded from the commit-graph file with commits that are\n> loaded directly from the object database. This includes checking the\n> root tree object ID, commit date, and parents.\n\nAll right, this part requires checking the object database.\n\n>\n> In addition, verify the generation number calculation is correct for all\n> commits in the commit-graph file.\n\nBut if I understand the code correctly, this one does not require\naccessing the object database. This is entirely separate check, and in\nmy opinion it should be a separate commit (a separate patch).\n\nAlso, there might be a problem with handling GENERATION_NUMBER_MAX.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 82 ++++++++++++++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 82 insertions(+)\n\nI guess testing for this would be hard - you would need to create\ninvalid commit-graph file...\n\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 80a2ac2a6a..b5bce2bac4 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -899,5 +899,87 @@ int check_commit_graph(struct commit_graph *g)\n> \t\t\t\t graph_commit->graph_pos, i);\n> \t}\n> \n> +\tfor (i = 0; i < g->num_commits; i++) {\n> +\t\tstruct commit *graph_commit, *odb_commit;\n> +\t\tstruct commit_list *graph_parents, *odb_parents;\n> +\t\tint num_parents = 0;\n> +\n> +\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n> +\n> +\t\tgraph_commit = lookup_commit(&cur_oid);\n> +\t\todb_commit = (struct commit *)create_object(cur_oid.hash, alloc_commit_node());\n> +\t\tif (parse_commit_internal(odb_commit, 0, 0))\n> +\t\t\tgraph_report(_(\"failed to parse %s from object database\"), oid_to_hex(&cur_oid));\n\nIs it an error to have commit in the commit-graph that is not present in\nthe object database?\n\nIt may happen (if commit-graph is not automatically updated on gc and\npruning), that since creating the commit-graph, the user have deleted\nthe branch and pruned object database -- then commit-graph can contain\nobjects that are not present in the object database.\n\n> +\n> +\t\tif (oidcmp(&get_commit_tree_in_graph_one(g, graph_commit)->object.oid,\n> +\t\t\t get_commit_tree_oid(odb_commit)))\n> +\t\t\tgraph_report(_(\"root tree object ID for commit %s in commit-graph is %s != %s\"),\n> +\t\t\t\t oid_to_hex(&cur_oid),\n> +\t\t\t\t oid_to_hex(get_commit_tree_oid(graph_commit)),\n> +\t\t\t\t oid_to_hex(get_commit_tree_oid(odb_commit)));\n\nLooks good to me, nicely detailed error message.\n\n> +\n> +\t\tif (graph_commit->date != odb_commit->date)\n> +\t\t\tgraph_report(_(\"commit date for commit %s in commit-graph is %\"PRItime\" != %\"PRItime\"\"),\n> +\t\t\t\t oid_to_hex(&cur_oid),\n> +\t\t\t\t graph_commit->date,\n> +\t\t\t\t odb_commit->date);\n\nLooks good to me, nicely detailed error message.\nIt is good to have the same format of the message.\n\n> +\n> +\n> +\t\tgraph_parents = graph_commit->parents;\n> +\t\todb_parents = odb_commit->parents;\n> +\n> +\t\twhile (graph_parents) {\n> +\t\t\tnum_parents++;\n> +\n> +\t\t\tif (odb_parents == NULL)\n> +\t\t\t\tgraph_report(_(\"commit-graph parent list for commit %s is too long (%d)\"),\n> +\t\t\t\t\t oid_to_hex(&cur_oid),\n> +\t\t\t\t\t num_parents);\n> +\n> +\t\t\tif (oidcmp(&graph_parents->item->object.oid, &odb_parents->item->object.oid))\n> +\t\t\t\tgraph_report(_(\"commit-graph parent for %s is %s != %s\"),\n> +\t\t\t\t\t oid_to_hex(&cur_oid),\n> +\t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid),\n> +\t\t\t\t\t oid_to_hex(&odb_parents->item->object.oid));\n> +\n> +\t\t\tgraph_parents = graph_parents->next;\n> +\t\t\todb_parents = odb_parents->next;\n> +\t\t}\n> +\n> +\t\tif (odb_parents != NULL)\n> +\t\t\tgraph_report(_(\"commit-graph parent list for commit %s terminates early\"),\n> +\t\t\t\t oid_to_hex(&cur_oid));\n\nAll right. We check that there are no parents in odb that are not\npresent in the commit-graph, and vice-versa (that there are no parents\nin commit-graph that are not present in odb), and we check that all\nparents match.\n\nLooks good to me, nicely detailed error message.\nIt is good to have the same format of the message.\n\n> +\n> +\t\tif (graph_commit->generation) {\n\nAll right, here we know that the generation number for the commit is not\nGENERATION_NUMBER_ZERO because we checked for it, and we know that it is\nnot GENERATION_NUMBER_INFINITY because the commit was in commit-graph.\n\n> +\t\t\tuint32_t max_generation = 0;\n> +\t\t\tgraph_parents = graph_commit->parents;\n> +\n> +\t\t\twhile (graph_parents) {\n> +\t\t\t\tif (graph_parents->item->generation == GENERATION_NUMBER_ZERO ||\n> +\t\t\t\t graph_parents->item->generation == GENERATION_NUMBER_INFINITY)\n> +\t\t\t\t\tgraph_report(_(\"commit-graph has valid generation for %s but not its parent, %s\"),\n\nWhat about the case of GENERATION_NUMBER_MAX? It is an error if parent\nhas GENERATION_NUMBER_MAX and the commit does not, but it would be\ncaught by the following check, if with the less helpful error message.\n\nBut if commit has GENERATION_NUMBER_MAX, then its parents can have also\nGENERATION_NUMBER_MAX, and the following check would fail event if it is\nvalid, isn't it?\n\n> +\t\t\t\t\t\t oid_to_hex(&cur_oid),\n> +\t\t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid));\n> +\t\t\t\tif (graph_parents->item->generation > max_generation)\n> +\t\t\t\t\tmax_generation = graph_parents->item->generation;\n> +\t\t\t\tgraph_parents = graph_parents->next;\n> +\t\t\t}\n> +\n> +\t\t\tif (graph_commit->generation != max_generation + 1)\n> +\t\t\t\tgraph_report(_(\"commit-graph has incorrect generation for %s\"),\n> +\t\t\t\t\t oid_to_hex(&cur_oid));\n\nI wonder if we might have to treat the case of parent-less commits (root\ncommits) special, but I guess that would complicate the code for no\nreason.\n\nThough perhaps adding \" (root commit)\" suffix would be a good idea.\nStill complicates code, though.\n\n> +\t\t} else {\n> +\t\t\tgraph_parents = graph_commit->parents;\n> +\n> +\t\t\twhile (graph_parents) {\n> +\t\t\t\tif (graph_parents->item->generation)\n> +\t\t\t\t\tgraph_report(_(\"commit-graph has generation ZERO for %s but not its parent, %s\"),\n\nGood check.\n\n> +\t\t\t\t\t\t oid_to_hex(&cur_oid),\n> +\t\t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid));\n> +\t\t\t\tgraph_parents = graph_parents->next;\n> +\t\t\t}\n> +\t\t}\n> +\t}\n> +\n> \treturn check_commit_graph_error;\n> }\n"},{"id":"345234","messageId":"86bmedervj.fsf@gmail.com","threadId":"48311","inReplyTo":"20180417181028.198397-10-dstolee@microsoft.com","subject":"Re: [RFC PATCH 09/12] fsck: check commit-graph","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-20T16:59:12Z","receivedAt":"2018-04-20T16:59:17Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> If a commit-graph file exists, check its contents during 'git fsck'.\n\nIs it \"if a commit-graph file exists\", or is it core.commitGraph feature\nis turned on?\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/fsck.c | 13 +++++++++++++\n> 1 file changed, 13 insertions(+)\n>\n> diff --git a/builtin/fsck.c b/builtin/fsck.c\n> index ef78c6c00c..9712f230ba 100644\n> --- a/builtin/fsck.c\n> +++ b/builtin/fsck.c\n> @@ -16,6 +16,7 @@\n> #include \"streaming.h\"\n> #include \"decorate.h\"\n> #include \"packfile.h\"\n> +#include \"run-command.h\"\n\nCouln't this be done internally, without run-command? Or is it just\npreliminary implementation?\n\n> \n> #define REACHABLE 0x0001\n> #define SEEN 0x0002\n> @@ -45,6 +46,7 @@ static int name_objects;\n> #define ERROR_REACHABLE 02\n> #define ERROR_PACK 04\n> #define ERROR_REFS 010\n> +#define ERROR_COMMIT_GRAPH 020\n\nI see that these error status codes are not documented anywhere. Still,\nI would expect at least mentioning commit-graph in the git-fsck manpage.\n\n> \n> static const char *describe_object(struct object *obj)\n> {\n> @@ -815,5 +817,16 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n> \t}\n> \n> \tcheck_connectivity();\n> +\n> +\tif (core_commit_graph) {\n> +\t\tstruct child_process commit_graph_check = CHILD_PROCESS_INIT;\n> +\t\tconst char *check_argv[] = { \"commit-graph\", \"check\", NULL, NULL };\n> +\t\tcommit_graph_check.argv = check_argv;\n> +\t\tcommit_graph_check.git_cmd = 1;\n> +\n> +\t\tif (run_command(&commit_graph_check))\n> +\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n> +\t}\n> +\n> \treturn errors_found;\n> }\n"},{"id":"345236","messageId":"864lk5er1j.fsf@gmail.com","threadId":"48311","inReplyTo":"20180417181028.198397-11-dstolee@microsoft.com","subject":"Re: [RFC PATCH 10/12] commit-graph: add '--reachable' option","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-20T17:17:12Z","receivedAt":"2018-04-20T17:17:18Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> When writing commit-graph files, it can be convenient to ask for all\n> reachable commits (starting at the ref set) in the resulting file. This\n> is particularly helpful when writing to stdin is complicated, such as a\n> future integration with 'git gc' which will call\n> 'git commit-graph write --reachable' after performing cleanup of the\n> object database.\n>\n> Signed-off-by: Derrick Stolee \n\nFor what it is worth, it looks good to me.\n\n> ---\n> Documentation/git-commit-graph.txt | 8 ++++--\n> builtin/commit-graph.c | 41 +++++++++++++++++++++++++++---\n> t/t5318-commit-graph.sh | 10 ++++++++\n> 3 files changed, 53 insertions(+), 6 deletions(-)\n>\n> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> index 93c7841ba2..1b14d40590 100644\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -37,12 +37,16 @@ Write a commit graph file based on the commits found in packfiles.\n> +\n> With the `--stdin-packs` option, generate the new commit graph by\n> walking objects only in the specified pack-indexes. (Cannot be combined\n> -with --stdin-commits.)\n> +with --stdin-commits or --reachable.)\n> +\n> With the `--stdin-commits` option, generate the new commit graph by\n> walking commits starting at the commits specified in stdin as a list\n> of OIDs in hex, one OID per line. (Cannot be combined with\n> ---stdin-packs.)\n> +--stdin-packs or --reachable.)\n> ++\n> +With the `--reachable` option, generate the new commit graph by walking\n> +commits starting at all refs. (Cannot be combined with --stdin-commits\n> +or --stind-packs.)\n\nI wonder if this \"cannot be combined\" is sustainable... ;-)\n\n\n[...]\n> @@ -113,6 +115,25 @@ static int graph_read(int argc, const char **argv)\n> \treturn 0;\n> }\n> \n> +struct hex_list {\n> +\tchar **hex_strs;\n> +\tint hex_nr;\n> +\tint hex_alloc;\n> +};\n> +\n> +static int add_ref_to_list(const char *refname,\n> +\t\t\t const struct object_id *oid,\n> +\t\t\t int flags, void *cb_data)\n> +{\n> +\tstruct hex_list *list = (struct hex_list*)cb_data;\n> +\n> +\tALLOC_GROW(list->hex_strs, list->hex_nr + 1, list->hex_alloc);\n> +\tlist->hex_strs[list->hex_nr] = xcalloc(GIT_MAX_HEXSZ + 1, 1);\n> +\tstrcpy(list->hex_strs[list->hex_nr], oid_to_hex(oid));\n> +\tlist->hex_nr++;\n> +\treturn 0;\n> +}\n> +\n> static int graph_write(int argc, const char **argv)\n> {\n> \tconst char **pack_indexes = NULL;\n> @@ -127,6 +148,8 @@ static int graph_write(int argc, const char **argv)\n> \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n> \t\t\tN_(\"dir\"),\n> \t\t\tN_(\"The object directory to store the graph\")),\n> +\t\tOPT_BOOL(0, \"reachable\", &opts.reachable,\n> +\t\t\tN_(\"start walk at all refs\")),\n> \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n> \t\t\tN_(\"scan pack-indexes listed by stdin for commits\")),\n> \t\tOPT_BOOL(0, \"stdin-commits\", &opts.stdin_commits,\n> @@ -140,8 +163,8 @@ static int graph_write(int argc, const char **argv)\n> \t\t\t builtin_commit_graph_write_options,\n> \t\t\t builtin_commit_graph_write_usage, 0);\n> \n> -\tif (opts.stdin_packs && opts.stdin_commits)\n> -\t\tdie(_(\"cannot use both --stdin-commits and --stdin-packs\"));\n> +\tif (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)\n> +\t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\n\nNice trick, but is it worth it (in place of boolean expression)? Though\nit lines up with the error message, though...\n\n> \tif (!opts.obj_dir)\n> \t\topts.obj_dir = get_object_directory();\n> \n> @@ -164,6 +187,16 @@ static int graph_write(int argc, const char **argv)\n> \t\t\tcommit_hex = lines;\n> \t\t\tcommits_nr = lines_nr;\n> \t\t}\n> +\t} else if (opts.reachable) {\n> +\t\tstruct hex_list list;\n> +\t\tlist.hex_nr = 0;\n> +\t\tlist.hex_alloc = 128;\n> +\t\tALLOC_ARRAY(list.hex_strs, list.hex_alloc);\n> +\n> +\t\tfor_each_ref(add_ref_to_list, &list);\n> +\n> +\t\tcommit_hex = (const char **)list.hex_strs;\n> +\t\tcommits_nr = list.hex_nr;\n\nNice trick!\n\n> \t}\n> \n> \twrite_commit_graph(opts.obj_dir,\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index e91053271a..ccadd22f57 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -200,6 +200,16 @@ test_expect_success 'build graph from commits with append' '\n> graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n> graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n> \n> +test_expect_success 'build graph using --reachable' '\n> +\tcd \"$TRASH_DIRECTORY/full\" &&\n> +\tgit commit-graph write --reachable &&\n> +\ttest_path_is_file $objdir/info/commit-graph &&\n> +\tgraph_read_expect \"11\" \"large_edges\"\n> +'\n> +\n> +graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n> +graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n> +\n> test_expect_success 'setup bare repo' '\n> \tcd \"$TRASH_DIRECTORY\" &&\n> \tgit clone --bare --no-local full bare &&\n"},{"id":"345239","messageId":"86wox1dbos.fsf@gmail.com","threadId":"48311","inReplyTo":"20180417181028.198397-12-dstolee@microsoft.com","subject":"Re: [RFC PATCH 11/12] gc: automatically write commit-graph files","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-20T17:34:11Z","receivedAt":"2018-04-20T17:34:18Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> The commit-graph file is a very helpful feature for speeding up git\n> operations. In order to make it more useful, write the commit-graph file\n> by default during standard garbage collection operations.\n>\n> Add a 'gc.commitGraph' config setting that triggers writing a\n> commit-graph file after any non-trivial 'git gc' command.\n\nOther than the question if 'gc.commitGraph' and 'core.commitGraph'\nshould be independent config variables, and the exact wording of the\ngit-gc docs, it looks good to me.\n\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-gc.txt | 4 ++++\n> builtin/gc.c | 8 ++++++++\n> 2 files changed, 12 insertions(+)\n>\n> diff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\n> index 571b5a7e3c..17dd654a59 100644\n> --- a/Documentation/git-gc.txt\n> +++ b/Documentation/git-gc.txt\n> @@ -119,6 +119,10 @@ The optional configuration variable `gc.packRefs` determines if\n> it within all non-bare repos or it can be set to a boolean value.\n> This defaults to true.\n> \n> +The optional configuration variable 'gc.commitGraph' determines if\n> +'git gc' runs 'git commit-graph write'. This can be set to a boolean\n> +value. This defaults to false.\n> +\n> The optional configuration variable `gc.aggressiveWindow` controls how\n> much time is spent optimizing the delta compression of the objects in\n> the repository when the --aggressive option is specified. The larger\n\n[...]\n"},{"id":"345247","messageId":"87fu3peni2.fsf@evledraar.gmail.com","threadId":"48311","inReplyTo":"86wox1dbos.fsf@gmail.com","subject":"Re: [RFC PATCH 11/12] gc: automatically write commit-graph files","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-04-20T18:33:41Z","receivedAt":"2018-04-20T18:33:47Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Apr 20 2018, Jakub Narebski wrote:\n\n> Derrick Stolee writes:\n>\n>> The commit-graph file is a very helpful feature for speeding up git\n>> operations. In order to make it more useful, write the commit-graph file\n>> by default during standard garbage collection operations.\n>>\n>> Add a 'gc.commitGraph' config setting that triggers writing a\n>> commit-graph file after any non-trivial 'git gc' command.\n>\n> Other than the question if 'gc.commitGraph' and 'core.commitGraph'\n> should be independent config variables, and the exact wording of the\n> git-gc docs, it looks good to me.\n\nSans doc errors you pointed out in other places (you need to set\ncore.commitGraph so it's read at all), I think it's very useful to have\nthese split up. It's simliar to pack.useBitmaps & pack.writeBitmaps.\n\nMakes it easy to start writing them, and then have a quick toggle to\nturn it off if there's any issues rathen than go around deleting the\nfiles or making sure they're not written out.\n"},{"id":"345252","messageId":"86r2n9d78l.fsf@gmail.com","threadId":"48311","inReplyTo":"20180417181028.198397-13-dstolee@microsoft.com","subject":"Re: [RFC PATCH 12/12] commit-graph: update design document","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-20T19:10:18Z","receivedAt":"2018-04-20T19:10:24Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> The commit-graph feature is now integrated with 'fsck' and 'gc',\n> so remove those items from the \"Future Work\" section of the\n> commit-graph design document.\n\nSee comments below, but this looks good to me.\n\nWhat is missing from the \"Future Work\" section (and which was somewhat\nimplied by now removed \"When this feature stabilizes enough to recommend\nto most users\") is safety against history [view] changing features:\ngit-replace, shallow clone and grafts file. I wrote about this in\n\"Re: [PATCH v8 04/14] graph: add commit graph design document\"\nhttps://public-inbox.org/git/86vacsjdcg.fsf@gmail.com/\n\nJN> The problem in my opinion lies in different direction, namely that\nJN> commit grafts can change, changing the view of the history. If we want\nJN> commit-graph file to follow user-visible view of the history of the\nJN> project, it needs to respect current version of commit grafts - but what\nJN> if commit grafts changed since commit-graph file was generated?\nJN> \nJN> Actually, there are currently three ways to affect the view of the\nJN> history:\nJN> \nJN> a. legacy commit grafts mechanism; it was first, but it is not safe,\nJN> cannot be transferred on fetch / push, and is now deprecated.\nJN> \nJN> b. shallow clones, which are kind of specialized and limited grafts;\nJN> they used to limit available functionality, but restrictions are\nJN> being lifted (or perhaps even got lifted)\nJN> \nJN> c. git-replace mechanism, where we can create an \"overlay\" of any\nJN> object, and is intended to be among others replacement for commit\nJN> grafts; safe, transferable, can be turned off with \"git\nJN> --no-replace-objects \"\nJN> \nJN> All those can change; some more likely than others. The problem is if\nJN> they change between writing commit-graph file (respecting old view of\nJN> the history) and reading it (where we expect to see the new view).\nJN> \nJN> a. grafts file can change: lines can be added, removed or changed\nJN> \nJN> b. shallow clones can be deepened or shortened, or even make\nJN> not shallow\nJN> \nJN> c. new replacements can be added, old removed, and existing edited\nJN> \nJN> \nJN> There are, as far as I can see, two ways of handling the issue of Git\nJN> features that can change the view of the project's history, namely:\nJN> \nJN> * Disable commit-graph reading when any of this features are used, and\nJN> always write full graph info.\nJN> \nJN> This may not matter much for shallow clones, where commit count\nJN> should be small anyway, but when using git-replace to stitch together\nJN> current repository with historical one, commit-graph would be\nJN> certainly useful. Also, git-replace does not need to change history.\nJN> \nJN> On the other hand I think it is the easier solution.\nJN> \nJN> Or\nJN> \nJN> * Detect somehow that the view of the history changed, and invalidate\nJN> commit-graph (perhaps even automatically regenerate it).\nJN> \nJN> For shallow clone changes I think one can still use the old\nJN> commit-graph file to generate the new one. For other cases, the\nJN> metadata is simple to modify, but indices such as generation number\nJN> would need to be at least partially calculated anew.\n\nNote that in all cases one can simply discard generation number\ninformation (treating it as if it was ZERO), and use commit-graph as\nvalues before applying history-changing feature: replacements, grafts or\nshallow.\n\nWell, at least for shallow you can do that for generation numbers: using\ngrafts (deprecated) or replacements to join repository with historical\none would mean that we are no longer have commit-graph transitively\nclosed under reachability.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/technical/commit-graph.txt | 9 ---------\n> 1 file changed, 9 deletions(-)\n>\n> diff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\n> index d9f2713efa..d04657b781 100644\n> --- a/Documentation/technical/commit-graph.txt\n> +++ b/Documentation/technical/commit-graph.txt\n> @@ -118,9 +118,6 @@ Future Work\n> - The commit graph feature currently does not honor commit grafts. This can\n> be remedied by duplicating or refactoring the current graft logic.\n> \n> -- The 'commit-graph' subcommand does not have a \"verify\" mode that is\n> - necessary for integration with fsck.\n> -\n\nAll right (though \"verify\" mode is actually done via \"check\" command).\n\n> - After computing and storing generation numbers, we must make graph\n> walks aware of generation numbers to gain the performance benefits they\n> enable. This will mostly be accomplished by swapping a commit-date-ordered\n> @@ -142,12 +139,6 @@ Future Work\n> such as \"ensure_tree_loaded(commit)\" that fully loads a tree before\n> using commit->tree.\n> \n> -- The current design uses the 'commit-graph' subcommand to generate the graph.\n> - When this feature stabilizes enough to recommend to most users, we should\n> - add automatic graph writes to common operations that create many commits.\n> - For example, one could compute a graph on 'clone', 'fetch', or 'repack'\n> - commands.\n> -\n\nI'm not sure if this paragraph should be deleted as a whole; it depends\non whether we decide that using git-gc to do automatic graph writes is\nenough (git-gc should be ran automatically by git if we get many new\nobjects anyway, so this gives us almost \"compute a graph on 'clone',\n'fetch', or 'repack'\").\n\n> - A server could provide a commit graph file as part of the network protocol\n> to avoid extra calculations by clients. This feature is only of benefit if\n> the user is willing to trust the file, because verifying the file is correct\n"},{"id":"347263","messageId":"20180510173345.40577-1-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180417181028.198397-1-dstolee@microsoft.com","subject":"[PATCH 00/12] Integrate commit-graph into fsck, gc, and fetch","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-10T17:34:02Z","receivedAt":"2018-05-10T17:34:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph feature is not useful to end users until the\ncommit-graph file is maintained automatically by Git during normal\nupkeep operations. One natural place to trigger this write is during\n'git gc'.\n\nBefore automatically generating a commit-graph, we need to be able to\nverify the contents of a commit-graph file. Integrate commit-graph\nchecks into 'fsck' that check the commit-graph contents against commits\nin the object database.\n\nThanks, Jakub, for the feedback on the RFC. I think there are still some\nthings to decide at a high-level before we dig too far into the review.\nSpecifically, the integration points in 'fsck', 'gc', and 'fetch' are\nworth considering our alternatives.\n\nFor 'fsck', the current integration is minimal: if core.commitGraph is\ntrue, then 'git fsck' calls 'git commit-graph verify --object-dir=X'\nfor the objects directory and every alternate. There are a few options\nto consider here:\n\n1. Keep this behavior: we should always check the commit-graph if it\n exists.\n\n2. Add a --[no-]commit-graph argument to 'fsck' that toggles the\n commit-graph verification.\n\n3. Remove all direct integration between 'fsck' and 'commit-graph' and\n instead rely on users checking 'git commit-graph verify' manually\n when they suspect a problem with the commit-graph file. While this\n option is worth considering, it is my least favorite since it requires\n more from users.\n\nFor 'gc' and 'fetch', these seemed like natural places to update the\ncommit-graph file. Relative to the other maintenance that occurs during\nthese commands, the 'git commit-graph write' command is fast, especially\nfor incremental updates; only the \"new\" commits are walked when computing\ngeneration numbers and other metadata for the commit-graph file.\n\nThe behavior in this patch series does the following:\n\n1. Near the end of 'git gc', run 'git commit-graph write'. The location\n of this code assumes that a 'git gc --auto' has not terminated early\n due to not meeting the auto threshold.\n\n2. At the end of 'git fetch', run 'git commit-graph write'. This means\n that every reachable commit will be in the commit-graph after a\n a successful fetch, which seems a reasonable frequency. Then, the\n only times we would be missing a reachable commit is after creating\n one locally. There is a problem with the current patch, though: every\n 'git fetch' call runs 'git commit-graph write', even if there were no\n ref updates or objects downloaded. Is there a simple way to detect if\n the fetch was non-trivial?\n\nOne obvious problem with this approach: if we compute this during 'gc'\nAND 'fetch', there will be times where a 'fetch' calls 'gc' and triggers\ntwo commit-graph writes. If I were to abandon one of these patches, it\nwould be the 'fetch' integration. A 'git gc' really wants to delete all\nreferences to unreachable commits, and without updating the commit-graph\nwe may still have commit data in the commit-graph file that is not in\nthe object database. In fact, deleting commits from the object database\nbut not from the commit-graph will cause 'git commit-graph verify' to\nfail!\n\nI welcome discussion on these ideas, as we are venturing out of the\n\"pure data structure\" world and into the \"user experience\" world. I am\nless confident in my skills in this world, but the feature is worthless\nif it does not improve the user experience.\n\nThanks,\n-Stolee\n\nDerrick Stolee (12):\n\nCommits 01-07 focus on the 'git commit-graph verify' subcommand. These\nare ready for full, rigorous review.\n\n commit-graph: add 'verify' subcommand\n commit-graph: verify file header information\n commit-graph: parse commit from chosen graph\n commit-graph: verify fanout and lookup table\n commit: force commit to parse from object database\n commit-graph: load a root tree from specific graph\n commit-graph: verify commit contents against odb\n\nCommit 08 integrates 'git commit-graph verify' into fsck. The work here\nis the minimum integration possible. (See above discussion of options.)\n\n fsck: verify commit-graph\n\nCommit 09 introduces a new '--reachable' option only to make the calls\nfrom 'gc' and 'fetch' simpler. Commits 10-11 integrate writing the\ncommit-graph into 'gc' and 'fetch', respectively. (See above disucssion.)\n\n commit-graph: add '--reachable' option\n gc: automatically write commit-graph files\n fetch: compute commit-graph by default\n\nCommit 12 simply deletes sections from the \"Future Work\" section\nof the commit-graph design document.\n\n commit-graph: update design document\n\n Documentation/config.txt | 10 ++\n Documentation/git-commit-graph.txt | 14 ++-\n Documentation/git-fsck.txt | 3 +\n Documentation/git-gc.txt | 4 +\n Documentation/technical/commit-graph.txt | 22 ----\n builtin/commit-graph.c | 79 +++++++++++++-\n builtin/fetch.c | 13 +++\n builtin/fsck.c | 21 ++++\n builtin/gc.c | 8 ++\n commit-graph.c | 175 ++++++++++++++++++++++++++++++-\n commit-graph.h | 2 +\n commit.c | 13 ++-\n commit.h | 1 +\n t/t5318-commit-graph.sh | 25 +++++\n 14 files changed, 353 insertions(+), 37 deletions(-)\n\n\nbase-commit: 34fdd433396ee0e3ef4de02eb2189f8226eafe4e\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347264","messageId":"20180510173345.40577-2-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180510173345.40577-1-dstolee@microsoft.com","subject":"[PATCH 01/12] commit-graph: add 'verify' subcommand","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-10T17:34:04Z","receivedAt":"2018-05-10T17:34:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In case the commit-graph file becomes corrupt, we need a way to\nverify its contents match the object database. In the manner of\n'git fsck' we will implement a 'git commit-graph verify' subcommand\nto report all issues with the file.\n\nAdd the 'verify' subcommand to the 'commit-graph' builtin and its\ndocumentation. The subcommand is currently a no-op except for\nloading the commit-graph into memory, which may trigger run-time\nerrors that would be caught by normal use. Add a simple test that\nensures the command returns a zero error code.\n\nIf no commit-graph file exists, this is an acceptable state. Do\nnot report any errors.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 6 ++++++\n builtin/commit-graph.c | 38 ++++++++++++++++++++++++++++++++++++++\n commit-graph.c | 5 +++++\n commit-graph.h | 2 ++\n t/t5318-commit-graph.sh | 10 ++++++++++\n 5 files changed, 61 insertions(+)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 4c97b555cc..1daefa7fb1 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git commit-graph read' [--object-dir ]\n+'git commit-graph verify' [--object-dir ]\n 'git commit-graph write' [--object-dir ]\n \n \n@@ -52,6 +53,11 @@ existing commit-graph file.\n Read a graph file given by the commit-graph file and output basic\n details about the graph file. Used for debugging purposes.\n \n+'verify'::\n+\n+Read the commit-graph file and verify its contents against the object\n+database. Used to verify for corrupted data.\n+\n \n EXAMPLES\n --------\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 37420ae0fd..f5d891f2b8 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -7,11 +7,17 @@\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n+\tN_(\"git commit-graph verify [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n+static const char * const builtin_commit_graph_verify_usage[] = {\n+\tN_(\"git commit-graph verify [--object-dir ]\"),\n+\tNULL\n+};\n+\n static const char * const builtin_commit_graph_read_usage[] = {\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tNULL\n@@ -29,6 +35,36 @@ static struct opts_commit_graph {\n \tint append;\n } opts;\n \n+\n+static int graph_verify(int argc, const char **argv)\n+{\n+\tstruct commit_graph *graph = 0;\n+\tchar *graph_name;\n+\n+\tstatic struct option builtin_commit_graph_verify_options[] = {\n+\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n+\t\t\t N_(\"dir\"),\n+\t\t\t N_(\"The object directory to store the graph\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t builtin_commit_graph_verify_options,\n+\t\t\t builtin_commit_graph_verify_usage, 0);\n+\n+\tif (!opts.obj_dir)\n+\t\topts.obj_dir = get_object_directory();\n+\n+\tgraph_name = get_commit_graph_filename(opts.obj_dir);\n+\tgraph = load_commit_graph_one(graph_name);\n+\n+\tif (!graph)\n+\t\treturn 0;\n+\tFREE_AND_NULL(graph_name);\n+\n+\treturn verify_commit_graph(graph);\n+}\n+\n static int graph_read(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n@@ -160,6 +196,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n \tif (argc > 0) {\n+\t\tif (!strcmp(argv[0], \"verify\"))\n+\t\t\treturn graph_verify(argc, argv);\n \t\tif (!strcmp(argv[0], \"read\"))\n \t\t\treturn graph_read(argc, argv);\n \t\tif (!strcmp(argv[0], \"write\"))\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a8c337dd77..b25aaed128 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -817,3 +817,8 @@ void write_commit_graph(const char *obj_dir,\n \toids.alloc = 0;\n \toids.nr = 0;\n }\n+\n+int verify_commit_graph(struct commit_graph *g)\n+{\n+\treturn !g;\n+}\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 96cccb10f3..71a39c5a57 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -53,4 +53,6 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tint nr_commits,\n \t\t\tint append);\n \n+int verify_commit_graph(struct commit_graph *g);\n+\n #endif\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 77d85aefe7..6ca451dfd2 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -11,6 +11,11 @@ test_expect_success 'setup full repo' '\n \tobjdir=\".git/objects\"\n '\n \n+test_expect_success 'verify graph with no graph file' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph verify\n+'\n+\n test_expect_success 'write graph with no packs' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write --object-dir . &&\n@@ -230,4 +235,9 @@ test_expect_success 'perform fast-forward merge in full repo' '\n \ttest_cmp expect output\n '\n \n+test_expect_success 'git commit-graph verify' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph verify >output\n+'\n+\n test_done\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347265","messageId":"20180510173345.40577-3-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180510173345.40577-1-dstolee@microsoft.com","subject":"[PATCH 02/12] commit-graph: verify file header information","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-10T17:34:05Z","receivedAt":"2018-05-10T17:34:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"During a run of 'git commit-graph verify', list the issues with the\nheader information in the commit-graph file. Some of this information\nis inferred from the loaded 'struct commit_graph'. Some header\ninformation is checked as part of load_commit_graph_one().\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 23 ++++++++++++++++++++++-\n 1 file changed, 22 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex b25aaed128..c3b8716c14 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -818,7 +818,28 @@ void write_commit_graph(const char *obj_dir,\n \toids.nr = 0;\n }\n \n+static int verify_commit_graph_error;\n+#define graph_report(...) \\\n+\tdo {\\\n+\t\tverify_commit_graph_error = 1;\\\n+\t\tprintf(__VA_ARGS__);\\\n+\t} while (0);\n+\n int verify_commit_graph(struct commit_graph *g)\n {\n-\treturn !g;\n+\tif (!g) {\n+\t\tgraph_report(_(\"no commit-graph file loaded\"));\n+\t\treturn 1;\n+\t}\n+\n+\tverify_commit_graph_error = 0;\n+\n+\tif (!g->chunk_oid_fanout)\n+\t\tgraph_report(_(\"commit-graph is missing the OID Fanout chunk\"));\n+\tif (!g->chunk_oid_lookup)\n+\t\tgraph_report(_(\"commit-graph is missing the OID Lookup chunk\"));\n+\tif (!g->chunk_commit_data)\n+\t\tgraph_report(_(\"commit-graph is missing the Commit Data chunk\"));\n+\n+\treturn verify_commit_graph_error;\n }\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347266","messageId":"20180510173345.40577-5-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180510173345.40577-1-dstolee@microsoft.com","subject":"[PATCH 04/12] commit-graph: verify fanout and lookup table","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-10T17:34:07Z","receivedAt":"2018-05-10T17:34:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"While running 'git commit-graph verify', verify that the object IDs\nare listed in lexicographic order and that the fanout table correctly\nnavigates into that list of object IDs.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 33 +++++++++++++++++++++++++++++++++\n 1 file changed, 33 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex ce11af1d20..b4c146c423 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -839,6 +839,9 @@ static int verify_commit_graph_error;\n \n int verify_commit_graph(struct commit_graph *g)\n {\n+\tuint32_t i, cur_fanout_pos = 0;\n+\tstruct object_id prev_oid, cur_oid;\n+\n \tif (!g) {\n \t\tgraph_report(_(\"no commit-graph file loaded\"));\n \t\treturn 1;\n@@ -853,5 +856,35 @@ int verify_commit_graph(struct commit_graph *g)\n \tif (!g->chunk_commit_data)\n \t\tgraph_report(_(\"commit-graph is missing the Commit Data chunk\"));\n \n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n+\n+\t\tif (i > 0 && oidcmp(&prev_oid, &cur_oid) >= 0)\n+\t\t\tgraph_report(_(\"commit-graph has incorrect oid order: %s then %s\"),\n+\n+\t\toid_to_hex(&prev_oid),\n+\t\toid_to_hex(&cur_oid));\n+\t\toidcpy(&prev_oid, &cur_oid);\n+\n+\t\twhile (cur_oid.hash[0] > cur_fanout_pos) {\n+\t\t\tuint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n+\t\t\tif (i != fanout_value)\n+\t\t\t\tgraph_report(_(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\"),\n+\t\t\t\t\t cur_fanout_pos, fanout_value, i);\n+\n+\t\t\tcur_fanout_pos++;\n+\t\t}\n+\t}\n+\n+\twhile (cur_fanout_pos < 256) {\n+\t\tuint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n+\n+\t\tif (g->num_commits != fanout_value)\n+\t\t\tgraph_report(_(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\"),\n+\t\t\t\t cur_fanout_pos, fanout_value, i);\n+\n+\t\tcur_fanout_pos++;\n+\t}\n+\n \treturn verify_commit_graph_error;\n }\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347267","messageId":"20180510173345.40577-7-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180510173345.40577-1-dstolee@microsoft.com","subject":"[PATCH 06/12] commit-graph: load a root tree from specific graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-10T17:34:10Z","receivedAt":"2018-05-10T17:34:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When lazy-loading a tree for a commit, it will be important to select\nthe tree from a specific struct commit_graph. Create a new method that\nspecifies the commit-graph file and use that in\nget_commit_tree_in_graph().\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 10 ++++++++--\n 1 file changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex b4c146c423..8c636abba9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -357,14 +357,20 @@ static struct tree *load_tree_for_commit(struct commit_graph *g, struct commit *\n \treturn c->maybe_tree;\n }\n \n-struct tree *get_commit_tree_in_graph(const struct commit *c)\n+static struct tree *get_commit_tree_in_graph_one(struct commit_graph *g,\n+\t\t\t\t\t\t const struct commit *c)\n {\n \tif (c->maybe_tree)\n \t\treturn c->maybe_tree;\n \tif (c->graph_pos == COMMIT_NOT_FROM_GRAPH)\n \t\tBUG(\"get_commit_tree_in_graph called from non-commit-graph commit\");\n \n-\treturn load_tree_for_commit(commit_graph, (struct commit *)c);\n+\treturn load_tree_for_commit(g, (struct commit *)c);\n+}\n+\n+struct tree *get_commit_tree_in_graph(const struct commit *c)\n+{\n+\treturn get_commit_tree_in_graph_one(commit_graph, c);\n }\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347268","messageId":"20180510173345.40577-9-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180510173345.40577-1-dstolee@microsoft.com","subject":"[PATCH 08/12] fsck: verify commit-graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-10T17:34:12Z","receivedAt":"2018-05-10T17:34:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"If core.commitGraph is true, verify the contents of the commit-graph\nduring 'git fsck' using the 'git commit-graph verify' subcommand. Run\nthis check on all alternates, as well.\n\nWe use a new process for two reasons:\n\n1. The subcommand decouples the details of loading and verifying a\n commit-graph file from the other fsck details.\n\n2. The commit-graph verification requires the commits to be loaded\n in a specific order to guarantee we parse from the commit-graph\n file for some objects and from the object database for others.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-fsck.txt | 3 +++\n builtin/fsck.c | 21 +++++++++++++++++++++\n t/t5318-commit-graph.sh | 5 +++++\n 3 files changed, 29 insertions(+)\n\ndiff --git a/Documentation/git-fsck.txt b/Documentation/git-fsck.txt\nindex b9f060e3b2..ab9a93fb9b 100644\n--- a/Documentation/git-fsck.txt\n+++ b/Documentation/git-fsck.txt\n@@ -110,6 +110,9 @@ Any corrupt objects you will have to find in backups or other archives\n (i.e., you can just remove them and do an 'rsync' with some other site in\n the hopes that somebody else has the object you have corrupted).\n \n+If core.commitGraph is true, the commit-graph file will also be inspected\n+using 'git commit-graph verify'. See linkgit:git-commit-graph[1].\n+\n Extracted Diagnostics\n ---------------------\n \ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex ef78c6c00c..a6d5045b77 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -16,6 +16,7 @@\n #include \"streaming.h\"\n #include \"decorate.h\"\n #include \"packfile.h\"\n+#include \"run-command.h\"\n \n #define REACHABLE 0x0001\n #define SEEN 0x0002\n@@ -45,6 +46,7 @@ static int name_objects;\n #define ERROR_REACHABLE 02\n #define ERROR_PACK 04\n #define ERROR_REFS 010\n+#define ERROR_COMMIT_GRAPH 020\n \n static const char *describe_object(struct object *obj)\n {\n@@ -815,5 +817,24 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t}\n \n \tcheck_connectivity();\n+\n+\tif (core_commit_graph) {\n+\t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n+\t\tconst char *verify_argv[] = { \"commit-graph\", \"verify\", NULL, NULL, NULL, NULL };\n+\t\tcommit_graph_verify.argv = verify_argv;\n+\t\tcommit_graph_verify.git_cmd = 1;\n+\n+\t\tif (run_command(&commit_graph_verify))\n+\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n+\n+\t\tprepare_alt_odb();\n+\t\tfor (alt = alt_odb_list; alt; alt = alt->next) {\n+\t\t\tverify_argv[2] = \"--object-dir\";\n+\t\t\tverify_argv[3] = alt->path;\n+\t\t\tif (run_command(&commit_graph_verify))\n+\t\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n+\t\t}\n+\t}\n+\n \treturn errors_found;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 6ca451dfd2..9680e6e6e0 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -240,4 +240,9 @@ test_expect_success 'git commit-graph verify' '\n \tgit commit-graph verify >output\n '\n \n+test_expect_success 'git fsck (checks commit-graph)' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit fsck\n+'\n+\n test_done\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347269","messageId":"20180510173345.40577-13-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180510173345.40577-1-dstolee@microsoft.com","subject":"[PATCH 12/12] commit-graph: update design document","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-10T17:34:16Z","receivedAt":"2018-05-10T17:34:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph feature is now integrated with 'fsck' and 'gc',\nso remove those items from the \"Future Work\" section of the\ncommit-graph design document.\n\nAlso remove the section on lazy-loading trees, as that was completed\nin an earlier patch series.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 22 ----------------------\n 1 file changed, 22 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex e1a883eb46..c664acbd76 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -118,9 +118,6 @@ Future Work\n - The commit graph feature currently does not honor commit grafts. This can\n be remedied by duplicating or refactoring the current graft logic.\n \n-- The 'commit-graph' subcommand does not have a \"verify\" mode that is\n- necessary for integration with fsck.\n-\n - After computing and storing generation numbers, we must make graph\n walks aware of generation numbers to gain the performance benefits they\n enable. This will mostly be accomplished by swapping a commit-date-ordered\n@@ -130,25 +127,6 @@ Future Work\n - 'log --topo-order'\n - 'tag --merged'\n \n-- Currently, parse_commit_gently() requires filling in the root tree\n- object for a commit. This passes through lookup_tree() and consequently\n- lookup_object(). Also, it calls lookup_commit() when loading the parents.\n- These method calls check the ODB for object existence, even if the\n- consumer does not need the content. For example, we do not need the\n- tree contents when computing merge bases. Now that commit parsing is\n- removed from the computation time, these lookup operations are the\n- slowest operations keeping graph walks from being fast. Consider\n- loading these objects without verifying their existence in the ODB and\n- only loading them fully when consumers need them. Consider a method\n- such as \"ensure_tree_loaded(commit)\" that fully loads a tree before\n- using commit->tree.\n-\n-- The current design uses the 'commit-graph' subcommand to generate the graph.\n- When this feature stabilizes enough to recommend to most users, we should\n- add automatic graph writes to common operations that create many commits.\n- For example, one could compute a graph on 'clone', 'fetch', or 'repack'\n- commands.\n-\n - A server could provide a commit graph file as part of the network protocol\n to avoid extra calculations by clients. This feature is only of benefit if\n the user is willing to trust the file, because verifying the file is correct\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347270","messageId":"20180510173345.40577-12-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180510173345.40577-1-dstolee@microsoft.com","subject":"[PATCH 11/12] fetch: compute commit-graph by default","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-10T17:34:15Z","receivedAt":"2018-05-10T17:34:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"During a call to 'git fetch', we expect new commits and updated refs.\nUse these updated refs to add the new commits to the commit-graph file,\nautomatically providing performance benefits in other calls.\n\nUse 'fetch.commitGraph' config setting to enable or disable this\nbehavior. Defaults to false while the commit-graph feature matures.\nSpecifically, we do not want this on by default until the commit-graph\nfeature integrates with history-modifying features such as shallow\nclones.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 4 ++++\n builtin/fetch.c | 13 +++++++++++++\n 2 files changed, 17 insertions(+)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 9a3abd87e7..3d8225600a 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1409,6 +1409,10 @@ fetch.output::\n \t`full` and `compact`. Default value is `full`. See section\n \tOUTPUT in linkgit:git-fetch[1] for detail.\n \n+fetch.commitGraph::\n+\tIf true, fetch will automatically update the commit-graph file.\n+\tSee linkgit:git-commit-graph[1].\n+\n format.attach::\n \tEnable multipart/mixed attachments as the default for\n \t'format-patch'. The value can also be a double quoted string\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 8ee998ea2e..254f6ecfb6 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -38,6 +38,7 @@ enum {\n static int fetch_prune_config = -1; /* unspecified */\n static int prune = -1; /* unspecified */\n #define PRUNE_BY_DEFAULT 0 /* do we prune by default? */\n+static int fetch_commit_graph = 0;\n \n static int all, append, dry_run, force, keep, multiple, update_head_ok, verbosity, deepen_relative;\n static int progress = -1;\n@@ -66,6 +67,11 @@ static int git_fetch_config(const char *k, const char *v, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(k, \"fetch.commitGraph\")) {\n+\t\tfetch_commit_graph = git_config_bool(k, v);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(k, \"submodule.recurse\")) {\n \t\tint r = git_config_bool(k, v) ?\n \t\t\tRECURSE_SUBMODULES_ON : RECURSE_SUBMODULES_OFF;\n@@ -1462,6 +1468,13 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\tresult = fetch_multiple(&list);\n \t}\n \n+\tif (!result && fetch_commit_graph) {\n+\t\tstruct argv_array commit_graph = ARGV_ARRAY_INIT;\n+\t\targv_array_pushl(&commit_graph, \"commit-graph\", \"write\", \"--reachable\", NULL);\n+\t\tif (run_command_v_opt(commit_graph.argv, RUN_GIT_CMD))\n+\t\t\tresult = 1;\n+\t}\n+\n \tif (!result && (recurse_submodules != RECURSE_SUBMODULES_OFF)) {\n \t\tstruct argv_array options = ARGV_ARRAY_INIT;\n \n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347271","messageId":"20180510173345.40577-8-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180510173345.40577-1-dstolee@microsoft.com","subject":"[PATCH 07/12] commit-graph: verify commit contents against odb","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-10T17:34:11Z","receivedAt":"2018-05-10T17:34:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When running 'git commit-graph verify', compare the contents of the\ncommits that are loaded from the commit-graph file with commits that are\nloaded directly from the object database. This includes checking the\nroot tree object ID, commit date, and parents.\n\nParse the commit from the graph during the initial loop through the\nobject IDs to guarantee we parse from the commit-graph file.\n\nIn addition, verify the generation number calculation is correct for all\ncommits in the commit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 88 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 88 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8c636abba9..24f5031f3e 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -863,6 +863,8 @@ int verify_commit_graph(struct commit_graph *g)\n \t\tgraph_report(_(\"commit-graph is missing the Commit Data chunk\"));\n \n \tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct commit *graph_commit;\n+\n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n \t\tif (i > 0 && oidcmp(&prev_oid, &cur_oid) >= 0)\n@@ -880,6 +882,10 @@ int verify_commit_graph(struct commit_graph *g)\n \n \t\t\tcur_fanout_pos++;\n \t\t}\n+\n+\t\tgraph_commit = lookup_commit(&cur_oid);\n+\t\tif (!parse_commit_in_graph_one(g, graph_commit))\n+\t\t\tgraph_report(_(\"failed to parse %s from commit-graph\"), oid_to_hex(&cur_oid));\n \t}\n \n \twhile (cur_fanout_pos < 256) {\n@@ -892,5 +898,87 @@ int verify_commit_graph(struct commit_graph *g)\n \t\tcur_fanout_pos++;\n \t}\n \n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct commit *graph_commit, *odb_commit;\n+\t\tstruct commit_list *graph_parents, *odb_parents;\n+\t\tint num_parents = 0;\n+\n+\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n+\n+\t\tgraph_commit = lookup_commit(&cur_oid);\n+\t\todb_commit = (struct commit *)create_object(cur_oid.hash, alloc_commit_node());\n+\t\tif (parse_commit_internal(odb_commit, 0, 0))\n+\t\t\tgraph_report(_(\"failed to parse %s from object database\"), oid_to_hex(&cur_oid));\n+\n+\t\tif (oidcmp(&get_commit_tree_in_graph_one(g, graph_commit)->object.oid,\n+\t\t\t get_commit_tree_oid(odb_commit)))\n+\t\t\tgraph_report(_(\"root tree object ID for commit %s in commit-graph is %s != %s\"),\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t oid_to_hex(get_commit_tree_oid(graph_commit)),\n+\t\t\t\t oid_to_hex(get_commit_tree_oid(odb_commit)));\n+\n+\t\tif (graph_commit->date != odb_commit->date)\n+\t\t\tgraph_report(_(\"commit date for commit %s in commit-graph is %\"PRItime\" != %\"PRItime\"\"),\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t graph_commit->date,\n+\t\t\t\t odb_commit->date);\n+\n+\n+\t\tgraph_parents = graph_commit->parents;\n+\t\todb_parents = odb_commit->parents;\n+\n+\t\twhile (graph_parents) {\n+\t\t\tnum_parents++;\n+\n+\t\t\tif (odb_parents == NULL)\n+\t\t\t\tgraph_report(_(\"commit-graph parent list for commit %s is too long (%d)\"),\n+\t\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t\t num_parents);\n+\n+\t\t\tif (oidcmp(&graph_parents->item->object.oid, &odb_parents->item->object.oid))\n+\t\t\t\tgraph_report(_(\"commit-graph parent for %s is %s != %s\"),\n+\t\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid),\n+\t\t\t\t\t oid_to_hex(&odb_parents->item->object.oid));\n+\n+\t\t\tgraph_parents = graph_parents->next;\n+\t\t\todb_parents = odb_parents->next;\n+\t\t}\n+\n+\t\tif (odb_parents != NULL)\n+\t\t\tgraph_report(_(\"commit-graph parent list for commit %s terminates early\"),\n+\t\t\t\t oid_to_hex(&cur_oid));\n+\n+\t\tif (graph_commit->generation) {\n+\t\t\tuint32_t max_generation = 0;\n+\t\t\tgraph_parents = graph_commit->parents;\n+\n+\t\t\twhile (graph_parents) {\n+\t\t\t\tif (graph_parents->item->generation == GENERATION_NUMBER_ZERO ||\n+\t\t\t\t graph_parents->item->generation == GENERATION_NUMBER_INFINITY)\n+\t\t\t\t\tgraph_report(_(\"commit-graph has valid generation for %s but not its parent, %s\"),\n+\t\t\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid));\n+\t\t\t\tif (graph_parents->item->generation > max_generation)\n+\t\t\t\t\tmax_generation = graph_parents->item->generation;\n+\t\t\t\tgraph_parents = graph_parents->next;\n+\t\t\t}\n+\n+\t\t\tif (graph_commit->generation != max_generation + 1)\n+\t\t\t\tgraph_report(_(\"commit-graph has incorrect generation for %s\"),\n+\t\t\t\t\t oid_to_hex(&cur_oid));\n+\t\t} else {\n+\t\t\tgraph_parents = graph_commit->parents;\n+\n+\t\t\twhile (graph_parents) {\n+\t\t\t\tif (graph_parents->item->generation)\n+\t\t\t\t\tgraph_report(_(\"commit-graph has generation ZERO for %s but not its parent, %s\"),\n+\t\t\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid));\n+\t\t\t\tgraph_parents = graph_parents->next;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n \treturn verify_commit_graph_error;\n }\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347272","messageId":"20180510173345.40577-10-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180510173345.40577-1-dstolee@microsoft.com","subject":"[PATCH 09/12] commit-graph: add '--reachable' option","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-10T17:34:13Z","receivedAt":"2018-05-10T17:34:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When writing commit-graph files, it can be convenient to ask for all\nreachable commits (starting at the ref set) in the resulting file. This\nis particularly helpful when writing to stdin is complicated, such as a\nfuture integration with 'git gc' which will call\n'git commit-graph write --reachable' after performing cleanup of the\nobject database.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 8 ++++++--\n builtin/commit-graph.c | 41 ++++++++++++++++++++++++++++++++++----\n t/t5318-commit-graph.sh | 10 ++++++++++\n 3 files changed, 53 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 1daefa7fb1..fbab3feba1 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -38,12 +38,16 @@ Write a commit graph file based on the commits found in packfiles.\n +\n With the `--stdin-packs` option, generate the new commit graph by\n walking objects only in the specified pack-indexes. (Cannot be combined\n-with --stdin-commits.)\n+with --stdin-commits or --reachable.)\n +\n With the `--stdin-commits` option, generate the new commit graph by\n walking commits starting at the commits specified in stdin as a list\n of OIDs in hex, one OID per line. (Cannot be combined with\n---stdin-packs.)\n+--stdin-packs or --reachable.)\n++\n+With the `--reachable` option, generate the new commit graph by walking\n+commits starting at all refs. (Cannot be combined with --stdin-commits\n+or --stind-packs.)\n +\n With the `--append` option, include all commits that are present in the\n existing commit-graph file.\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex f5d891f2b8..4c9328cdf2 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -3,13 +3,14 @@\n #include \"dir.h\"\n #include \"lockfile.h\"\n #include \"parse-options.h\"\n+#include \"refs.h\"\n #include \"commit-graph.h\"\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph verify [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -24,12 +25,13 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n static struct opts_commit_graph {\n \tconst char *obj_dir;\n+\tint reachable;\n \tint stdin_packs;\n \tint stdin_commits;\n \tint append;\n@@ -113,6 +115,25 @@ static int graph_read(int argc, const char **argv)\n \treturn 0;\n }\n \n+struct hex_list {\n+\tchar **hex_strs;\n+\tint hex_nr;\n+\tint hex_alloc;\n+};\n+\n+static int add_ref_to_list(const char *refname,\n+\t\t\t const struct object_id *oid,\n+\t\t\t int flags, void *cb_data)\n+{\n+\tstruct hex_list *list = (struct hex_list*)cb_data;\n+\n+\tALLOC_GROW(list->hex_strs, list->hex_nr + 1, list->hex_alloc);\n+\tlist->hex_strs[list->hex_nr] = xcalloc(GIT_MAX_HEXSZ + 1, 1);\n+\tstrcpy(list->hex_strs[list->hex_nr], oid_to_hex(oid));\n+\tlist->hex_nr++;\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tconst char **pack_indexes = NULL;\n@@ -127,6 +148,8 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n \t\t\tN_(\"dir\"),\n \t\t\tN_(\"The object directory to store the graph\")),\n+\t\tOPT_BOOL(0, \"reachable\", &opts.reachable,\n+\t\t\tN_(\"start walk at all refs\")),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\tN_(\"scan pack-indexes listed by stdin for commits\")),\n \t\tOPT_BOOL(0, \"stdin-commits\", &opts.stdin_commits,\n@@ -140,8 +163,8 @@ static int graph_write(int argc, const char **argv)\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\n \n-\tif (opts.stdin_packs && opts.stdin_commits)\n-\t\tdie(_(\"cannot use both --stdin-commits and --stdin-packs\"));\n+\tif (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)\n+\t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n \n@@ -164,6 +187,16 @@ static int graph_write(int argc, const char **argv)\n \t\t\tcommit_hex = lines;\n \t\t\tcommits_nr = lines_nr;\n \t\t}\n+\t} else if (opts.reachable) {\n+\t\tstruct hex_list list;\n+\t\tlist.hex_nr = 0;\n+\t\tlist.hex_alloc = 128;\n+\t\tALLOC_ARRAY(list.hex_strs, list.hex_alloc);\n+\n+\t\tfor_each_ref(add_ref_to_list, &list);\n+\n+\t\tcommit_hex = (const char **)list.hex_strs;\n+\t\tcommits_nr = list.hex_nr;\n \t}\n \n \twrite_commit_graph(opts.obj_dir,\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 9680e6e6e0..82f95eb11f 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -205,6 +205,16 @@ test_expect_success 'build graph from commits with append' '\n graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n \n+test_expect_success 'build graph using --reachable' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph write --reachable &&\n+\ttest_path_is_file $objdir/info/commit-graph &&\n+\tgraph_read_expect \"11\" \"large_edges\"\n+'\n+\n+graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n+graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n+\n test_expect_success 'setup bare repo' '\n \tcd \"$TRASH_DIRECTORY\" &&\n \tgit clone --bare --no-local full bare &&\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347273","messageId":"20180510173345.40577-11-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180510173345.40577-1-dstolee@microsoft.com","subject":"[PATCH 10/12] gc: automatically write commit-graph files","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-10T17:34:14Z","receivedAt":"2018-05-10T17:34:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file is a very helpful feature for speeding up git\noperations. In order to make it more useful, write the commit-graph file\nby default during standard garbage collection operations.\n\nAdd a 'gc.commitGraph' config setting that triggers writing a\ncommit-graph file after any non-trivial 'git gc' command. Defaults to\nfalse while the commit-graph feature matures. We specifically do not\nwant to turn this on by default until the commit-graph feature is fully\nintegrated with history-modifying features like shallow clones.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 6 ++++++\n Documentation/git-gc.txt | 4 ++++\n builtin/gc.c | 8 ++++++++\n 3 files changed, 18 insertions(+)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 11f027194e..9a3abd87e7 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1553,6 +1553,12 @@ gc.autoDetach::\n \tMake `git gc --auto` return immediately and run in background\n \tif the system supports it. Default is true.\n \n+gc.commitGraph::\n+\tIf true, then gc will rewrite the commit-graph file after any\n+\tchange to the object database. If '--auto' is used, then the\n+\tcommit-graph will not be updated unless the threshold is met.\n+\tSee linkgit:git-commit-graph[1] for details.\n+\n gc.logExpiry::\n \tIf the file gc.log exists, then `git gc --auto` won't run\n \tunless that file is more than 'gc.logExpiry' old. Default is\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 571b5a7e3c..17dd654a59 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -119,6 +119,10 @@ The optional configuration variable `gc.packRefs` determines if\n it within all non-bare repos or it can be set to a boolean value.\n This defaults to true.\n \n+The optional configuration variable 'gc.commitGraph' determines if\n+'git gc' runs 'git commit-graph write'. This can be set to a boolean\n+value. This defaults to false.\n+\n The optional configuration variable `gc.aggressiveWindow` controls how\n much time is spent optimizing the delta compression of the objects in\n the repository when the --aggressive option is specified. The larger\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 77fa720bd0..8403445738 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -34,6 +34,7 @@ static int aggressive_depth = 50;\n static int aggressive_window = 250;\n static int gc_auto_threshold = 6700;\n static int gc_auto_pack_limit = 50;\n+static int gc_commit_graph = 0;\n static int detach_auto = 1;\n static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n@@ -46,6 +47,7 @@ static struct argv_array repack = ARGV_ARRAY_INIT;\n static struct argv_array prune = ARGV_ARRAY_INIT;\n static struct argv_array prune_worktrees = ARGV_ARRAY_INIT;\n static struct argv_array rerere = ARGV_ARRAY_INIT;\n+static struct argv_array commit_graph = ARGV_ARRAY_INIT;\n \n static struct tempfile *pidfile;\n static struct lock_file log_lock;\n@@ -121,6 +123,7 @@ static void gc_config(void)\n \tgit_config_get_int(\"gc.aggressivedepth\", &aggressive_depth);\n \tgit_config_get_int(\"gc.auto\", &gc_auto_threshold);\n \tgit_config_get_int(\"gc.autopacklimit\", &gc_auto_pack_limit);\n+\tgit_config_get_bool(\"gc.commitgraph\", &gc_commit_graph);\n \tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n \tgit_config_get_expiry(\"gc.pruneexpire\", &prune_expire);\n \tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n@@ -374,6 +377,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \targv_array_pushl(&prune, \"prune\", \"--expire\", NULL);\n \targv_array_pushl(&prune_worktrees, \"worktree\", \"prune\", \"--expire\", NULL);\n \targv_array_pushl(&rerere, \"rerere\", \"gc\", NULL);\n+\targv_array_pushl(&commit_graph, \"commit-graph\", \"write\", \"--reachable\", NULL);\n \n \t/* default expiry time, overwritten in gc_config */\n \tgc_config();\n@@ -480,6 +484,10 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tif (pack_garbage.nr > 0)\n \t\tclean_pack_garbage();\n \n+\tif (gc_commit_graph)\n+\t\tif (run_command_v_opt(commit_graph.argv, RUN_GIT_CMD))\n+\t\t\treturn error(FAILED_RUN, commit_graph.argv[0]);\n+\n \tif (auto_gc && too_many_loose_objects())\n \t\twarning(_(\"There are too many unreachable loose objects; \"\n \t\t\t\"run 'git prune' to remove them.\"));\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347274","messageId":"20180510173345.40577-6-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180510173345.40577-1-dstolee@microsoft.com","subject":"[PATCH 05/12] commit: force commit to parse from object database","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-10T17:34:08Z","receivedAt":"2018-05-10T17:34:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In anticipation of verifying commit-graph file contents against the\nobject database, create parse_commit_internal() to allow side-stepping\nthe commit-graph file and parse directly from the object database.\n\nMost consumers expect that if a commit exists in the commit-graph, then\nthe commit was loaded from the commit-graph so values such as generation\nnumbers are loaded. Hence, this method should not be called unless the\nintention is explicit in avoiding commits from the commit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 13 +++++++++----\n commit.h | 1 +\n 2 files changed, 10 insertions(+), 4 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex 1d28677dfb..7c92350373 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -392,7 +392,7 @@ int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long s\n \treturn 0;\n }\n \n-int parse_commit_gently(struct commit *item, int quiet_on_missing)\n+int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph)\n {\n \tenum object_type type;\n \tvoid *buffer;\n@@ -403,17 +403,17 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \t\treturn -1;\n \tif (item->object.parsed)\n \t\treturn 0;\n-\tif (parse_commit_in_graph(item))\n+\tif (use_commit_graph && parse_commit_in_graph(item))\n \t\treturn 0;\n \tbuffer = read_sha1_file(item->object.oid.hash, &type, &size);\n \tif (!buffer)\n \t\treturn quiet_on_missing ? -1 :\n \t\t\terror(\"Could not read %s\",\n-\t\t\t oid_to_hex(&item->object.oid));\n+\t\t\t\t\toid_to_hex(&item->object.oid));\n \tif (type != OBJ_COMMIT) {\n \t\tfree(buffer);\n \t\treturn error(\"Object %s not a commit\",\n-\t\t\t oid_to_hex(&item->object.oid));\n+\t\t\t\toid_to_hex(&item->object.oid));\n \t}\n \tret = parse_commit_buffer(item, buffer, size, 0);\n \tif (save_commit_buffer && !ret) {\n@@ -424,6 +424,11 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \treturn ret;\n }\n \n+int parse_commit_gently(struct commit *item, int quiet_on_missing)\n+{\n+\treturn parse_commit_internal(item, quiet_on_missing, 1);\n+}\n+\n void parse_commit_or_die(struct commit *item)\n {\n \tif (parse_commit(item))\ndiff --git a/commit.h b/commit.h\nindex b5afde1ae9..5fde74fcd7 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -73,6 +73,7 @@ struct commit *lookup_commit_reference_by_name(const char *name);\n struct commit *lookup_commit_or_die(const struct object_id *oid, const char *ref_name);\n \n int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph);\n+int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph);\n int parse_commit_gently(struct commit *item, int quiet_on_missing);\n static inline int parse_commit(struct commit *item)\n {\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347275","messageId":"20180510173345.40577-4-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180510173345.40577-1-dstolee@microsoft.com","subject":"[PATCH 03/12] commit-graph: parse commit from chosen graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-10T17:34:06Z","receivedAt":"2018-05-10T17:34:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Before verifying a commit-graph file against the object database, we\nneed to parse all commits from the given commit-graph file. Create\nparse_commit_in_graph_one() to target a given struct commit_graph.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 18 +++++++++++++++---\n 1 file changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex c3b8716c14..ce11af1d20 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -309,7 +309,7 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \t}\n }\n \n-int parse_commit_in_graph(struct commit *item)\n+int parse_commit_in_graph_one(struct commit_graph *g, struct commit *item)\n {\n \tuint32_t pos;\n \n@@ -317,9 +317,21 @@ int parse_commit_in_graph(struct commit *item)\n \t\treturn 0;\n \tif (item->object.parsed)\n \t\treturn 1;\n+\n+\tif (find_commit_in_graph(item, g, &pos))\n+\t\treturn fill_commit_in_graph(item, g, pos);\n+\n+\treturn 0;\n+}\n+\n+int parse_commit_in_graph(struct commit *item)\n+{\n+\tif (!core_commit_graph)\n+\t\treturn 0;\n+\n \tprepare_commit_graph();\n-\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n-\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n+\tif (commit_graph)\n+\t\treturn parse_commit_in_graph_one(commit_graph, item);\n \treturn 0;\n }\n \n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347280","messageId":"CAN0heSrg7W86UFKjCXv-NC1+tYr42B-aUksRhWv7Oug1PG8g8Q@mail.gmail.com","threadId":"48311","inReplyTo":"20180510173345.40577-2-dstolee@microsoft.com","subject":"Re: [PATCH 01/12] commit-graph: add 'verify' subcommand","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2018-05-10T18:15:40Z","receivedAt":"2018-05-10T18:15:45Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 10 May 2018 at 19:34, Derrick Stolee wrote:\n> In case the commit-graph file becomes corrupt, we need a way to\n> verify its contents match the object database. In the manner of\n\ns/verify its/verify that its/ might read better.\n\n> 'git fsck' we will implement a 'git commit-graph verify' subcommand\n> to report all issues with the file.\n>\n> Add the 'verify' subcommand to the 'commit-graph' builtin and its\n> documentation. The subcommand is currently a no-op except for\n> loading the commit-graph into memory, which may trigger run-time\n> errors that would be caught by normal use. Add a simple test that\n> ensures the command returns a zero error code.\n>\n> If no commit-graph file exists, this is an acceptable state. Do\n> not report any errors.\n\nThis all makes sense to me.\n\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n\n> +'verify'::\n> +\n> +Read the commit-graph file and verify its contents against the object\n> +database. Used to verify for corrupted data.\n\ns/verify for/check for/?\n\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -7,11 +7,17 @@\n>\n> static char const * const builtin_commit_graph_usage[] = {\n> N_(\"git commit-graph [--object-dir ]\"),\n> + N_(\"git commit-graph verify [--object-dir ]\"),\n> N_(\"git commit-graph read [--object-dir ]\"),\n> N_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n\nMinor nit: In the man-page, you added verify after read, which makes\nmore sense I think (r < v < w).\n\n(I also note that the man-page synopsis doesn't give the no-subcommand\nusage.)\n\n> +static int graph_verify(int argc, const char **argv)\n> +{\n> + struct commit_graph *graph = 0;\n> + char *graph_name;\n> +\n> + static struct option builtin_commit_graph_verify_options[] = {\n> + OPT_STRING(0, \"object-dir\", &opts.obj_dir,\n> + N_(\"dir\"),\n> + N_(\"The object directory to store the graph\")),\n> + OPT_END(),\n> + };\n> +\n> + argc = parse_options(argc, argv, NULL,\n> + builtin_commit_graph_verify_options,\n> + builtin_commit_graph_verify_usage, 0);\n> +\n> + if (!opts.obj_dir)\n> + opts.obj_dir = get_object_directory();\n> +\n> + graph_name = get_commit_graph_filename(opts.obj_dir);\n> + graph = load_commit_graph_one(graph_name);\n> +\n> + if (!graph)\n> + return 0;\n> + FREE_AND_NULL(graph_name);\n\nMaybe the FREE_AND_NULL could go immediately after the call to\n`load_commit_graph_one()`. It makes it more obvious that you're done\nwith the name, and -- perhaps more importantly -- means that throwing a\nleak-checker at this won't complain if we take the early return.\n\n> +\n> + return verify_commit_graph(graph);\n\nA leak-checker would still complain about leaking `graph`. I think it\nwould be ok to just UNLEAK it before calling `verify_commit_graph()`.\nThis is IMHO close enough to returning from `cmd_commit_graph()` to make\nUNLEAK an acceptable, or even the correct, solution.\n\nI realize that `graph_read()` is doing something similar to this patch\nalready, so what you have here is certainly the most consistent code.\n\nMartin\n"},{"id":"347281","messageId":"CAN0heSrNdHGyrZ2qF6J2daD0VdjXDhErs1nVzO+KT5GvOPfAPw@mail.gmail.com","threadId":"48311","inReplyTo":"20180510173345.40577-3-dstolee@microsoft.com","subject":"Re: [PATCH 02/12] commit-graph: verify file header information","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2018-05-10T18:21:47Z","receivedAt":"2018-05-10T18:21:51Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 10 May 2018 at 19:34, Derrick Stolee wrote:\n> During a run of 'git commit-graph verify', list the issues with the\n> header information in the commit-graph file. Some of this information\n> is inferred from the loaded 'struct commit_graph'. Some header\n> information is checked as part of load_commit_graph_one().\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 23 ++++++++++++++++++++++-\n> 1 file changed, 22 insertions(+), 1 deletion(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index b25aaed128..c3b8716c14 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -818,7 +818,28 @@ void write_commit_graph(const char *obj_dir,\n> oids.nr = 0;\n> }\n>\n> +static int verify_commit_graph_error;\n> +#define graph_report(...) \\\n> + do {\\\n> + verify_commit_graph_error = 1;\\\n> + printf(__VA_ARGS__);\\\n> + } while (0);\n> +\n\nIt seems to me that other users of __VA_ARGS__ are protected with a\ncheck for HAVE_VARIADIC_MACROS and provide an alternative\nnon-__VA_ARGS__-implementation. Or maybe I've missed something in my\ngrepping and we are actually (slowly) moving towards assuming\n__VA_ARGS__ is always available?\n\n> int verify_commit_graph(struct commit_graph *g)\n> {\n> - return !g;\n> + if (!g) {\n> + graph_report(_(\"no commit-graph file loaded\"));\n> + return 1;\n> + }\n> +\n> + verify_commit_graph_error = 0;\n> +\n> + if (!g->chunk_oid_fanout)\n> + graph_report(_(\"commit-graph is missing the OID Fanout chunk\"));\n> + if (!g->chunk_oid_lookup)\n> + graph_report(_(\"commit-graph is missing the OID Lookup chunk\"));\n> + if (!g->chunk_commit_data)\n> + graph_report(_(\"commit-graph is missing the Commit Data chunk\"));\n> +\n> + return verify_commit_graph_error;\n\nIf you can't rely on __VA_ARGS__, maybe bite the bullet and introduce\nbraces... The expanded code wouldn't be too horrible, albeit a bit\nrepetitive.\n\nMartin\n"},{"id":"347284","messageId":"CAN0heSoaTdkb5xcrEE4dY7aUpcTGCROyUbZ-HUUKFBHJ9zTGew@mail.gmail.com","threadId":"48311","inReplyTo":"20180510173345.40577-5-dstolee@microsoft.com","subject":"Re: [PATCH 04/12] commit-graph: verify fanout and lookup table","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2018-05-10T18:29:13Z","receivedAt":"2018-05-10T18:29:22Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 10 May 2018 at 19:34, Derrick Stolee wrote:\n> While running 'git commit-graph verify', verify that the object IDs\n> are listed in lexicographic order and that the fanout table correctly\n> navigates into that list of object IDs.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 33 +++++++++++++++++++++++++++++++++\n> 1 file changed, 33 insertions(+)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index ce11af1d20..b4c146c423 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -839,6 +839,9 @@ static int verify_commit_graph_error;\n>\n> int verify_commit_graph(struct commit_graph *g)\n> {\n> + uint32_t i, cur_fanout_pos = 0;\n> + struct object_id prev_oid, cur_oid;\n> +\n> if (!g) {\n> graph_report(_(\"no commit-graph file loaded\"));\n> return 1;\n> @@ -853,5 +856,35 @@ int verify_commit_graph(struct commit_graph *g)\n> if (!g->chunk_commit_data)\n> graph_report(_(\"commit-graph is missing the Commit Data chunk\"));\n>\n> + for (i = 0; i < g->num_commits; i++) {\n> + hashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n> +\n> + if (i > 0 && oidcmp(&prev_oid, &cur_oid) >= 0)\n> + graph_report(_(\"commit-graph has incorrect oid order: %s then %s\"),\n\nMinor: I think our style would prefer s/i > 0/i/.\n\nI suppose the second check should be s/>=/>/, but it's not like it\nshould matter. ;-)\n\nI wonder if this is a message that would virtually never make sense to a\nuser, but more to a developer. Leave it untranslated to make sure any\nbug reports to the list are readable to us?\n\n> +\n> + oid_to_hex(&prev_oid),\n> + oid_to_hex(&cur_oid));\n\nHmm, these two lines do not actually achieve anything?\n\n> + oidcpy(&prev_oid, &cur_oid);\n> +\n> + while (cur_oid.hash[0] > cur_fanout_pos) {\n> + uint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n> + if (i != fanout_value)\n> + graph_report(_(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\"),\n> + cur_fanout_pos, fanout_value, i);\n\nSame though re `_()`, even more so because of the more technical\nnotation.\n\n> +\n> + cur_fanout_pos++;\n> + }\n> + }\n> +\n> + while (cur_fanout_pos < 256) {\n> + uint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n> +\n> + if (g->num_commits != fanout_value)\n> + graph_report(_(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\"),\n> + cur_fanout_pos, fanout_value, i);\n\nSame here. Or maybe these should just give a translated user-readable\nbasic idea of what is wrong and skip the details?\n\nMartin\n"},{"id":"347290","messageId":"CAN0heSo-1+BvE0SiZMKT1zHZ4o+-mK0W9S+xxDA4Pyw3iTvpDQ@mail.gmail.com","threadId":"48311","inReplyTo":"20180510173345.40577-1-dstolee@microsoft.com","subject":"Re: [PATCH 00/12] Integrate commit-graph into fsck, gc, and fetch","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2018-05-10T19:05:27Z","receivedAt":"2018-05-10T19:05:31Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 10 May 2018 at 19:34, Derrick Stolee wrote:\n\n> Commits 01-07 focus on the 'git commit-graph verify' subcommand. These\n> are ready for full, rigorous review.\n\nI don't know about \"full\" and \"rigorous\", but I tried to offer my thoughts.\n\nA lingering feeling I have is that users could possibly benefit from\nseeing \"the commit-graph has a bad foo\" a bit more than just \"the\ncommit-graph is bad\". But adding \"the bar is baz, should have been\nfrotz\" might not bring that much. Maybe you could keep the translatable\nstring somewhat simple, then, if the more technical data could be useful\nto Git developers, dump it in a non-translated format. (I guess it could\nbe hidden behind a debug switch, but let's take one step at a time..)\nThis is nothing I feel strongly about.\n\n> t/t5318-commit-graph.sh | 25 +++++\n\nI wonder about tests. Some tests seem to use `dd` to corrupt files and\ncheck that it gets caught. Doing this in a a hash-agnostic way could be\ntricky, but maybe not impossible. I guess we could do something\nprobabilistic, like \"set the first two bytes of the very last OID to\nzero -- surely all OIDs can't start with 16 zero bits\". Hmm, that might\nstill require knowing the size of the OIDs...\n\nI hope to find time to do some more hands-on testing of this to see that\nerrors actually do get caught.\n\nI saw you redirect stdout to a file \"output\", and anticipated later\ncommits to actually look into it. I never saw that though. (I did not\napply the patches, so I could have missed something.)\n\nMartin\n"},{"id":"347293","messageId":"87mux72uds.fsf@evledraar.gmail.com","threadId":"48311","inReplyTo":"20180510173345.40577-1-dstolee@microsoft.com","subject":"Re: [PATCH 00/12] Integrate commit-graph into fsck, gc, and fetch","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-05-10T19:17:03Z","receivedAt":"2018-05-10T19:17:13Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, May 10 2018, Derrick Stolee wrote:\n\n> The behavior in this patch series does the following:\n>\n> 1. Near the end of 'git gc', run 'git commit-graph write'. The location\n> of this code assumes that a 'git gc --auto' has not terminated early\n> due to not meeting the auto threshold.\n>\n> 2. At the end of 'git fetch', run 'git commit-graph write'. This means\n> that every reachable commit will be in the commit-graph after a\n> a successful fetch, which seems a reasonable frequency. Then, the\n> only times we would be missing a reachable commit is after creating\n> one locally. There is a problem with the current patch, though: every\n> 'git fetch' call runs 'git commit-graph write', even if there were no\n> ref updates or objects downloaded. Is there a simple way to detect if\n> the fetch was non-trivial?\n>\n> One obvious problem with this approach: if we compute this during 'gc'\n> AND 'fetch', there will be times where a 'fetch' calls 'gc' and triggers\n> two commit-graph writes. If I were to abandon one of these patches, it\n> would be the 'fetch' integration. A 'git gc' really wants to delete all\n> references to unreachable commits, and without updating the commit-graph\n> we may still have commit data in the commit-graph file that is not in\n> the object database. In fact, deleting commits from the object database\n> but not from the commit-graph will cause 'git commit-graph verify' to\n> fail!\n>\n> I welcome discussion on these ideas, as we are venturing out of the\n> \"pure data structure\" world and into the \"user experience\" world. I am\n> less confident in my skills in this world, but the feature is worthless\n> if it does not improve the user experience.\n\nI really like #1 here, but I wonder why #2 is necessary.\n\nI.e. is it critical for the performance of the commit graph feature that\nit be kept really up-to-date, moreso than other things that rely on gc\n--auto (e.g. the optional bitmap index)?\n\nEven if that's the case, I think something that does this via gc --auto\nis a much better option. I.e. now we have gc.auto & gc.autoPackLimit, if\nthe answer to my question above is \"yes\" this could also be accomplished\nby introducing a new graph-specific gc.* setting, and --auto would just\nupdate the graph more often, but leave the rest.\n"},{"id":"347295","messageId":"CAGZ79kY1m_6AdUoRx1L2=HzX7hG_uUUcYnLsb=VNdxPcvnpOEg@mail.gmail.com","threadId":"48311","inReplyTo":"CAN0heSo-1+BvE0SiZMKT1zHZ4o+-mK0W9S+xxDA4Pyw3iTvpDQ@mail.gmail.com","subject":"Re: [PATCH 00/12] Integrate commit-graph into fsck, gc, and fetch","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-10T19:22:12Z","receivedAt":"2018-05-10T19:22:16Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, May 10, 2018 at 12:05 PM, Martin Ågren wrote:\n> On 10 May 2018 at 19:34, Derrick Stolee wrote:\n>\n>> Commits 01-07 focus on the 'git commit-graph verify' subcommand. These\n>> are ready for full, rigorous review.\n>\n> I don't know about \"full\" and \"rigorous\", but I tried to offer my thoughts.\n>\n> A lingering feeling I have is that users could possibly benefit from\n> seeing \"the commit-graph has a bad foo\" a bit more than just \"the\n> commit-graph is bad\". But adding \"the bar is baz, should have been\n> frotz\" might not bring that much. Maybe you could keep the translatable\n> string somewhat simple, then, if the more technical data could be useful\n> to Git developers, dump it in a non-translated format. (I guess it could\n> be hidden behind a debug switch, but let's take one step at a time..)\n> This is nothing I feel strongly about.\n>\n>> t/t5318-commit-graph.sh | 25 +++++\n>\n> I wonder about tests. Some tests seem to use `dd` to corrupt files and\n> check that it gets caught. Doing this in a a hash-agnostic way could be\n> tricky, but maybe not impossible. I guess we could do something\n> probabilistic, like \"set the first two bytes of the very last OID to\n> zero -- surely all OIDs can't start with 16 zero bits\". Hmm, that might\n> still require knowing the size of the OIDs...\n>\n> I hope to find time to do some more hands-on testing of this to see that\n> errors actually do get caught.\n\nGiven that the commit graph is secondary data, the users work around\nto quickly get back to a well working repository is most likely to remove\nthe file and regenerate it.\nAs a developer who wants to fix the bug, a stacktrace/datadump and the\nhistory of git commands might be most valuable, but I agree we should\nhide that behind a debug flag.\n\nPackfiles and loose objects are primary data, which means that those\nneed a more advanced way to diagnose and repair them, so I would imagine\nthe commit graph fsck is closer to bitmaps fsck, which I would have suspected\nto be found in t5310, but a quick read doesn't reveal many tests that are\nchecking for integrity. So I guess the test coverage here is ok, (although we\nshould always ask for more)\n\nThanks,\nStefan\n"},{"id":"347362","messageId":"62c7d8c3-87e3-f8e3-a83a-fb53f956f89d@gmail.com","threadId":"48311","inReplyTo":"CAN0heSoaTdkb5xcrEE4dY7aUpcTGCROyUbZ-HUUKFBHJ9zTGew@mail.gmail.com","subject":"Re: [PATCH 04/12] commit-graph: verify fanout and lookup table","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-11T15:17:26Z","receivedAt":"2018-05-11T15:17:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/10/2018 2:29 PM, Martin Ågren wrote:\n> On 10 May 2018 at 19:34, Derrick Stolee wrote:\n>> While running 'git commit-graph verify', verify that the object IDs\n>> are listed in lexicographic order and that the fanout table correctly\n>> navigates into that list of object IDs.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> commit-graph.c | 33 +++++++++++++++++++++++++++++++++\n>> 1 file changed, 33 insertions(+)\n>>\n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index ce11af1d20..b4c146c423 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> @@ -839,6 +839,9 @@ static int verify_commit_graph_error;\n>>\n>> int verify_commit_graph(struct commit_graph *g)\n>> {\n>> + uint32_t i, cur_fanout_pos = 0;\n>> + struct object_id prev_oid, cur_oid;\n>> +\n>> if (!g) {\n>> graph_report(_(\"no commit-graph file loaded\"));\n>> return 1;\n>> @@ -853,5 +856,35 @@ int verify_commit_graph(struct commit_graph *g)\n>> if (!g->chunk_commit_data)\n>> graph_report(_(\"commit-graph is missing the Commit Data chunk\"));\n>>\n>> + for (i = 0; i < g->num_commits; i++) {\n>> + hashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n>> +\n>> + if (i > 0 && oidcmp(&prev_oid, &cur_oid) >= 0)\n>> + graph_report(_(\"commit-graph has incorrect oid order: %s then %s\"),\n> Minor: I think our style would prefer s/i > 0/i/.\n>\n> I suppose the second check should be s/>=/>/, but it's not like it\n> should matter. ;-)\n\nIt shouldn't, but a duplicate commit is still an incorrect oid order.\n\n> I wonder if this is a message that would virtually never make sense to a\n> user, but more to a developer. Leave it untranslated to make sure any\n> bug reports to the list are readable to us?\n\nYeah, I'll make all of the errors untranslated since they are for \ndeveloper debugging, not end-user information.\n\n>> +\n>> + oid_to_hex(&prev_oid),\n>> + oid_to_hex(&cur_oid));\n> Hmm, these two lines do not actually achieve anything?\n\nIt's a formatting bug: They complete the statement starting with \n'graph_report(' above.\n\n>\n>> + oidcpy(&prev_oid, &cur_oid);\n>> +\n>> + while (cur_oid.hash[0] > cur_fanout_pos) {\n>> + uint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n>> + if (i != fanout_value)\n>> + graph_report(_(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\"),\n>> + cur_fanout_pos, fanout_value, i);\n> Same though re `_()`, even more so because of the more technical\n> notation.\n>\n>> +\n>> + cur_fanout_pos++;\n>> + }\n>> + }\n>> +\n>> + while (cur_fanout_pos < 256) {\n>> + uint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n>> +\n>> + if (g->num_commits != fanout_value)\n>> + graph_report(_(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\"),\n>> + cur_fanout_pos, fanout_value, i);\n> Same here. Or maybe these should just give a translated user-readable\n> basic idea of what is wrong and skip the details?\n\nAs someone who is responsible for probably inserting these problems, I \nthink the details are important.\n\nThanks,\n-Stolee\n"},{"id":"347366","messageId":"bfaa96f8-2b9d-5ec6-0d30-7c1018146335@gmail.com","threadId":"48311","inReplyTo":"87mux72uds.fsf@evledraar.gmail.com","subject":"Re: [PATCH 00/12] Integrate commit-graph into fsck, gc, and fetch","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-11T17:23:21Z","receivedAt":"2018-05-11T17:23:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/10/2018 3:17 PM, Ævar Arnfjörð Bjarmason wrote:\n> On Thu, May 10 2018, Derrick Stolee wrote:\n>\n>> The behavior in this patch series does the following:\n>>\n>> 1. Near the end of 'git gc', run 'git commit-graph write'. The location\n>> of this code assumes that a 'git gc --auto' has not terminated early\n>> due to not meeting the auto threshold.\n>>\n>> 2. At the end of 'git fetch', run 'git commit-graph write'. This means\n>> that every reachable commit will be in the commit-graph after a\n>> a successful fetch, which seems a reasonable frequency. Then, the\n>> only times we would be missing a reachable commit is after creating\n>> one locally. There is a problem with the current patch, though: every\n>> 'git fetch' call runs 'git commit-graph write', even if there were no\n>> ref updates or objects downloaded. Is there a simple way to detect if\n>> the fetch was non-trivial?\n>>\n>> One obvious problem with this approach: if we compute this during 'gc'\n>> AND 'fetch', there will be times where a 'fetch' calls 'gc' and triggers\n>> two commit-graph writes. If I were to abandon one of these patches, it\n>> would be the 'fetch' integration. A 'git gc' really wants to delete all\n>> references to unreachable commits, and without updating the commit-graph\n>> we may still have commit data in the commit-graph file that is not in\n>> the object database. In fact, deleting commits from the object database\n>> but not from the commit-graph will cause 'git commit-graph verify' to\n>> fail!\n>>\n>> I welcome discussion on these ideas, as we are venturing out of the\n>> \"pure data structure\" world and into the \"user experience\" world. I am\n>> less confident in my skills in this world, but the feature is worthless\n>> if it does not improve the user experience.\n> I really like #1 here, but I wonder why #2 is necessary.\n>\n> I.e. is it critical for the performance of the commit graph feature that\n> it be kept really up-to-date, moreso than other things that rely on gc\n> --auto (e.g. the optional bitmap index)?\n\nIt is not critical. The feature has been designed to have recent commits \nnot in the file. For simplicity, it is probably best to limit ourselves \nto writing after a non-trivial 'gc'.\n\n> Even if that's the case, I think something that does this via gc --auto\n> is a much better option. I.e. now we have gc.auto & gc.autoPackLimit, if\n> the answer to my question above is \"yes\" this could also be accomplished\n> by introducing a new graph-specific gc.* setting, and --auto would just\n> update the graph more often, but leave the rest.\n\nThis is an excellent idea for a follow-up series, if we find we want the \ncommit-graph written more frequently. For now, I'm satisfied with one \nplace where it is automatically computed.\n\nI'll drop the fetch integration in my v2 series.\n\nThanks,\n-Stolee\n"},{"id":"347367","messageId":"eb3de182-c939-022f-92ed-3449e220af59@gmail.com","threadId":"48311","inReplyTo":"CAGZ79kY1m_6AdUoRx1L2=HzX7hG_uUUcYnLsb=VNdxPcvnpOEg@mail.gmail.com","subject":"Re: [PATCH 00/12] Integrate commit-graph into fsck, gc, and fetch","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-11T17:23:30Z","receivedAt":"2018-05-11T17:23:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/10/2018 3:22 PM, Stefan Beller wrote:\n> On Thu, May 10, 2018 at 12:05 PM, Martin Ågren wrote:\n>> On 10 May 2018 at 19:34, Derrick Stolee wrote:\n>>\n>>> Commits 01-07 focus on the 'git commit-graph verify' subcommand. These\n>>> are ready for full, rigorous review.\n>> I don't know about \"full\" and \"rigorous\", but I tried to offer my thoughts.\n>>\n>> A lingering feeling I have is that users could possibly benefit from\n>> seeing \"the commit-graph has a bad foo\" a bit more than just \"the\n>> commit-graph is bad\". But adding \"the bar is baz, should have been\n>> frotz\" might not bring that much. Maybe you could keep the translatable\n>> string somewhat simple, then, if the more technical data could be useful\n>> to Git developers, dump it in a non-translated format. (I guess it could\n>> be hidden behind a debug switch, but let's take one step at a time..)\n>> This is nothing I feel strongly about.\n>>\n>>> t/t5318-commit-graph.sh | 25 +++++\n>> I wonder about tests. Some tests seem to use `dd` to corrupt files and\n>> check that it gets caught. Doing this in a a hash-agnostic way could be\n>> tricky, but maybe not impossible. I guess we could do something\n>> probabilistic, like \"set the first two bytes of the very last OID to\n>> zero -- surely all OIDs can't start with 16 zero bits\". Hmm, that might\n>> still require knowing the size of the OIDs...\n>>\n>> I hope to find time to do some more hands-on testing of this to see that\n>> errors actually do get caught.\n> Given that the commit graph is secondary data, the users work around\n> to quickly get back to a well working repository is most likely to remove\n> the file and regenerate it.\n> As a developer who wants to fix the bug, a stacktrace/datadump and the\n> history of git commands might be most valuable, but I agree we should\n> hide that behind a debug flag.\n>\n> Packfiles and loose objects are primary data, which means that those\n> need a more advanced way to diagnose and repair them, so I would imagine\n> the commit graph fsck is closer to bitmaps fsck, which I would have suspected\n> to be found in t5310, but a quick read doesn't reveal many tests that are\n> checking for integrity. So I guess the test coverage here is ok, (although we\n> should always ask for more)\n\nMy main goal is to help developers figure out what is wrong with a file, \nand then we can use other diagnostic tools to discover how it got into \nthat state.\n\nMartin's initial test cases are wonderful. I've adapted them to test the \nother conditions in the verify_commit_graph() method and caught some \ninteresting behavior in the process. I'm preparing v2 so we can \ninvestigate the direction of the tests.\n\nThanks,\n-Stolee\n"},{"id":"347368","messageId":"CAN0heSqr+UZrdRW4MqEGK927iwg+6JNrWxebF8UJYcVt9V2aqw@mail.gmail.com","threadId":"48311","inReplyTo":"eb3de182-c939-022f-92ed-3449e220af59@gmail.com","subject":"Re: [PATCH 00/12] Integrate commit-graph into fsck, gc, and fetch","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2018-05-11T17:30:06Z","receivedAt":"2018-05-11T17:30:11Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 11 May 2018 at 19:23, Derrick Stolee wrote:\n\n> Martin's initial test cases are wonderful. I've adapted them to test the\n> other conditions in the verify_commit_graph() method and caught some\n> interesting behavior in the process. I'm preparing v2 so we can investigate\n> the direction of the tests.\n\nCool, I'm glad you found them useful. One thought I had was that you\ncould possibly write the tests such that you introduce errors from the\nback of the file. That might enable you to do less of the \"backup\ncommit-graph file and restore it\"-dance. Just a thought.\n\nMartin\n"},{"id":"347381","messageId":"20180511211504.79877-1-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180510173345.40577-1-dstolee@microsoft.com","subject":"[PATCH v2 00/12] Integrate commit-graph into fsck and gc","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-11T21:15:14Z","receivedAt":"2018-05-11T21:15:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"I'm sending this v2 re-roll rather quickly after the previous version\nbecause Martin provided a framework to add tests to the 'verify'\nsubcommand. I took that framework and added tests for the other checks\nof the commit-graph data. This also found some interesting things about\nthe command:\n\n1. There were some segfaults because we were not checking for bad data\n carefully enough.\n\n2. To avoid segfaults, we will now terminate the check early if we find\n problems earlier in the file, such as in the header, or OID lookup.\n\n3. We were not writing newlines between reports. This now happens by\n default in graph_report().\n\nThe integration into 'fetch' is dropped (thanks Ævar!).\n\nDerrick Stolee (12):\n commit-graph: add 'verify' subcommand\n commit-graph: verify file header information\n commit-graph: test that 'verify' finds corruption\n commit-graph: parse commit from chosen graph\n commit-graph: verify fanout and lookup table\n commit: force commit to parse from object database\n commit-graph: load a root tree from specific graph\n commit-graph: verify commit contents against odb\n fsck: verify commit-graph\n commit-graph: add '--reachable' option\n gc: automatically write commit-graph files\n commit-graph: update design document\n\n Documentation/config.txt | 6 +\n Documentation/git-commit-graph.txt | 14 ++-\n Documentation/git-fsck.txt | 3 +\n Documentation/git-gc.txt | 4 +\n Documentation/technical/commit-graph.txt | 22 ----\n builtin/commit-graph.c | 81 ++++++++++++-\n builtin/fsck.c | 21 ++++\n builtin/gc.c | 8 ++\n commit-graph.c | 199 ++++++++++++++++++++++++++++++-\n commit-graph.h | 2 +\n commit.c | 13 +-\n commit.h | 1 +\n t/t5318-commit-graph.sh | 173 +++++++++++++++++++++++++++\n 13 files changed, 509 insertions(+), 38 deletions(-)\n\n\nbase-commit: 34fdd433396ee0e3ef4de02eb2189f8226eafe4e\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347382","messageId":"20180511211504.79877-3-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180511211504.79877-1-dstolee@microsoft.com","subject":"[PATCH v2 02/12] commit-graph: verify file header information","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-11T21:15:16Z","receivedAt":"2018-05-11T21:15:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"During a run of 'git commit-graph verify', list the issues with the\nheader information in the commit-graph file. Some of this information\nis inferred from the loaded 'struct commit_graph'. Some header\ninformation is checked as part of load_commit_graph_one().\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 32 +++++++++++++++++++++++++++++++-\n 1 file changed, 31 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex b25aaed128..d2db20e49a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -818,7 +818,37 @@ void write_commit_graph(const char *obj_dir,\n \toids.nr = 0;\n }\n \n+static int verify_commit_graph_error;\n+\n+static void graph_report(const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tverify_commit_graph_error = 1;\n+\n+\tva_start(ap, fmt);\n+\tstrbuf_vaddf(&sb, fmt, ap);\n+\n+\tfprintf(stderr, \"%s\\n\", sb.buf);\n+\tstrbuf_release(&sb);\n+\tva_end(ap);\n+}\n+\n int verify_commit_graph(struct commit_graph *g)\n {\n-\treturn !g;\n+\tif (!g) {\n+\t\tgraph_report(\"no commit-graph file loaded\");\n+\t\treturn 1;\n+\t}\n+\n+\tverify_commit_graph_error = 0;\n+\n+\tif (!g->chunk_oid_fanout)\n+\t\tgraph_report(\"commit-graph is missing the OID Fanout chunk\");\n+\tif (!g->chunk_oid_lookup)\n+\t\tgraph_report(\"commit-graph is missing the OID Lookup chunk\");\n+\tif (!g->chunk_commit_data)\n+\t\tgraph_report(\"commit-graph is missing the Commit Data chunk\");\n+\n+\treturn verify_commit_graph_error;\n }\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347383","messageId":"20180511211504.79877-2-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180511211504.79877-1-dstolee@microsoft.com","subject":"[PATCH v2 01/12] commit-graph: add 'verify' subcommand","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-11T21:15:15Z","receivedAt":"2018-05-11T21:15:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"If the commit-graph file becomes corrupt, we need a way to verify\nthat its contents match the object database. In the manner of\n'git fsck' we will implement a 'git commit-graph verify' subcommand\nto report all issues with the file.\n\nAdd the 'verify' subcommand to the 'commit-graph' builtin and its\ndocumentation. The subcommand is currently a no-op except for\nloading the commit-graph into memory, which may trigger run-time\nerrors that would be caught by normal use. Add a simple test that\nensures the command returns a zero error code.\n\nIf no commit-graph file exists, this is an acceptable state. Do\nnot report any errors.\n\nDuring review, we noticed that a FREE_AND_NULL(graph_name) was\nplaced after a possible 'return', and this pattern was also in\ngraph_read(). Fix that case, too.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 6 ++++++\n builtin/commit-graph.c | 40 +++++++++++++++++++++++++++++++++++++-\n commit-graph.c | 5 +++++\n commit-graph.h | 2 ++\n t/t5318-commit-graph.sh | 10 ++++++++++\n 5 files changed, 62 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 4c97b555cc..a222cfab08 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git commit-graph read' [--object-dir ]\n+'git commit-graph verify' [--object-dir ]\n 'git commit-graph write' [--object-dir ]\n \n \n@@ -52,6 +53,11 @@ existing commit-graph file.\n Read a graph file given by the commit-graph file and output basic\n details about the graph file. Used for debugging purposes.\n \n+'verify'::\n+\n+Read the commit-graph file and verify its contents against the object\n+database. Used to check for corrupted data.\n+\n \n EXAMPLES\n --------\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 37420ae0fd..af3101291f 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -8,10 +8,16 @@\n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n+\tN_(\"git commit-graph verify [--object-dir ]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n+static const char * const builtin_commit_graph_verify_usage[] = {\n+\tN_(\"git commit-graph verify [--object-dir ]\"),\n+\tNULL\n+};\n+\n static const char * const builtin_commit_graph_read_usage[] = {\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tNULL\n@@ -29,6 +35,36 @@ static struct opts_commit_graph {\n \tint append;\n } opts;\n \n+\n+static int graph_verify(int argc, const char **argv)\n+{\n+\tstruct commit_graph *graph = 0;\n+\tchar *graph_name;\n+\n+\tstatic struct option builtin_commit_graph_verify_options[] = {\n+\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n+\t\t\t N_(\"dir\"),\n+\t\t\t N_(\"The object directory to store the graph\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t builtin_commit_graph_verify_options,\n+\t\t\t builtin_commit_graph_verify_usage, 0);\n+\n+\tif (!opts.obj_dir)\n+\t\topts.obj_dir = get_object_directory();\n+\n+\tgraph_name = get_commit_graph_filename(opts.obj_dir);\n+\tgraph = load_commit_graph_one(graph_name);\n+\tFREE_AND_NULL(graph_name);\n+\n+\tif (!graph)\n+\t\treturn 0;\n+\n+\treturn verify_commit_graph(graph);\n+}\n+\n static int graph_read(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n@@ -50,10 +86,10 @@ static int graph_read(int argc, const char **argv)\n \n \tgraph_name = get_commit_graph_filename(opts.obj_dir);\n \tgraph = load_commit_graph_one(graph_name);\n+\tFREE_AND_NULL(graph_name);\n \n \tif (!graph)\n \t\tdie(\"graph file %s does not exist\", graph_name);\n-\tFREE_AND_NULL(graph_name);\n \n \tprintf(\"header: %08x %d %d %d %d\\n\",\n \t\tntohl(*(uint32_t*)graph->data),\n@@ -160,6 +196,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n \tif (argc > 0) {\n+\t\tif (!strcmp(argv[0], \"verify\"))\n+\t\t\treturn graph_verify(argc, argv);\n \t\tif (!strcmp(argv[0], \"read\"))\n \t\t\treturn graph_read(argc, argv);\n \t\tif (!strcmp(argv[0], \"write\"))\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a8c337dd77..b25aaed128 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -817,3 +817,8 @@ void write_commit_graph(const char *obj_dir,\n \toids.alloc = 0;\n \toids.nr = 0;\n }\n+\n+int verify_commit_graph(struct commit_graph *g)\n+{\n+\treturn !g;\n+}\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 96cccb10f3..71a39c5a57 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -53,4 +53,6 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tint nr_commits,\n \t\t\tint append);\n \n+int verify_commit_graph(struct commit_graph *g);\n+\n #endif\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 77d85aefe7..6ca451dfd2 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -11,6 +11,11 @@ test_expect_success 'setup full repo' '\n \tobjdir=\".git/objects\"\n '\n \n+test_expect_success 'verify graph with no graph file' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph verify\n+'\n+\n test_expect_success 'write graph with no packs' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write --object-dir . &&\n@@ -230,4 +235,9 @@ test_expect_success 'perform fast-forward merge in full repo' '\n \ttest_cmp expect output\n '\n \n+test_expect_success 'git commit-graph verify' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph verify >output\n+'\n+\n test_done\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347384","messageId":"20180511211504.79877-4-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180511211504.79877-1-dstolee@microsoft.com","subject":"[PATCH v2 03/12] commit-graph: test that 'verify' finds corruption","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-11T21:15:17Z","receivedAt":"2018-05-11T21:15:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Add test cases to t5318-commit-graph.sh that corrupt the commit-graph\nfile and check that the 'git commit-graph verify' command fails. These\ntests verify the header and chunk information is checked carefully.\n\nHelped-by: Martin Ågren \nSigned-off-by: Derrick Stolee \n---\n t/t5318-commit-graph.sh | 53 +++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 53 insertions(+)\n\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 6ca451dfd2..0cb88232fa 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -240,4 +240,57 @@ test_expect_success 'git commit-graph verify' '\n \tgit commit-graph verify >output\n '\n \n+# usage: corrupt_data []\n+corrupt_data() {\n+\tfile=$1\n+\tpos=$2\n+\tdata=\"${3:-\\0}\"\n+\tprintf \"$data\" | dd of=\"$file\" bs=1 seek=\"$pos\" conv=notrunc\n+}\n+\n+test_expect_success 'detect bad signature' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tcp $objdir/info/commit-graph commit-graph-backup &&\n+\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n+\tcorrupt_data $objdir/info/commit-graph 0 \"\\0\" &&\n+\ttest_must_fail git commit-graph verify 2>err &&\n+\tgrep -v \"^\\+\" err > verify-errors &&\n+\ttest_line_count = 1 verify-errors &&\n+\tgrep \"graph signature\" verify-errors\n+'\n+\n+test_expect_success 'detect bad version number' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tcp $objdir/info/commit-graph commit-graph-backup &&\n+\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n+\tcorrupt_data $objdir/info/commit-graph 4 \"\\02\" &&\n+\ttest_must_fail git commit-graph verify 2>err &&\n+\tgrep -v \"^\\+\" err > verify-errors &&\n+\ttest_line_count = 1 verify-errors &&\n+\tgrep \"graph version\" verify-errors\n+'\n+\n+test_expect_success 'detect bad hash version' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tcp $objdir/info/commit-graph commit-graph-backup &&\n+\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n+\tcorrupt_data $objdir/info/commit-graph 5 \"\\02\" &&\n+\ttest_must_fail git commit-graph verify 2>err &&\n+\tgrep -v \"^\\+\" err > verify-errors &&\n+\ttest_line_count = 1 verify-errors &&\n+\tgrep \"hash version\" verify-errors\n+'\n+\n+test_expect_success 'detect too small chunk-count' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tcp $objdir/info/commit-graph commit-graph-backup &&\n+\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n+\tcorrupt_data $objdir/info/commit-graph 6 \"\\01\" &&\n+\ttest_must_fail git commit-graph verify 2>err &&\n+\tgrep -v \"^\\+\" err > verify-errors &&\n+\ttest_line_count = 2 verify-errors &&\n+\tgrep \"missing the OID Lookup chunk\" verify-errors &&\n+\tgrep \"missing the Commit Data chunk\" verify-errors\n+'\n+\n test_done\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347385","messageId":"20180511211504.79877-5-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180511211504.79877-1-dstolee@microsoft.com","subject":"[PATCH v2 04/12] commit-graph: parse commit from chosen graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-11T21:15:18Z","receivedAt":"2018-05-11T21:15:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Before verifying a commit-graph file against the object database, we\nneed to parse all commits from the given commit-graph file. Create\nparse_commit_in_graph_one() to target a given struct commit_graph.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 18 +++++++++++++++---\n 1 file changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d2db20e49a..4dfff7e752 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -309,7 +309,7 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \t}\n }\n \n-int parse_commit_in_graph(struct commit *item)\n+int parse_commit_in_graph_one(struct commit_graph *g, struct commit *item)\n {\n \tuint32_t pos;\n \n@@ -317,9 +317,21 @@ int parse_commit_in_graph(struct commit *item)\n \t\treturn 0;\n \tif (item->object.parsed)\n \t\treturn 1;\n+\n+\tif (find_commit_in_graph(item, g, &pos))\n+\t\treturn fill_commit_in_graph(item, g, pos);\n+\n+\treturn 0;\n+}\n+\n+int parse_commit_in_graph(struct commit *item)\n+{\n+\tif (!core_commit_graph)\n+\t\treturn 0;\n+\n \tprepare_commit_graph();\n-\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n-\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n+\tif (commit_graph)\n+\t\treturn parse_commit_in_graph_one(commit_graph, item);\n \treturn 0;\n }\n \n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347386","messageId":"20180511211504.79877-6-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180511211504.79877-1-dstolee@microsoft.com","subject":"[PATCH v2 05/12] commit-graph: verify fanout and lookup table","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-11T21:15:19Z","receivedAt":"2018-05-11T21:15:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"While running 'git commit-graph verify', verify that the object IDs\nare listed in lexicographic order and that the fanout table correctly\nnavigates into that list of object IDs.\n\nAdd tests that check these corruptions are caught by the verify\nsubcommand. Most of the tests check the full output matches the exact\nerror we inserted, but since our OID order test triggers incorrect\nfanout values (with possibly different numbers of output lines) we\nfocus only that the correct error is written in that case.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 36 ++++++++++++++++++++++++++++++++++++\n t/t5318-commit-graph.sh | 31 +++++++++++++++++++++++++++++++\n 2 files changed, 67 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 4dfff7e752..b0fd1d5320 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -848,6 +848,9 @@ static void graph_report(const char *fmt, ...)\n \n int verify_commit_graph(struct commit_graph *g)\n {\n+\tuint32_t i, cur_fanout_pos = 0;\n+\tstruct object_id prev_oid, cur_oid;\n+\n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n \t\treturn 1;\n@@ -862,5 +865,38 @@ int verify_commit_graph(struct commit_graph *g)\n \tif (!g->chunk_commit_data)\n \t\tgraph_report(\"commit-graph is missing the Commit Data chunk\");\n \n+\tif (verify_commit_graph_error)\n+\t\treturn 1;\n+\n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n+\n+\t\tif (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n+\t\t\tgraph_report(\"commit-graph has incorrect oid order: %s then %s\",\n+\t\t\t\t oid_to_hex(&prev_oid),\n+\t\t\t\t oid_to_hex(&cur_oid));\n+\n+\t\toidcpy(&prev_oid, &cur_oid);\n+\n+\t\twhile (cur_oid.hash[0] > cur_fanout_pos) {\n+\t\t\tuint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n+\t\t\tif (i != fanout_value)\n+\t\t\t\tgraph_report(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\",\n+\t\t\t\t\t cur_fanout_pos, fanout_value, i);\n+\n+\t\t\tcur_fanout_pos++;\n+\t\t}\n+\t}\n+\n+\twhile (cur_fanout_pos < 256) {\n+\t\tuint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n+\n+\t\tif (g->num_commits != fanout_value)\n+\t\t\tgraph_report(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\",\n+\t\t\t\t cur_fanout_pos, fanout_value, i);\n+\n+\t\tcur_fanout_pos++;\n+\t}\n+\n \treturn verify_commit_graph_error;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 0cb88232fa..6fb306b0da 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -293,4 +293,35 @@ test_expect_success 'detect too small chunk-count' '\n \tgrep \"missing the Commit Data chunk\" verify-errors\n '\n \n+test_expect_success 'detect incorrect chunk lookup value' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tcp $objdir/info/commit-graph commit-graph-backup &&\n+\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n+\tcorrupt_data $objdir/info/commit-graph 25 \"\\01\" &&\n+\ttest_must_fail git commit-graph verify 2>err &&\n+\tgrep -v \"^\\+\" err > verify-errors &&\n+\ttest_line_count = 1 verify-errors &&\n+\tgrep \"improper chunk offset\" verify-errors\n+'\n+\n+test_expect_success 'detect incorrect fanout' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tcp $objdir/info/commit-graph commit-graph-backup &&\n+\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n+\tcorrupt_data $objdir/info/commit-graph 128 \"\\01\" &&\n+\ttest_must_fail git commit-graph verify 2>err &&\n+\tgrep -v \"^\\+\" err > verify-errors &&\n+\ttest_line_count = 1 verify-errors &&\n+\tgrep \"fanout value\" verify-errors\n+'\n+\n+test_expect_success 'detect incorrect OID order' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tcp $objdir/info/commit-graph commit-graph-backup &&\n+\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n+\tcorrupt_data $objdir/info/commit-graph 1272 \"\\01\" &&\n+\ttest_must_fail git commit-graph verify 2>err &&\n+\tgrep \"incorrect oid order\" err\n+'\n+\n test_done\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347387","messageId":"20180511211504.79877-7-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180511211504.79877-1-dstolee@microsoft.com","subject":"[PATCH v2 06/12] commit: force commit to parse from object database","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-11T21:15:20Z","receivedAt":"2018-05-11T21:15:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In anticipation of verifying commit-graph file contents against the\nobject database, create parse_commit_internal() to allow side-stepping\nthe commit-graph file and parse directly from the object database.\n\nDue to the use of generation numbers, this method should not be called\nunless the intention is explicit in avoiding commits from the\ncommit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 13 +++++++++----\n commit.h | 1 +\n 2 files changed, 10 insertions(+), 4 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex 1d28677dfb..7c92350373 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -392,7 +392,7 @@ int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long s\n \treturn 0;\n }\n \n-int parse_commit_gently(struct commit *item, int quiet_on_missing)\n+int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph)\n {\n \tenum object_type type;\n \tvoid *buffer;\n@@ -403,17 +403,17 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \t\treturn -1;\n \tif (item->object.parsed)\n \t\treturn 0;\n-\tif (parse_commit_in_graph(item))\n+\tif (use_commit_graph && parse_commit_in_graph(item))\n \t\treturn 0;\n \tbuffer = read_sha1_file(item->object.oid.hash, &type, &size);\n \tif (!buffer)\n \t\treturn quiet_on_missing ? -1 :\n \t\t\terror(\"Could not read %s\",\n-\t\t\t oid_to_hex(&item->object.oid));\n+\t\t\t\t\toid_to_hex(&item->object.oid));\n \tif (type != OBJ_COMMIT) {\n \t\tfree(buffer);\n \t\treturn error(\"Object %s not a commit\",\n-\t\t\t oid_to_hex(&item->object.oid));\n+\t\t\t\toid_to_hex(&item->object.oid));\n \t}\n \tret = parse_commit_buffer(item, buffer, size, 0);\n \tif (save_commit_buffer && !ret) {\n@@ -424,6 +424,11 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \treturn ret;\n }\n \n+int parse_commit_gently(struct commit *item, int quiet_on_missing)\n+{\n+\treturn parse_commit_internal(item, quiet_on_missing, 1);\n+}\n+\n void parse_commit_or_die(struct commit *item)\n {\n \tif (parse_commit(item))\ndiff --git a/commit.h b/commit.h\nindex b5afde1ae9..5fde74fcd7 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -73,6 +73,7 @@ struct commit *lookup_commit_reference_by_name(const char *name);\n struct commit *lookup_commit_or_die(const struct object_id *oid, const char *ref_name);\n \n int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph);\n+int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph);\n int parse_commit_gently(struct commit *item, int quiet_on_missing);\n static inline int parse_commit(struct commit *item)\n {\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347388","messageId":"20180511211504.79877-10-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180511211504.79877-1-dstolee@microsoft.com","subject":"[PATCH v2 09/12] fsck: verify commit-graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-11T21:15:23Z","receivedAt":"2018-05-11T21:15:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"If core.commitGraph is true, verify the contents of the commit-graph\nduring 'git fsck' using the 'git commit-graph verify' subcommand. Run\nthis check on all alternates, as well.\n\nWe use a new process for two reasons:\n\n1. The subcommand decouples the details of loading and verifying a\n commit-graph file from the other fsck details.\n\n2. The commit-graph verification requires the commits to be loaded\n in a specific order to guarantee we parse from the commit-graph\n file for some objects and from the object database for others.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-fsck.txt | 3 +++\n builtin/fsck.c | 21 +++++++++++++++++++++\n t/t5318-commit-graph.sh | 21 ++++++++++++++++++---\n 3 files changed, 42 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-fsck.txt b/Documentation/git-fsck.txt\nindex b9f060e3b2..ab9a93fb9b 100644\n--- a/Documentation/git-fsck.txt\n+++ b/Documentation/git-fsck.txt\n@@ -110,6 +110,9 @@ Any corrupt objects you will have to find in backups or other archives\n (i.e., you can just remove them and do an 'rsync' with some other site in\n the hopes that somebody else has the object you have corrupted).\n \n+If core.commitGraph is true, the commit-graph file will also be inspected\n+using 'git commit-graph verify'. See linkgit:git-commit-graph[1].\n+\n Extracted Diagnostics\n ---------------------\n \ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex ef78c6c00c..a6d5045b77 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -16,6 +16,7 @@\n #include \"streaming.h\"\n #include \"decorate.h\"\n #include \"packfile.h\"\n+#include \"run-command.h\"\n \n #define REACHABLE 0x0001\n #define SEEN 0x0002\n@@ -45,6 +46,7 @@ static int name_objects;\n #define ERROR_REACHABLE 02\n #define ERROR_PACK 04\n #define ERROR_REFS 010\n+#define ERROR_COMMIT_GRAPH 020\n \n static const char *describe_object(struct object *obj)\n {\n@@ -815,5 +817,24 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t}\n \n \tcheck_connectivity();\n+\n+\tif (core_commit_graph) {\n+\t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n+\t\tconst char *verify_argv[] = { \"commit-graph\", \"verify\", NULL, NULL, NULL, NULL };\n+\t\tcommit_graph_verify.argv = verify_argv;\n+\t\tcommit_graph_verify.git_cmd = 1;\n+\n+\t\tif (run_command(&commit_graph_verify))\n+\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n+\n+\t\tprepare_alt_odb();\n+\t\tfor (alt = alt_odb_list; alt; alt = alt->next) {\n+\t\t\tverify_argv[2] = \"--object-dir\";\n+\t\t\tverify_argv[3] = alt->path;\n+\t\t\tif (run_command(&commit_graph_verify))\n+\t\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n+\t\t}\n+\t}\n+\n \treturn errors_found;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 5ab268a024..91c8406d97 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -205,6 +205,16 @@ test_expect_success 'build graph from commits with append' '\n graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n \n+test_expect_success 'build graph using --reachable' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph write --reachable &&\n+\ttest_path_is_file $objdir/info/commit-graph &&\n+\tgraph_read_expect \"11\" \"large_edges\"\n+'\n+\n+graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n+graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n+\n test_expect_success 'setup bare repo' '\n \tcd \"$TRASH_DIRECTORY\" &&\n \tgit clone --bare --no-local full bare &&\n@@ -335,7 +345,7 @@ test_expect_success 'detect OID not in object database' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tcp $objdir/info/commit-graph commit-graph-backup &&\n \ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n-\tcorrupt_data $objdir/info/commit-graph 1134 \"\\01\" &&\n+\tcorrupt_data $objdir/info/commit-graph 1134 \"\\00\" &&\n \ttest_must_fail git commit-graph verify 2>err &&\n \tgrep -v \"^\\+\" err > verify-errors &&\n \ttest_line_count = 3 verify-errors &&\n@@ -348,7 +358,7 @@ test_expect_success 'detect incorrect tree OID' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tcp $objdir/info/commit-graph commit-graph-backup &&\n \ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n-\tcorrupt_data $objdir/info/commit-graph 1312 \"\\01\" &&\n+\tcorrupt_data $objdir/info/commit-graph 1312 \"\\00\" &&\n \ttest_must_fail git commit-graph verify 2>err &&\n \tgrep -v \"^\\+\" err > verify-errors &&\n \ttest_line_count = 1 verify-errors &&\n@@ -382,10 +392,15 @@ test_expect_success 'detect incorrect commit date and generation number' '\n \tcp $objdir/info/commit-graph commit-graph-backup &&\n \ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n \tcorrupt_data $objdir/info/commit-graph 1340 \"\\01\" &&\n-\tcorrupt_data $objdir/info/commit-graph 1344 \"\\01\" &&\n+\tcorrupt_data $objdir/info/commit-graph 1344 \"\\00\" &&\n \ttest_must_fail git commit-graph verify 2>err &&\n \tgrep \"incorrect generation\" err &&\n \tgrep \"commit date\" err\n '\n \n+test_expect_success 'git fsck (checks commit-graph)' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit fsck\n+'\n+\n test_done\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347389","messageId":"20180511211504.79877-12-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180511211504.79877-1-dstolee@microsoft.com","subject":"[PATCH v2 11/12] gc: automatically write commit-graph files","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-11T21:15:24Z","receivedAt":"2018-05-11T21:15:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file is a very helpful feature for speeding up git\noperations. In order to make it more useful, write the commit-graph file\nby default during standard garbage collection operations.\n\nAdd a 'gc.commitGraph' config setting that triggers writing a\ncommit-graph file after any non-trivial 'git gc' command. Defaults to\nfalse while the commit-graph feature matures. We specifically do not\nwant to turn this on by default until the commit-graph feature is fully\nintegrated with history-modifying features like shallow clones.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 6 ++++++\n Documentation/git-gc.txt | 4 ++++\n builtin/gc.c | 8 ++++++++\n 3 files changed, 18 insertions(+)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 11f027194e..9a3abd87e7 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1553,6 +1553,12 @@ gc.autoDetach::\n \tMake `git gc --auto` return immediately and run in background\n \tif the system supports it. Default is true.\n \n+gc.commitGraph::\n+\tIf true, then gc will rewrite the commit-graph file after any\n+\tchange to the object database. If '--auto' is used, then the\n+\tcommit-graph will not be updated unless the threshold is met.\n+\tSee linkgit:git-commit-graph[1] for details.\n+\n gc.logExpiry::\n \tIf the file gc.log exists, then `git gc --auto` won't run\n \tunless that file is more than 'gc.logExpiry' old. Default is\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 571b5a7e3c..17dd654a59 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -119,6 +119,10 @@ The optional configuration variable `gc.packRefs` determines if\n it within all non-bare repos or it can be set to a boolean value.\n This defaults to true.\n \n+The optional configuration variable 'gc.commitGraph' determines if\n+'git gc' runs 'git commit-graph write'. This can be set to a boolean\n+value. This defaults to false.\n+\n The optional configuration variable `gc.aggressiveWindow` controls how\n much time is spent optimizing the delta compression of the objects in\n the repository when the --aggressive option is specified. The larger\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 77fa720bd0..8403445738 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -34,6 +34,7 @@ static int aggressive_depth = 50;\n static int aggressive_window = 250;\n static int gc_auto_threshold = 6700;\n static int gc_auto_pack_limit = 50;\n+static int gc_commit_graph = 0;\n static int detach_auto = 1;\n static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n@@ -46,6 +47,7 @@ static struct argv_array repack = ARGV_ARRAY_INIT;\n static struct argv_array prune = ARGV_ARRAY_INIT;\n static struct argv_array prune_worktrees = ARGV_ARRAY_INIT;\n static struct argv_array rerere = ARGV_ARRAY_INIT;\n+static struct argv_array commit_graph = ARGV_ARRAY_INIT;\n \n static struct tempfile *pidfile;\n static struct lock_file log_lock;\n@@ -121,6 +123,7 @@ static void gc_config(void)\n \tgit_config_get_int(\"gc.aggressivedepth\", &aggressive_depth);\n \tgit_config_get_int(\"gc.auto\", &gc_auto_threshold);\n \tgit_config_get_int(\"gc.autopacklimit\", &gc_auto_pack_limit);\n+\tgit_config_get_bool(\"gc.commitgraph\", &gc_commit_graph);\n \tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n \tgit_config_get_expiry(\"gc.pruneexpire\", &prune_expire);\n \tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n@@ -374,6 +377,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \targv_array_pushl(&prune, \"prune\", \"--expire\", NULL);\n \targv_array_pushl(&prune_worktrees, \"worktree\", \"prune\", \"--expire\", NULL);\n \targv_array_pushl(&rerere, \"rerere\", \"gc\", NULL);\n+\targv_array_pushl(&commit_graph, \"commit-graph\", \"write\", \"--reachable\", NULL);\n \n \t/* default expiry time, overwritten in gc_config */\n \tgc_config();\n@@ -480,6 +484,10 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tif (pack_garbage.nr > 0)\n \t\tclean_pack_garbage();\n \n+\tif (gc_commit_graph)\n+\t\tif (run_command_v_opt(commit_graph.argv, RUN_GIT_CMD))\n+\t\t\treturn error(FAILED_RUN, commit_graph.argv[0]);\n+\n \tif (auto_gc && too_many_loose_objects())\n \t\twarning(_(\"There are too many unreachable loose objects; \"\n \t\t\t\"run 'git prune' to remove them.\"));\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347390","messageId":"20180511211504.79877-13-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180511211504.79877-1-dstolee@microsoft.com","subject":"[PATCH v2 12/12] commit-graph: update design document","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-11T21:15:25Z","receivedAt":"2018-05-11T21:15:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph feature is now integrated with 'fsck' and 'gc',\nso remove those items from the \"Future Work\" section of the\ncommit-graph design document.\n\nAlso remove the section on lazy-loading trees, as that was completed\nin an earlier patch series.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 22 ----------------------\n 1 file changed, 22 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex e1a883eb46..c664acbd76 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -118,9 +118,6 @@ Future Work\n - The commit graph feature currently does not honor commit grafts. This can\n be remedied by duplicating or refactoring the current graft logic.\n \n-- The 'commit-graph' subcommand does not have a \"verify\" mode that is\n- necessary for integration with fsck.\n-\n - After computing and storing generation numbers, we must make graph\n walks aware of generation numbers to gain the performance benefits they\n enable. This will mostly be accomplished by swapping a commit-date-ordered\n@@ -130,25 +127,6 @@ Future Work\n - 'log --topo-order'\n - 'tag --merged'\n \n-- Currently, parse_commit_gently() requires filling in the root tree\n- object for a commit. This passes through lookup_tree() and consequently\n- lookup_object(). Also, it calls lookup_commit() when loading the parents.\n- These method calls check the ODB for object existence, even if the\n- consumer does not need the content. For example, we do not need the\n- tree contents when computing merge bases. Now that commit parsing is\n- removed from the computation time, these lookup operations are the\n- slowest operations keeping graph walks from being fast. Consider\n- loading these objects without verifying their existence in the ODB and\n- only loading them fully when consumers need them. Consider a method\n- such as \"ensure_tree_loaded(commit)\" that fully loads a tree before\n- using commit->tree.\n-\n-- The current design uses the 'commit-graph' subcommand to generate the graph.\n- When this feature stabilizes enough to recommend to most users, we should\n- add automatic graph writes to common operations that create many commits.\n- For example, one could compute a graph on 'clone', 'fetch', or 'repack'\n- commands.\n-\n - A server could provide a commit graph file as part of the network protocol\n to avoid extra calculations by clients. This feature is only of benefit if\n the user is willing to trust the file, because verifying the file is correct\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347391","messageId":"20180511211504.79877-8-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180511211504.79877-1-dstolee@microsoft.com","subject":"[PATCH v2 07/12] commit-graph: load a root tree from specific graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-11T21:15:21Z","receivedAt":"2018-05-11T21:15:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When lazy-loading a tree for a commit, it will be important to select\nthe tree from a specific struct commit_graph. Create a new method that\nspecifies the commit-graph file and use that in\nget_commit_tree_in_graph().\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 10 ++++++++--\n 1 file changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex b0fd1d5320..5bb93e533c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -357,14 +357,20 @@ static struct tree *load_tree_for_commit(struct commit_graph *g, struct commit *\n \treturn c->maybe_tree;\n }\n \n-struct tree *get_commit_tree_in_graph(const struct commit *c)\n+static struct tree *get_commit_tree_in_graph_one(struct commit_graph *g,\n+\t\t\t\t\t\t const struct commit *c)\n {\n \tif (c->maybe_tree)\n \t\treturn c->maybe_tree;\n \tif (c->graph_pos == COMMIT_NOT_FROM_GRAPH)\n \t\tBUG(\"get_commit_tree_in_graph called from non-commit-graph commit\");\n \n-\treturn load_tree_for_commit(commit_graph, (struct commit *)c);\n+\treturn load_tree_for_commit(g, (struct commit *)c);\n+}\n+\n+struct tree *get_commit_tree_in_graph(const struct commit *c)\n+{\n+\treturn get_commit_tree_in_graph_one(commit_graph, c);\n }\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347392","messageId":"20180511211504.79877-9-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180511211504.79877-1-dstolee@microsoft.com","subject":"[PATCH v2 08/12] commit-graph: verify commit contents against odb","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-11T21:15:22Z","receivedAt":"2018-05-11T21:15:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When running 'git commit-graph verify', compare the contents of the\ncommits that are loaded from the commit-graph file with commits that are\nloaded directly from the object database. This includes checking the\nroot tree object ID, commit date, and parents.\n\nParse the commit from the graph during the initial loop through the\nobject IDs to guarantee we parse from the commit-graph file.\n\nIn addition, verify the generation number calculation is correct for all\ncommits in the commit-graph file.\n\nWhile testing, we discovered that mutating the integer value for a\nparent to be outside the accepted range causes a segmentation fault. Add\na new check in insert_parent_or_die() that prevents this fault. Check\nfor that error during the test, both in the typical parents and in the\nlist of parents for octopus merges.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++++\n t/t5318-commit-graph.sh | 64 +++++++++++++++++++++++++++++++\n 2 files changed, 164 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 5bb93e533c..a15ad9710d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -237,6 +237,10 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n {\n \tstruct commit *c;\n \tstruct object_id oid;\n+\n+\tif (pos >= g->num_commits)\n+\t\tdie(\"invalide parent position %\"PRIu64, pos);\n+\n \thashcpy(oid.hash, g->chunk_oid_lookup + g->hash_len * pos);\n \tc = lookup_commit(&oid);\n \tif (!c)\n@@ -875,6 +879,8 @@ int verify_commit_graph(struct commit_graph *g)\n \t\treturn 1;\n \n \tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct commit *graph_commit;\n+\n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n \t\tif (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n@@ -892,6 +898,10 @@ int verify_commit_graph(struct commit_graph *g)\n \n \t\t\tcur_fanout_pos++;\n \t\t}\n+\n+\t\tgraph_commit = lookup_commit(&cur_oid);\n+\t\tif (!parse_commit_in_graph_one(g, graph_commit))\n+\t\t\tgraph_report(\"failed to parse %s from commit-graph\", oid_to_hex(&cur_oid));\n \t}\n \n \twhile (cur_fanout_pos < 256) {\n@@ -904,5 +914,95 @@ int verify_commit_graph(struct commit_graph *g)\n \t\tcur_fanout_pos++;\n \t}\n \n+\tif (verify_commit_graph_error)\n+\t\treturn 1;\n+\n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct commit *graph_commit, *odb_commit;\n+\t\tstruct commit_list *graph_parents, *odb_parents;\n+\t\tint num_parents = 0;\n+\n+\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n+\n+\t\tgraph_commit = lookup_commit(&cur_oid);\n+\t\todb_commit = (struct commit *)create_object(cur_oid.hash, alloc_commit_node());\n+\t\tif (parse_commit_internal(odb_commit, 0, 0)) {\n+\t\t\tgraph_report(\"failed to parse %s from object database\", oid_to_hex(&cur_oid));\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (oidcmp(&get_commit_tree_in_graph_one(g, graph_commit)->object.oid,\n+\t\t\t get_commit_tree_oid(odb_commit)))\n+\t\t\tgraph_report(\"root tree object ID for commit %s in commit-graph is %s != %s\",\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t oid_to_hex(get_commit_tree_oid(graph_commit)),\n+\t\t\t\t oid_to_hex(get_commit_tree_oid(odb_commit)));\n+\n+\t\tif (graph_commit->date != odb_commit->date)\n+\t\t\tgraph_report(\"commit date for commit %s in commit-graph is %\"PRItime\" != %\"PRItime\"\",\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t graph_commit->date,\n+\t\t\t\t odb_commit->date);\n+\n+\n+\t\tgraph_parents = graph_commit->parents;\n+\t\todb_parents = odb_commit->parents;\n+\n+\t\twhile (graph_parents) {\n+\t\t\tnum_parents++;\n+\n+\t\t\tif (odb_parents == NULL)\n+\t\t\t\tgraph_report(\"commit-graph parent list for commit %s is too long (%d)\",\n+\t\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t\t num_parents);\n+\n+\t\t\tif (oidcmp(&graph_parents->item->object.oid, &odb_parents->item->object.oid))\n+\t\t\t\tgraph_report(\"commit-graph parent for %s is %s != %s\",\n+\t\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid),\n+\t\t\t\t\t oid_to_hex(&odb_parents->item->object.oid));\n+\n+\t\t\tgraph_parents = graph_parents->next;\n+\t\t\todb_parents = odb_parents->next;\n+\t\t}\n+\n+\t\tif (odb_parents != NULL)\n+\t\t\tgraph_report(\"commit-graph parent list for commit %s terminates early\",\n+\t\t\t\t oid_to_hex(&cur_oid));\n+\n+\t\tif (graph_commit->generation) {\n+\t\t\tuint32_t max_generation = 0;\n+\t\t\tgraph_parents = graph_commit->parents;\n+\n+\t\t\twhile (graph_parents) {\n+\t\t\t\tif (graph_parents->item->generation == GENERATION_NUMBER_ZERO ||\n+\t\t\t\t graph_parents->item->generation == GENERATION_NUMBER_INFINITY)\n+\t\t\t\t\tgraph_report(\"commit-graph has valid generation for %s but not its parent, %s\",\n+\t\t\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid));\n+\t\t\t\tif (graph_parents->item->generation > max_generation)\n+\t\t\t\t\tmax_generation = graph_parents->item->generation;\n+\t\t\t\tgraph_parents = graph_parents->next;\n+\t\t\t}\n+\n+\t\t\tif (max_generation == GENERATION_NUMBER_MAX)\n+\t\t\t\tmax_generation--;\n+\n+\t\t\tif (graph_commit->generation != max_generation + 1)\n+\t\t\t\tgraph_report(\"commit-graph has incorrect generation for %s\",\n+\t\t\t\t\t oid_to_hex(&cur_oid));\n+\t\t} else {\n+\t\t\tgraph_parents = graph_commit->parents;\n+\n+\t\t\twhile (graph_parents) {\n+\t\t\t\tif (graph_parents->item->generation)\n+\t\t\t\t\tgraph_report(\"commit-graph has generation ZERO for %s but not its parent, %s\",\n+\t\t\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid));\n+\t\t\t\tgraph_parents = graph_parents->next;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n \treturn verify_commit_graph_error;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 6fb306b0da..5ab268a024 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -235,8 +235,15 @@ test_expect_success 'perform fast-forward merge in full repo' '\n \ttest_cmp expect output\n '\n \n+# the verify tests below expect the commit-graph to contain all\n+# commits from the 'full' repo to be in the commit-graph file.\n+# If the file changes the set of commits in the list, then the\n+# offsets into the binary file will result in different edits\n+# and the tests will likely break.\n+\n test_expect_success 'git commit-graph verify' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit show-ref -s | git commit-graph write --stdin-commits &&\n \tgit commit-graph verify >output\n '\n \n@@ -324,4 +331,61 @@ test_expect_success 'detect incorrect OID order' '\n \tgrep \"incorrect oid order\" err\n '\n \n+test_expect_success 'detect OID not in object database' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tcp $objdir/info/commit-graph commit-graph-backup &&\n+\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n+\tcorrupt_data $objdir/info/commit-graph 1134 \"\\01\" &&\n+\ttest_must_fail git commit-graph verify 2>err &&\n+\tgrep -v \"^\\+\" err > verify-errors &&\n+\ttest_line_count = 3 verify-errors &&\n+\tgrep \"Could not read\" verify-errors &&\n+\tgrep \"parent\" verify-errors &&\n+\tgrep \"from object database\" verify-errors\n+'\n+\n+test_expect_success 'detect incorrect tree OID' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tcp $objdir/info/commit-graph commit-graph-backup &&\n+\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n+\tcorrupt_data $objdir/info/commit-graph 1312 \"\\01\" &&\n+\ttest_must_fail git commit-graph verify 2>err &&\n+\tgrep -v \"^\\+\" err > verify-errors &&\n+\ttest_line_count = 1 verify-errors &&\n+\tgrep \"root tree object ID for commit \" verify-errors\n+'\n+\n+test_expect_success 'detect incorrect parent id' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tcp $objdir/info/commit-graph commit-graph-backup &&\n+\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n+\tcorrupt_data $objdir/info/commit-graph 1332 \"\\01\" &&\n+\ttest_must_fail git commit-graph verify 2>err &&\n+\tgrep -v \"^\\+\" err > verify-errors &&\n+\ttest_line_count = 1 verify-errors &&\n+\tgrep \"parent \" verify-errors\n+'\n+\n+test_expect_success 'detect incorrect parent id in large edges' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tcp $objdir/info/commit-graph commit-graph-backup &&\n+\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n+\tcorrupt_data $objdir/info/commit-graph 1712 \"\\01\" &&\n+\ttest_must_fail git commit-graph verify 2>err &&\n+\tgrep -v \"^\\+\" err > verify-errors &&\n+\ttest_line_count = 1 verify-errors &&\n+\tgrep \"parent \" verify-errors\n+'\n+\n+test_expect_success 'detect incorrect commit date and generation number' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tcp $objdir/info/commit-graph commit-graph-backup &&\n+\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n+\tcorrupt_data $objdir/info/commit-graph 1340 \"\\01\" &&\n+\tcorrupt_data $objdir/info/commit-graph 1344 \"\\01\" &&\n+\ttest_must_fail git commit-graph verify 2>err &&\n+\tgrep \"incorrect generation\" err &&\n+\tgrep \"commit date\" err\n+'\n+\n test_done\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347393","messageId":"20180511211504.79877-11-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180511211504.79877-1-dstolee@microsoft.com","subject":"[PATCH v2 10/12] commit-graph: add '--reachable' option","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-11T21:15:24Z","receivedAt":"2018-05-11T21:15:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When writing commit-graph files, it can be convenient to ask for all\nreachable commits (starting at the ref set) in the resulting file. This\nis particularly helpful when writing to stdin is complicated, such as a\nfuture integration with 'git gc' which will call\n'git commit-graph write --reachable' after performing cleanup of the\nobject database.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 8 ++++++--\n builtin/commit-graph.c | 41 ++++++++++++++++++++++++++++++++++----\n 2 files changed, 43 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex a222cfab08..cc1715a823 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -38,12 +38,16 @@ Write a commit graph file based on the commits found in packfiles.\n +\n With the `--stdin-packs` option, generate the new commit graph by\n walking objects only in the specified pack-indexes. (Cannot be combined\n-with --stdin-commits.)\n+with --stdin-commits or --reachable.)\n +\n With the `--stdin-commits` option, generate the new commit graph by\n walking commits starting at the commits specified in stdin as a list\n of OIDs in hex, one OID per line. (Cannot be combined with\n---stdin-packs.)\n+--stdin-packs or --reachable.)\n++\n+With the `--reachable` option, generate the new commit graph by walking\n+commits starting at all refs. (Cannot be combined with --stdin-commits\n+or --stind-packs.)\n +\n With the `--append` option, include all commits that are present in the\n existing commit-graph file.\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex af3101291f..7cb94a4813 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -3,13 +3,14 @@\n #include \"dir.h\"\n #include \"lockfile.h\"\n #include \"parse-options.h\"\n+#include \"refs.h\"\n #include \"commit-graph.h\"\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph verify [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -24,12 +25,13 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n static struct opts_commit_graph {\n \tconst char *obj_dir;\n+\tint reachable;\n \tint stdin_packs;\n \tint stdin_commits;\n \tint append;\n@@ -113,6 +115,25 @@ static int graph_read(int argc, const char **argv)\n \treturn 0;\n }\n \n+struct hex_list {\n+\tchar **hex_strs;\n+\tint hex_nr;\n+\tint hex_alloc;\n+};\n+\n+static int add_ref_to_list(const char *refname,\n+\t\t\t const struct object_id *oid,\n+\t\t\t int flags, void *cb_data)\n+{\n+\tstruct hex_list *list = (struct hex_list*)cb_data;\n+\n+\tALLOC_GROW(list->hex_strs, list->hex_nr + 1, list->hex_alloc);\n+\tlist->hex_strs[list->hex_nr] = xcalloc(GIT_MAX_HEXSZ + 1, 1);\n+\tstrcpy(list->hex_strs[list->hex_nr], oid_to_hex(oid));\n+\tlist->hex_nr++;\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tconst char **pack_indexes = NULL;\n@@ -127,6 +148,8 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n \t\t\tN_(\"dir\"),\n \t\t\tN_(\"The object directory to store the graph\")),\n+\t\tOPT_BOOL(0, \"reachable\", &opts.reachable,\n+\t\t\tN_(\"start walk at all refs\")),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\tN_(\"scan pack-indexes listed by stdin for commits\")),\n \t\tOPT_BOOL(0, \"stdin-commits\", &opts.stdin_commits,\n@@ -140,8 +163,8 @@ static int graph_write(int argc, const char **argv)\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\n \n-\tif (opts.stdin_packs && opts.stdin_commits)\n-\t\tdie(_(\"cannot use both --stdin-commits and --stdin-packs\"));\n+\tif (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)\n+\t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n \n@@ -164,6 +187,16 @@ static int graph_write(int argc, const char **argv)\n \t\t\tcommit_hex = lines;\n \t\t\tcommits_nr = lines_nr;\n \t\t}\n+\t} else if (opts.reachable) {\n+\t\tstruct hex_list list;\n+\t\tlist.hex_nr = 0;\n+\t\tlist.hex_alloc = 128;\n+\t\tALLOC_ARRAY(list.hex_strs, list.hex_alloc);\n+\n+\t\tfor_each_ref(add_ref_to_list, &list);\n+\n+\t\tcommit_hex = (const char **)list.hex_strs;\n+\t\tcommits_nr = list.hex_nr;\n \t}\n \n \twrite_commit_graph(opts.obj_dir,\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"347456","messageId":"CAN0heSotYYgrwMLT1yyFVk4VEZ4OQKF_Xwm9-wZosJ6_-gAVBg@mail.gmail.com","threadId":"48311","inReplyTo":"20180511211504.79877-2-dstolee@microsoft.com","subject":"Re: [PATCH v2 01/12] commit-graph: add 'verify' subcommand","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2018-05-12T13:31:34Z","receivedAt":"2018-05-12T13:31:38Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 11 May 2018 at 23:15, Derrick Stolee wrote:\n\n> graph_name = get_commit_graph_filename(opts.obj_dir);\n> graph = load_commit_graph_one(graph_name);\n> + FREE_AND_NULL(graph_name);\n>\n> if (!graph)\n> die(\"graph file %s does not exist\", graph_name);\n> - FREE_AND_NULL(graph_name);\n\nThis is probably because of something I said, but this does not look\ncorrect. The `die()` would typically print \"(null)\" or segfault. If the\n`die()` means we don't free `graph_name`, that should be fine.\n\nYou're still leaking `graph` here (possibly nothing this patch should\nworry about) and in `graph_verify()`. UNLEAK-ing it immediately before\ncalling `verify_commit_graph()` should be ok. I also think punting on\nthis UNLEAK-business entirely would be ok; I was just a bit surprised to\nsee one variable getting freed and the other one ignored.\n\nMartin\n"},{"id":"347457","messageId":"CAN0heSrKFJbgwXXzUKaJQn24pY43S2bsn9Zx6Yyy6M2X2wDJ7w@mail.gmail.com","threadId":"48311","inReplyTo":"20180511211504.79877-3-dstolee@microsoft.com","subject":"Re: [PATCH v2 02/12] commit-graph: verify file header information","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2018-05-12T13:35:21Z","receivedAt":"2018-05-12T13:35:25Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 11 May 2018 at 23:15, Derrick Stolee wrote:\n> During a run of 'git commit-graph verify', list the issues with the\n> header information in the commit-graph file. Some of this information\n> is inferred from the loaded 'struct commit_graph'. Some header\n> information is checked as part of load_commit_graph_one().\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 32 +++++++++++++++++++++++++++++++-\n> 1 file changed, 31 insertions(+), 1 deletion(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index b25aaed128..d2db20e49a 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -818,7 +818,37 @@ void write_commit_graph(const char *obj_dir,\n> oids.nr = 0;\n> }\n>\n> +static int verify_commit_graph_error;\n> +\n> +static void graph_report(const char *fmt, ...)\n> +{\n> + va_list ap;\n> + struct strbuf sb = STRBUF_INIT;\n> + verify_commit_graph_error = 1;\n> +\n> + va_start(ap, fmt);\n> + strbuf_vaddf(&sb, fmt, ap);\n> +\n> + fprintf(stderr, \"%s\\n\", sb.buf);\n> + strbuf_release(&sb);\n> + va_end(ap);\n> +}\n\nRight, so this replaces the macro-trickery from v1, and we print a\nnewline after each error.\n\n> int verify_commit_graph(struct commit_graph *g)\n> {\n> - return !g;\n> + if (!g) {\n> + graph_report(\"no commit-graph file loaded\");\n> + return 1;\n> + }\n\n> +\n> + return verify_commit_graph_error;\n> }\n\nNot sure it matters much: I suppose you could introduce the parts that I\nhave quoted here in the previous patch. Or maybe not.\n\nMartin\n"},{"id":"347458","messageId":"CAN0heSqYOY4HyinuL7ye0XL3u+rUxWeT9KwsiXWCcsVN_hmc2Q@mail.gmail.com","threadId":"48311","inReplyTo":"20180511211504.79877-4-dstolee@microsoft.com","subject":"Re: [PATCH v2 03/12] commit-graph: test that 'verify' finds corruption","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2018-05-12T13:43:36Z","receivedAt":"2018-05-12T13:43:40Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 11 May 2018 at 23:15, Derrick Stolee wrote:\n\n> +test_expect_success 'detect bad signature' '\n> + cd \"$TRASH_DIRECTORY/full\" &&\n\nI was a bit surprised at the \"cd outside subshell\", but then realized\nthat this file already does that. It will only be a problem if later\ntests think they're somewhere else. Let's read on.\n\n> + cp $objdir/info/commit-graph commit-graph-backup &&\n> + test_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n> + corrupt_data $objdir/info/commit-graph 0 \"\\0\" &&\n> + test_must_fail git commit-graph verify 2>err &&\n> + grep -v \"^\\+\" err > verify-errors &&\n> + test_line_count = 1 verify-errors &&\n> + grep \"graph signature\" verify-errors\n> +'\n> +\n> +test_expect_success 'detect bad version number' '\n> + cd \"$TRASH_DIRECTORY/full\" &&\n> + cp $objdir/info/commit-graph commit-graph-backup &&\n> + test_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n> + corrupt_data $objdir/info/commit-graph 4 \"\\02\" &&\n> + test_must_fail git commit-graph verify 2>err &&\n> + grep -v \"^\\+\" err > verify-errors &&\n> + test_line_count = 1 verify-errors &&\n> + grep \"graph version\" verify-errors\n> +'\n> +\n> +test_expect_success 'detect bad hash version' '\n> + cd \"$TRASH_DIRECTORY/full\" &&\n> + cp $objdir/info/commit-graph commit-graph-backup &&\n> + test_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n> + corrupt_data $objdir/info/commit-graph 5 \"\\02\" &&\n> + test_must_fail git commit-graph verify 2>err &&\n> + grep -v \"^\\+\" err > verify-errors &&\n> + test_line_count = 1 verify-errors &&\n> + grep \"hash version\" verify-errors\n> +'\n\nThese look a bit boiler-platey. Maybe not too bad though.\n\n> +test_expect_success 'detect too small chunk-count' '\n\ns/too small/bad/?\n\nTo be honest, I wrote this title without thinking too hard about the\nproblem. In general, it would be quite hard for `git commit-graph\nverify` to say \"*this* is wrong in your file\" (\"the number of chunks is\ntoo small\") -- it should be much easier to say \"*something* is wrong\".\n\n> + cd \"$TRASH_DIRECTORY/full\" &&\n> + cp $objdir/info/commit-graph commit-graph-backup &&\n> + test_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n> + corrupt_data $objdir/info/commit-graph 6 \"\\01\" &&\n> + test_must_fail git commit-graph verify 2>err &&\n> + grep -v \"^\\+\" err > verify-errors &&\n> + test_line_count = 2 verify-errors &&\n> + grep \"missing the OID Lookup chunk\" verify-errors &&\n> + grep \"missing the Commit Data chunk\" verify-errors\n\nMaybe these tests could go with the previous patch(es). IMVHO I would\nprefer reading the test with the implementation. A separate commit for\nthe tests might make sense if they're really tricky and need some\nexplaining, but I don't think that's the case here.\n\nAll of these comments are just minor nits, or not even that. I will\ncontinue with the other patches at another time.\n\nThank you, I'm really looking forward to Git with commit-graph magic.\n\nMartin\n"},{"id":"347471","messageId":"CAN0heSqHPYjZJbzgFbvJTLH0UZFB2uUhxKBNPxMX5ufYqMXKaw@mail.gmail.com","threadId":"48311","inReplyTo":"20180511211504.79877-5-dstolee@microsoft.com","subject":"Re: [PATCH v2 04/12] commit-graph: parse commit from chosen graph","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2018-05-12T20:50:22Z","receivedAt":"2018-05-12T20:50:26Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"> -int parse_commit_in_graph(struct commit *item)\n> +int parse_commit_in_graph_one(struct commit_graph *g, struct commit *item)\n\nI think this function should be static.\n"},{"id":"347472","messageId":"CAN0heSo681Db75Nj=kfUT441+zmL6gDzoPAHTzC7R+Mh2ozb_Q@mail.gmail.com","threadId":"48311","inReplyTo":"20180511211504.79877-7-dstolee@microsoft.com","subject":"Re: [PATCH v2 06/12] commit: force commit to parse from object database","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2018-05-12T20:54:20Z","receivedAt":"2018-05-12T20:54:24Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 11 May 2018 at 23:15, Derrick Stolee wrote:\n\n> -int parse_commit_gently(struct commit *item, int quiet_on_missing)\n> +int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph)\n> {\n> enum object_type type;\n> void *buffer;\n> @@ -403,17 +403,17 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n> return -1;\n> if (item->object.parsed)\n> return 0;\n> - if (parse_commit_in_graph(item))\n> + if (use_commit_graph && parse_commit_in_graph(item))\n> return 0;\n\nRight, this is where we check the graph. It's the only place we need to\nconsider the new flag.\n\n> buffer = read_sha1_file(item->object.oid.hash, &type, &size);\n> if (!buffer)\n> return quiet_on_missing ? -1 :\n> error(\"Could not read %s\",\n> - oid_to_hex(&item->object.oid));\n> + oid_to_hex(&item->object.oid));\n> if (type != OBJ_COMMIT) {\n> free(buffer);\n> return error(\"Object %s not a commit\",\n> - oid_to_hex(&item->object.oid));\n> + oid_to_hex(&item->object.oid));\n\nSome spurious indentation reshuffling going on in two lines here.\n\n> --- a/commit.h\n> +++ b/commit.h\n> @@ -73,6 +73,7 @@ struct commit *lookup_commit_reference_by_name(const char *name);\n> struct commit *lookup_commit_or_die(const struct object_id *oid, const char *ref_name);\n>\n> int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph);\n> +int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph);\n\nUnlike my comment on a previous patch, this one is meant for external\nuse. That's why it's not marked as static above. Ok.\n\nMartin\n"},{"id":"347473","messageId":"CAN0heSqbObaKK0MfUmy7g1fkdi4UYjVSxPaVvEPADHLXG8so0g@mail.gmail.com","threadId":"48311","inReplyTo":"20180511211504.79877-8-dstolee@microsoft.com","subject":"Re: [PATCH v2 07/12] commit-graph: load a root tree from specific graph","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2018-05-12T20:55:48Z","receivedAt":"2018-05-12T20:55:52Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 11 May 2018 at 23:15, Derrick Stolee wrote:\n\n> -struct tree *get_commit_tree_in_graph(const struct commit *c)\n> +static struct tree *get_commit_tree_in_graph_one(struct commit_graph *g,\n> + const struct commit *c)\n> {\n> if (c->maybe_tree)\n> return c->maybe_tree;\n> if (c->graph_pos == COMMIT_NOT_FROM_GRAPH)\n> BUG(\"get_commit_tree_in_graph called from non-commit-graph commit\");\n\nUpdate the function name in the BUG? Not that it will ever matter. ;-)\n\n(This one is now static, ok.)\n"},{"id":"347475","messageId":"CAN0heSo2sTgVSoxUiugSaEjev+0eyzCRA5ZmQVr6FRtKkyGJsQ@mail.gmail.com","threadId":"48311","inReplyTo":"20180511211504.79877-9-dstolee@microsoft.com","subject":"Re: [PATCH v2 08/12] commit-graph: verify commit contents against odb","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2018-05-12T21:17:37Z","receivedAt":"2018-05-12T21:17:41Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 11 May 2018 at 23:15, Derrick Stolee wrote:\n> When running 'git commit-graph verify', compare the contents of the\n> commits that are loaded from the commit-graph file with commits that are\n> loaded directly from the object database. This includes checking the\n> root tree object ID, commit date, and parents.\n>\n> Parse the commit from the graph during the initial loop through the\n> object IDs to guarantee we parse from the commit-graph file.\n>\n> In addition, verify the generation number calculation is correct for all\n> commits in the commit-graph file.\n>\n> While testing, we discovered that mutating the integer value for a\n> parent to be outside the accepted range causes a segmentation fault. Add\n> a new check in insert_parent_or_die() that prevents this fault. Check\n> for that error during the test, both in the typical parents and in the\n> list of parents for octopus merges.\n\nThis paragraph and the corresponding fix and test feel like a separate\npatch to me. (The commit message of it could be \"To test the next patch,\nwe threw invalid data at `git commit-graph verify, and it crashed in\npre-existing code, so let's fix that first\" -- there is definitely a\nconnection.) Is this important enough to fast-track to master in time\nfor 2.18? My guess would be no.\n\n> +\n> + if (pos >= g->num_commits)\n> + die(\"invalide parent position %\"PRIu64, pos);\n\ns/invalide/invalid/\n\n> @@ -875,6 +879,8 @@ int verify_commit_graph(struct commit_graph *g)\n> return 1;\n>\n> for (i = 0; i < g->num_commits; i++) {\n> + struct commit *graph_commit;\n> +\n> hashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n>\n> if (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n> @@ -892,6 +898,10 @@ int verify_commit_graph(struct commit_graph *g)\n>\n> cur_fanout_pos++;\n> }\n> +\n> + graph_commit = lookup_commit(&cur_oid);\n> + if (!parse_commit_in_graph_one(g, graph_commit))\n> + graph_report(\"failed to parse %s from commit-graph\", oid_to_hex(&cur_oid));\n> }\n\nCould this end up giving ridiculous amounts of output? It would depend\non the input, I guess.\n\n> while (cur_fanout_pos < 256) {\n> @@ -904,5 +914,95 @@ int verify_commit_graph(struct commit_graph *g)\n> cur_fanout_pos++;\n> }\n>\n> + if (verify_commit_graph_error)\n> + return 1;\n\nWell, here we give up before running into *too* much problem.\n\n> + for (i = 0; i < g->num_commits; i++) {\n> + struct commit *graph_commit, *odb_commit;\n> + struct commit_list *graph_parents, *odb_parents;\n> + int num_parents = 0;\n> +\n> + hashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n> +\n> + graph_commit = lookup_commit(&cur_oid);\n> + odb_commit = (struct commit *)create_object(cur_oid.hash, alloc_commit_node());\n> + if (parse_commit_internal(odb_commit, 0, 0)) {\n> + graph_report(\"failed to parse %s from object database\", oid_to_hex(&cur_oid));\n> + continue;\n> + }\n> +\n> + if (oidcmp(&get_commit_tree_in_graph_one(g, graph_commit)->object.oid,\n> + get_commit_tree_oid(odb_commit)))\n> + graph_report(\"root tree object ID for commit %s in commit-graph is %s != %s\",\n> + oid_to_hex(&cur_oid),\n> + oid_to_hex(get_commit_tree_oid(graph_commit)),\n> + oid_to_hex(get_commit_tree_oid(odb_commit)));\n> +\n> + if (graph_commit->date != odb_commit->date)\n> + graph_report(\"commit date for commit %s in commit-graph is %\"PRItime\" != %\"PRItime\"\",\n> + oid_to_hex(&cur_oid),\n> + graph_commit->date,\n> + odb_commit->date);\n> +\n> +\n> + graph_parents = graph_commit->parents;\n> + odb_parents = odb_commit->parents;\n> +\n> + while (graph_parents) {\n> + num_parents++;\n> +\n> + if (odb_parents == NULL)\n> + graph_report(\"commit-graph parent list for commit %s is too long (%d)\",\n> + oid_to_hex(&cur_oid),\n> + num_parents);\n> +\n> + if (oidcmp(&graph_parents->item->object.oid, &odb_parents->item->object.oid))\n> + graph_report(\"commit-graph parent for %s is %s != %s\",\n> + oid_to_hex(&cur_oid),\n> + oid_to_hex(&graph_parents->item->object.oid),\n> + oid_to_hex(&odb_parents->item->object.oid));\n> +\n> + graph_parents = graph_parents->next;\n> + odb_parents = odb_parents->next;\n> + }\n> +\n> + if (odb_parents != NULL)\n> + graph_report(\"commit-graph parent list for commit %s terminates early\",\n> + oid_to_hex(&cur_oid));\n> +\n> + if (graph_commit->generation) {\n> + uint32_t max_generation = 0;\n> + graph_parents = graph_commit->parents;\n> +\n> + while (graph_parents) {\n> + if (graph_parents->item->generation == GENERATION_NUMBER_ZERO ||\n> + graph_parents->item->generation == GENERATION_NUMBER_INFINITY)\n> + graph_report(\"commit-graph has valid generation for %s but not its parent, %s\",\n> + oid_to_hex(&cur_oid),\n> + oid_to_hex(&graph_parents->item->object.oid));\n> + if (graph_parents->item->generation > max_generation)\n> + max_generation = graph_parents->item->generation;\n> + graph_parents = graph_parents->next;\n> + }\n> +\n> + if (max_generation == GENERATION_NUMBER_MAX)\n> + max_generation--;\n\nI'm not too familiar with these concepts. Is this a trick in preparation\nfor this:\n\n> +\n> + if (graph_commit->generation != max_generation + 1)\n\nAny way that could give a false negative? (I'm not sure it would matter\nmuch.) Maybe \"if (!MAX && generation != max + 1)\".\n\n> + graph_report(\"commit-graph has incorrect generation for %s\",\n> + oid_to_hex(&cur_oid));\n> + } else {\n> + graph_parents = graph_commit->parents;\n> +\n> + while (graph_parents) {\n> + if (graph_parents->item->generation)\n> + graph_report(\"commit-graph has generation ZERO for %s but not its parent, %s\",\n> + oid_to_hex(&cur_oid),\n> + oid_to_hex(&graph_parents->item->object.oid));\n> + graph_parents = graph_parents->next;\n> + }\n> + }\n> + }\n> +\n> return verify_commit_graph_error;\n> }\n\nAt this point, I should admit that I went through the above thinking\n\"right, makes sense, ok, sure\". I was not really going \"hmm, I wonder\n...\" This looks like the real meat of \"verify\", and I'll try to look it\nover with a fresh pair of eyes tomorrow.\n\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n\n> + corrupt_data $objdir/info/commit-graph 1134 \"\\01\" &&\n\n> + corrupt_data $objdir/info/commit-graph 1312 \"\\01\" &&\n\n> + corrupt_data $objdir/info/commit-graph 1332 \"\\01\" &&\n\n> + corrupt_data $objdir/info/commit-graph 1712 \"\\01\" &&\n\n> + corrupt_data $objdir/info/commit-graph 1340 \"\\01\" &&\n> + corrupt_data $objdir/info/commit-graph 1344 \"\\01\" &&\n\nCould you document these numbers somehow? (Maybe even calculate them\nfrom constant inputs, although that might be a form of premature\noptimization.) When some poor soul has to derive the corresponding\nnumbers for a commit-graph with NewHash, they will thank you.\n\nMartin\n"},{"id":"347598","messageId":"3cd99f4e-82cb-43e8-53eb-e8d38d449e57@gmail.com","threadId":"48311","inReplyTo":"CAN0heSotYYgrwMLT1yyFVk4VEZ4OQKF_Xwm9-wZosJ6_-gAVBg@mail.gmail.com","subject":"Re: [PATCH v2 01/12] commit-graph: add 'verify' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-14T13:27:21Z","receivedAt":"2018-05-14T13:27:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/12/2018 9:31 AM, Martin Ågren wrote:\n> On 11 May 2018 at 23:15, Derrick Stolee wrote:\n>\n>> graph_name = get_commit_graph_filename(opts.obj_dir);\n>> graph = load_commit_graph_one(graph_name);\n>> + FREE_AND_NULL(graph_name);\n>>\n>> if (!graph)\n>> die(\"graph file %s does not exist\", graph_name);\n>> - FREE_AND_NULL(graph_name);\n> This is probably because of something I said, but this does not look\n> correct. The `die()` would typically print \"(null)\" or segfault. If the\n> `die()` means we don't free `graph_name`, that should be fine.\n>\n> You're still leaking `graph` here (possibly nothing this patch should\n> worry about) and in `graph_verify()`. UNLEAK-ing it immediately before\n> calling `verify_commit_graph()` should be ok. I also think punting on\n> this UNLEAK-business entirely would be ok; I was just a bit surprised to\n> see one variable getting freed and the other one ignored.\n\nThanks, Martin. I was just blindly searching for FREE_AND_NULL() and \nshouldn't have been so careless.\n\n-Stolee\n"},{"id":"347600","messageId":"0964380e-1ec9-46c4-ea59-f8890d9ed711@gmail.com","threadId":"48311","inReplyTo":"CAN0heSrKFJbgwXXzUKaJQn24pY43S2bsn9Zx6Yyy6M2X2wDJ7w@mail.gmail.com","subject":"Re: [PATCH v2 02/12] commit-graph: verify file header information","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-14T13:31:28Z","receivedAt":"2018-05-14T13:31:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/12/2018 9:35 AM, Martin Ågren wrote:\n> +static int verify_commit_graph_error;\n> +\n> +static void graph_report(const char *fmt, ...)\n> +{\n> + va_list ap;\n> + struct strbuf sb = STRBUF_INIT;\n> + verify_commit_graph_error = 1;\n> +\n> + va_start(ap, fmt);\n> + strbuf_vaddf(&sb, fmt, ap);\n> +\n> + fprintf(stderr, \"%s\\n\", sb.buf);\n> + strbuf_release(&sb);\n> + va_end(ap);\n> +}\n\nThat's a good idea. Makes that patch a bit less trivial and this one a \nbit less difficult.\n"},{"id":"347601","messageId":"a7637e48-b7a7-54fe-a085-b98f84df106e@gmail.com","threadId":"48311","inReplyTo":"CAN0heSo2sTgVSoxUiugSaEjev+0eyzCRA5ZmQVr6FRtKkyGJsQ@mail.gmail.com","subject":"Re: [PATCH v2 08/12] commit-graph: verify commit contents against odb","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-14T13:44:03Z","receivedAt":"2018-05-14T13:44:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/12/2018 5:17 PM, Martin Ågren wrote:\n> On 11 May 2018 at 23:15, Derrick Stolee wrote:\n>> When running 'git commit-graph verify', compare the contents of the\n>> commits that are loaded from the commit-graph file with commits that are\n>> loaded directly from the object database. This includes checking the\n>> root tree object ID, commit date, and parents.\n>>\n>> Parse the commit from the graph during the initial loop through the\n>> object IDs to guarantee we parse from the commit-graph file.\n>>\n>> In addition, verify the generation number calculation is correct for all\n>> commits in the commit-graph file.\n>>\n>> While testing, we discovered that mutating the integer value for a\n>> parent to be outside the accepted range causes a segmentation fault. Add\n>> a new check in insert_parent_or_die() that prevents this fault. Check\n>> for that error during the test, both in the typical parents and in the\n>> list of parents for octopus merges.\n> This paragraph and the corresponding fix and test feel like a separate\n> patch to me. (The commit message of it could be \"To test the next patch,\n> we threw invalid data at `git commit-graph verify, and it crashed in\n> pre-existing code, so let's fix that first\" -- there is definitely a\n> connection.) Is this important enough to fast-track to master in time\n> for 2.18? My guess would be no.\n>\n>> +\n>> + if (pos >= g->num_commits)\n>> + die(\"invalide parent position %\"PRIu64, pos);\n> s/invalide/invalid/\n>\n>> @@ -875,6 +879,8 @@ int verify_commit_graph(struct commit_graph *g)\n>> return 1;\n>>\n>> for (i = 0; i < g->num_commits; i++) {\n>> + struct commit *graph_commit;\n>> +\n>> hashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n>>\n>> if (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n>> @@ -892,6 +898,10 @@ int verify_commit_graph(struct commit_graph *g)\n>>\n>> cur_fanout_pos++;\n>> }\n>> +\n>> + graph_commit = lookup_commit(&cur_oid);\n>> + if (!parse_commit_in_graph_one(g, graph_commit))\n>> + graph_report(\"failed to parse %s from commit-graph\", oid_to_hex(&cur_oid));\n>> }\n> Could this end up giving ridiculous amounts of output? It would depend\n> on the input, I guess.\n>\n>> while (cur_fanout_pos < 256) {\n>> @@ -904,5 +914,95 @@ int verify_commit_graph(struct commit_graph *g)\n>> cur_fanout_pos++;\n>> }\n>>\n>> + if (verify_commit_graph_error)\n>> + return 1;\n> Well, here we give up before running into *too* much problem.\n>\n>> + for (i = 0; i < g->num_commits; i++) {\n>> + struct commit *graph_commit, *odb_commit;\n>> + struct commit_list *graph_parents, *odb_parents;\n>> + int num_parents = 0;\n>> +\n>> + hashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n>> +\n>> + graph_commit = lookup_commit(&cur_oid);\n>> + odb_commit = (struct commit *)create_object(cur_oid.hash, alloc_commit_node());\n>> + if (parse_commit_internal(odb_commit, 0, 0)) {\n>> + graph_report(\"failed to parse %s from object database\", oid_to_hex(&cur_oid));\n>> + continue;\n>> + }\n>> +\n>> + if (oidcmp(&get_commit_tree_in_graph_one(g, graph_commit)->object.oid,\n>> + get_commit_tree_oid(odb_commit)))\n>> + graph_report(\"root tree object ID for commit %s in commit-graph is %s != %s\",\n>> + oid_to_hex(&cur_oid),\n>> + oid_to_hex(get_commit_tree_oid(graph_commit)),\n>> + oid_to_hex(get_commit_tree_oid(odb_commit)));\n>> +\n>> + if (graph_commit->date != odb_commit->date)\n>> + graph_report(\"commit date for commit %s in commit-graph is %\"PRItime\" != %\"PRItime\"\",\n>> + oid_to_hex(&cur_oid),\n>> + graph_commit->date,\n>> + odb_commit->date);\n>> +\n>> +\n>> + graph_parents = graph_commit->parents;\n>> + odb_parents = odb_commit->parents;\n>> +\n>> + while (graph_parents) {\n>> + num_parents++;\n>> +\n>> + if (odb_parents == NULL)\n>> + graph_report(\"commit-graph parent list for commit %s is too long (%d)\",\n>> + oid_to_hex(&cur_oid),\n>> + num_parents);\n>> +\n>> + if (oidcmp(&graph_parents->item->object.oid, &odb_parents->item->object.oid))\n>> + graph_report(\"commit-graph parent for %s is %s != %s\",\n>> + oid_to_hex(&cur_oid),\n>> + oid_to_hex(&graph_parents->item->object.oid),\n>> + oid_to_hex(&odb_parents->item->object.oid));\n>> +\n>> + graph_parents = graph_parents->next;\n>> + odb_parents = odb_parents->next;\n>> + }\n>> +\n>> + if (odb_parents != NULL)\n>> + graph_report(\"commit-graph parent list for commit %s terminates early\",\n>> + oid_to_hex(&cur_oid));\n>> +\n>> + if (graph_commit->generation) {\n>> + uint32_t max_generation = 0;\n>> + graph_parents = graph_commit->parents;\n>> +\n>> + while (graph_parents) {\n>> + if (graph_parents->item->generation == GENERATION_NUMBER_ZERO ||\n>> + graph_parents->item->generation == GENERATION_NUMBER_INFINITY)\n>> + graph_report(\"commit-graph has valid generation for %s but not its parent, %s\",\n>> + oid_to_hex(&cur_oid),\n>> + oid_to_hex(&graph_parents->item->object.oid));\n>> + if (graph_parents->item->generation > max_generation)\n>> + max_generation = graph_parents->item->generation;\n>> + graph_parents = graph_parents->next;\n>> + }\n>> +\n>> + if (max_generation == GENERATION_NUMBER_MAX)\n>> + max_generation--;\n> I'm not too familiar with these concepts. Is this a trick in preparation\n> for this:\n>\n>> +\n>> + if (graph_commit->generation != max_generation + 1)\n> Any way that could give a false negative? (I'm not sure it would matter\n> much.) Maybe \"if (!MAX && generation != max + 1)\".\n\nYou're right that this is confusing. Seems worth a comment.\n\nWhen we have a commit-graph with computed generation numbers, the \ngeneration number for a commit is exactly one more than the maximum \ngeneration number of its parents EXCEPT in the case that we would have a \ngeneration number too large to store in the commit-graph. In this case \n(that is not currently possible with any repo in existence) we \"squash\" \nthe generation number at GENERATION_NUMBER_MAX, so we have equality \nbetween the generation at the commit and the generation of a parent.\n\n>\n>> + graph_report(\"commit-graph has incorrect generation for %s\",\n>> + oid_to_hex(&cur_oid));\n>> + } else {\n>> + graph_parents = graph_commit->parents;\n>> +\n>> + while (graph_parents) {\n>> + if (graph_parents->item->generation)\n>> + graph_report(\"commit-graph has generation ZERO for %s but not its parent, %s\",\n>> + oid_to_hex(&cur_oid),\n>> + oid_to_hex(&graph_parents->item->object.oid));\n>> + graph_parents = graph_parents->next;\n>> + }\n>> + }\n>> + }\n>> +\n>> return verify_commit_graph_error;\n>> }\n> At this point, I should admit that I went through the above thinking\n> \"right, makes sense, ok, sure\". I was not really going \"hmm, I wonder\n> ...\" This looks like the real meat of \"verify\", and I'll try to look it\n> over with a fresh pair of eyes tomorrow.\n\nI appreciate the level of inspection you are giving this series!\n\n>\n>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n>> + corrupt_data $objdir/info/commit-graph 1134 \"\\01\" &&\n>> + corrupt_data $objdir/info/commit-graph 1312 \"\\01\" &&\n>> + corrupt_data $objdir/info/commit-graph 1332 \"\\01\" &&\n>> + corrupt_data $objdir/info/commit-graph 1712 \"\\01\" &&\n>> + corrupt_data $objdir/info/commit-graph 1340 \"\\01\" &&\n>> + corrupt_data $objdir/info/commit-graph 1344 \"\\01\" &&\n> Could you document these numbers somehow? (Maybe even calculate them\n> from constant inputs, although that might be a form of premature\n> optimization.) When some poor soul has to derive the corresponding\n> numbers for a commit-graph with NewHash, they will thank you.\n\nYeah, this is a bit of a mess. The arithmetic to get these numbers is \nnot hard, so I could do that math in the script (at run time instead of \nat dev time).\n\nI also realize that the way I write the commit-graph to be \"fixed\" \nbefore these tests is not so fixed: if a new ref is added, then the \ntests break. I'll update the commit-graph to be reachable from commits/8 \nso we have a more concrete example:\n\ngit rev-parse commits/8 | git commit-graph write --stdin-commits\n\n(This already changes the written commit-graph because of the 'git pull' \ntest, so it will \"help\" me be sure my math is correct when recomputing \nthe new offsets for the corruption tests.)\n\nThanks,\n-Stolee\n"},{"id":"347720","messageId":"CAN0heSrE39eXQ0A4tncmKvi36Q9yZmR9TuiDJdRW7F1Q4hHquQ@mail.gmail.com","threadId":"48311","inReplyTo":"20180511211504.79877-9-dstolee@microsoft.com","subject":"Re: [PATCH v2 08/12] commit-graph: verify commit contents against odb","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2018-05-15T21:12:27Z","receivedAt":"2018-05-15T21:12:31Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 11 May 2018 at 23:15, Derrick Stolee wrote:\n\nI finally sat down today and familiarized myself with the commit-graph\ncode a little. My biggest surprise was when I noticed that there is a\nhash checksum at the end of the commit-graph-file. That in combination\nwith the tests where you flip some bytes...\n\nIt turns out, if my reading is right, that the hash value is written as\nthe commit-graph is generated, but that it is not verified as the\ncommit-graph is later read. I could not find any mention of your plans\nhere -- I understand why you would not want to verify the hash in\n`load_commit_graph_one()`, at least not in every run. Anyway, this is\njust an observation. Verifying the hash would affect the tests this\nseries adds. They might need to rewrite the hash or set some magic\nenvironment variable. :-/ But that's for another day.\n\n> + for (i = 0; i < g->num_commits; i++) {\n> + struct commit *graph_commit, *odb_commit;\n> + struct commit_list *graph_parents, *odb_parents;\n> + int num_parents = 0;\n> + hashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n\n`num_commits` was derived as the commit-graph was loaded. It was derived\nfrom offsets which were verified to be in the mmap-ed memory. So this\nsource address is guaranteed to be so, as well. Ok.\n\n(Once brian's latest series hits master, this could use `oidread(...)`.)\n\n> + graph_commit = lookup_commit(&cur_oid);\n\nDo we know this comes from the graph? Even with a more-or-less-messed-up\ncommit graph? See below.\n\n> + odb_commit = (struct commit *)create_object(cur_oid.hash, alloc_commit_node());\n> + if (parse_commit_internal(odb_commit, 0, 0)) {\n> + graph_report(\"failed to parse %s from object database\", oid_to_hex(&cur_oid));\n> + continue;\n> + }\n> +\n> + if (oidcmp(&get_commit_tree_in_graph_one(g, graph_commit)->object.oid,\n> + get_commit_tree_oid(odb_commit)))\n\n`get_commit_tree_in_graph_one()` will BUG rather than return NULL. So\nthis will not dereference NULL. Good. But might it hit the BUG? That is,\ncan we trust the commit coming out of `lookup_commit()` not to have\n`graph_pos == COMMIT_NOT_FROM_GRAPH`?\n\n> + graph_report(\"root tree object ID for commit %s in commit-graph is %s != %s\",\n> + oid_to_hex(&cur_oid),\n> + oid_to_hex(get_commit_tree_oid(graph_commit)),\n> + oid_to_hex(get_commit_tree_oid(odb_commit)));\n> +\n> + if (graph_commit->date != odb_commit->date)\n> + graph_report(\"commit date for commit %s in commit-graph is %\"PRItime\" != %\"PRItime\"\",\n> + oid_to_hex(&cur_oid),\n> + graph_commit->date,\n> + odb_commit->date);\n> +\n> +\n\n(Extra blank line?)\n\n> + graph_parents = graph_commit->parents;\n> + odb_parents = odb_commit->parents;\n> +\n> + while (graph_parents) {\n> + num_parents++;\n> +\n> + if (odb_parents == NULL)\n> + graph_report(\"commit-graph parent list for commit %s is too long (%d)\",\n> + oid_to_hex(&cur_oid),\n> + num_parents);\n> +\n> + if (oidcmp(&graph_parents->item->object.oid, &odb_parents->item->object.oid))\n> + graph_report(\"commit-graph parent for %s is %s != %s\",\n> + oid_to_hex(&cur_oid),\n> + oid_to_hex(&graph_parents->item->object.oid),\n> + oid_to_hex(&odb_parents->item->object.oid));\n> +\n> + graph_parents = graph_parents->next;\n> + odb_parents = odb_parents->next;\n> + }\n> +\n> + if (odb_parents != NULL)\n> + graph_report(\"commit-graph parent list for commit %s terminates early\",\n> + oid_to_hex(&cur_oid));\n\nOk, ensure the lists are equally long and compare the entries.\n\n> +\n> + if (graph_commit->generation) {\n\nIf the commit has a generation number (not an old commit graph)...\n\n> + uint32_t max_generation = 0;\n> + graph_parents = graph_commit->parents;\n> +\n> + while (graph_parents) {\n> + if (graph_parents->item->generation == GENERATION_NUMBER_ZERO ||\n> + graph_parents->item->generation == GENERATION_NUMBER_INFINITY)\n> + graph_report(\"commit-graph has valid generation for %s but not its parent, %s\",\n> + oid_to_hex(&cur_oid),\n> + oid_to_hex(&graph_parents->item->object.oid));\n\n... then it's odd if a parent has no generation number or is not in the\ngraph file at all.\n\n> + if (graph_parents->item->generation > max_generation)\n> + max_generation = graph_parents->item->generation;\n> + graph_parents = graph_parents->next;\n> + }\n> +\n> + if (max_generation == GENERATION_NUMBER_MAX)\n> + max_generation--;\n> +\n> + if (graph_commit->generation != max_generation + 1)\n> + graph_report(\"commit-graph has incorrect generation for %s\",\n> + oid_to_hex(&cur_oid));\n\nOk. Thanks for considering adding a comment.\n\n> + } else {\n> + graph_parents = graph_commit->parents;\n> +\n> + while (graph_parents) {\n> + if (graph_parents->item->generation)\n> + graph_report(\"commit-graph has generation ZERO for %s but not its parent, %s\",\n> + oid_to_hex(&cur_oid),\n> + oid_to_hex(&graph_parents->item->object.oid));\n> + graph_parents = graph_parents->next;\n> + }\n\nRight.\n\n> + }\n> + }\n> +\n> return verify_commit_graph_error;\n> }\n\nNice. My question about `get_commit_tree_in_graph_one()` might be me\nmaking things much harder than they are. I sort of suspect that your\nusage will be quite obviously safe in retrospect.\n\nMartin\n"},{"id":"347914","messageId":"CAN0heSrkRTfqhbSYgBcJPh-9XjUzmKmjyfbbFQVHRSZB7sDNyQ@mail.gmail.com","threadId":"48311","inReplyTo":"20180511211504.79877-10-dstolee@microsoft.com","subject":"Re: [PATCH v2 09/12] fsck: verify commit-graph","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2018-05-17T18:13:00Z","receivedAt":"2018-05-17T18:13:05Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 11 May 2018 at 23:15, Derrick Stolee wrote:\n> If core.commitGraph is true, verify the contents of the commit-graph\n> during 'git fsck' using the 'git commit-graph verify' subcommand. Run\n> this check on all alternates, as well.\n\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 5ab268a024..91c8406d97 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -205,6 +205,16 @@ test_expect_success 'build graph from commits with append' '\n> graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n> graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n>\n> +test_expect_success 'build graph using --reachable' '\n> + cd \"$TRASH_DIRECTORY/full\" &&\n> + git commit-graph write --reachable &&\n> + test_path_is_file $objdir/info/commit-graph &&\n> + graph_read_expect \"11\" \"large_edges\"\n> +'\n\nThis should be in the next patch.\n\n> +graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n> +graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n\n(Possibly the same here.)\n\n> test_expect_success 'setup bare repo' '\n> cd \"$TRASH_DIRECTORY\" &&\n> git clone --bare --no-local full bare &&\n> @@ -335,7 +345,7 @@ test_expect_success 'detect OID not in object database' '\n> cd \"$TRASH_DIRECTORY/full\" &&\n> cp $objdir/info/commit-graph commit-graph-backup &&\n> test_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n> - corrupt_data $objdir/info/commit-graph 1134 \"\\01\" &&\n> + corrupt_data $objdir/info/commit-graph 1134 \"\\00\" &&\n\nThis and two similar ones as well, I guess.\n\nActually, I can drop them altogether and the tests still pass. Rebase\nmishap?\n\n> +test_expect_success 'git fsck (checks commit-graph)' '\n> + cd \"$TRASH_DIRECTORY/full\" &&\n> + git fsck\n> +'\n\nMaybe inject an error and verify that `git fsck` does indeed catch it,\ni.e., it does call out to check the commit-graph.\n\nMaybe also a run with `-c core.commitGraph=no` where the error should\nnot be found because the commit-graph should not be checked?\n\nMartin\n"},{"id":"347915","messageId":"CAN0heSodgrk-yUaJVgNQ=qjj1nExe0o+55wAEvGoa9rf-9NKmw@mail.gmail.com","threadId":"48311","inReplyTo":"20180511211504.79877-11-dstolee@microsoft.com","subject":"Re: [PATCH v2 10/12] commit-graph: add '--reachable' option","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2018-05-17T18:16:22Z","receivedAt":"2018-05-17T18:16:26Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 11 May 2018 at 23:15, Derrick Stolee wrote:\n> When writing commit-graph files, it can be convenient to ask for all\n> reachable commits (starting at the ref set) in the resulting file. This\n> is particularly helpful when writing to stdin is complicated, such as a\n> future integration with 'git gc' which will call\n> 'git commit-graph write --reachable' after performing cleanup of the\n> object database.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-commit-graph.txt | 8 ++++++--\n> builtin/commit-graph.c | 41 ++++++++++++++++++++++++++++++++++----\n> 2 files changed, 43 insertions(+), 6 deletions(-)\n>\n> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> index a222cfab08..cc1715a823 100644\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -38,12 +38,16 @@ Write a commit graph file based on the commits found in packfiles.\n> +\n> With the `--stdin-packs` option, generate the new commit graph by\n> walking objects only in the specified pack-indexes. (Cannot be combined\n> -with --stdin-commits.)\n> +with --stdin-commits or --reachable.)\n\nYou could enclose --reachable in `...` for nicer rendering and fix\n--stdin-commits as well while you're here.\n\n> With the `--stdin-commits` option, generate the new commit graph by\n> walking commits starting at the commits specified in stdin as a list\n> of OIDs in hex, one OID per line. (Cannot be combined with\n> ---stdin-packs.)\n> +--stdin-packs or --reachable.)\n\nDitto.\n\n> +With the `--reachable` option, generate the new commit graph by walking\n> +commits starting at all refs. (Cannot be combined with --stdin-commits\n> +or --stind-packs.)\n\nDitto. Also, s/stind/stdin/.\n\nMartin\n"},{"id":"347916","messageId":"CAN0heSr+Aeu1By2LXxtmQN=Lq3RYwuHgYv36GjRN2Cfg73LA4Q@mail.gmail.com","threadId":"48311","inReplyTo":"20180511211504.79877-12-dstolee@microsoft.com","subject":"Re: [PATCH v2 11/12] gc: automatically write commit-graph files","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2018-05-17T18:20:57Z","receivedAt":"2018-05-17T18:21:01Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 11 May 2018 at 23:15, Derrick Stolee wrote:\n> The commit-graph file is a very helpful feature for speeding up git\n> operations. In order to make it more useful, write the commit-graph file\n> by default during standard garbage collection operations.\n\nSo does it really write by default...\n\n> Add a 'gc.commitGraph' config setting that triggers writing a\n> commit-graph file after any non-trivial 'git gc' command. Defaults to\n> false while the commit-graph feature matures. We specifically do not\n\nor not...? I guess the first paragraph has simply been there since\nbefore you changed your mind about the default?\n\n> want to turn this on by default until the commit-graph feature is fully\n> integrated with history-modifying features like shallow clones.\n\nSo if someone would turn this on with a shallow clone, ... Do we want\nsome note (warning?) around that in the user documentation?\n\nMartin\n"},{"id":"348095","messageId":"86d0xqeddq.fsf@gmail.com","threadId":"48311","inReplyTo":"20180511211504.79877-2-dstolee@microsoft.com","subject":"Re: [PATCH v2 01/12] commit-graph: add 'verify' subcommand","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-05-20T12:10:41Z","receivedAt":"2018-05-20T12:10:50Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> If the commit-graph file becomes corrupt, we need a way to verify\n> that its contents match the object database. In the manner of\n> 'git fsck' we will implement a 'git commit-graph verify' subcommand\n> to report all issues with the file.\n>\n> Add the 'verify' subcommand to the 'commit-graph' builtin and its\n> documentation. The subcommand is currently a no-op except for\n> loading the commit-graph into memory, which may trigger run-time\n> errors that would be caught by normal use. Add a simple test that\n> ensures the command returns a zero error code.\n>\n> If no commit-graph file exists, this is an acceptable state. Do\n> not report any errors.\n\nAll right. Nice introductory patch.\n\n>\n> During review, we noticed that a FREE_AND_NULL(graph_name) was\n> placed after a possible 'return', and this pattern was also in\n> graph_read(). Fix that case, too.\n\nThis should probably be a separate [micro-]patch. Especially as Martin\nÅgren noticed it is not correct...\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-commit-graph.txt | 6 ++++++\n> builtin/commit-graph.c | 40 +++++++++++++++++++++++++++++++++++++-\n> commit-graph.c | 5 +++++\n> commit-graph.h | 2 ++\n> t/t5318-commit-graph.sh | 10 ++++++++++\n> 5 files changed, 62 insertions(+), 1 deletion(-)\n>\n> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> index 4c97b555cc..a222cfab08 100644\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -10,6 +10,7 @@ SYNOPSIS\n> --------\n> [verse]\n> 'git commit-graph read' [--object-dir ]\n> +'git commit-graph verify' [--object-dir ]\n> 'git commit-graph write' [--object-dir ]\n> \n> \n> @@ -52,6 +53,11 @@ existing commit-graph file.\n> Read a graph file given by the commit-graph file and output basic\n> details about the graph file. Used for debugging purposes.\n> \n> +'verify'::\n> +\n> +Read the commit-graph file and verify its contents against the object\n> +database. Used to check for corrupted data.\n\nI wonder if it would be useful to have an option to verify commit-graph\nfile without accessing the object database, checking just that it is\nwell formed.\n\nAnyway, it could be added later, if needed.\n\n> +\n> \n> EXAMPLES\n> --------\n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index 37420ae0fd..af3101291f 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -8,10 +8,16 @@\n> static char const * const builtin_commit_graph_usage[] = {\n> \tN_(\"git commit-graph [--object-dir ]\"),\n> \tN_(\"git commit-graph read [--object-dir ]\"),\n> +\tN_(\"git commit-graph verify [--object-dir ]\"),\n> \tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n> \tNULL\n> };\n> \n> +static const char * const builtin_commit_graph_verify_usage[] = {\n> +\tN_(\"git commit-graph verify [--object-dir ]\"),\n> +\tNULL\n> +};\n> +\n> static const char * const builtin_commit_graph_read_usage[] = {\n> \tN_(\"git commit-graph read [--object-dir ]\"),\n> \tNULL\n> @@ -29,6 +35,36 @@ static struct opts_commit_graph {\n> \tint append;\n> } opts;\n> \n> +\n> +static int graph_verify(int argc, const char **argv)\n\nA reminder for myself: exit code 0 means no errors.\n\n> +{\n> +\tstruct commit_graph *graph = 0;\n> +\tchar *graph_name;\n> +\n> +\tstatic struct option builtin_commit_graph_verify_options[] = {\n> +\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n> +\t\t\t N_(\"dir\"),\n> +\t\t\t N_(\"The object directory to store the graph\")),\n> +\t\tOPT_END(),\n> +\t};\n> +\n> +\targc = parse_options(argc, argv, NULL,\n> +\t\t\t builtin_commit_graph_verify_options,\n> +\t\t\t builtin_commit_graph_verify_usage, 0);\n> +\n> +\tif (!opts.obj_dir)\n> +\t\topts.obj_dir = get_object_directory();\n\nAll right, simple handling of a subcommand and its options.\n\n\nI still think that '--object-dir=' should be a git wrapper option,\nlike '--git-dir=' and '--work-tree=' (and\n'--namespace=') are. It would be command-line option equivalent\nto the GIT_OBJECT_DIRECTORY environment variable, just like\n--git-dir= is for GIT_DIR, and --work-tree= is for\nGIT_WORK_TREE, etc.\n\nThis way the code would be implemented once for all commands, and there\nwould be no duplicated code for each git-commit-graph subcommand.\n\nBut that may be a matter of a separate patch.\n\n> +\n> +\tgraph_name = get_commit_graph_filename(opts.obj_dir);\n\nThis returns full path of commit-graph file, allocating it.\n\n> +\tgraph = load_commit_graph_one(graph_name);\n\nThis reads the file (no checking if core.commitGraph is set, no handling\nof alternatives), and verifies that:\n - file is not too small, i.e. smaller than GRAPH_MIN_SIZE\n - it has correct signature\n - it has correct graph version\n - it has correct hash version\n - chunk [offsets] fit within file\n - that OID Fanout, OID Lookup, Commit Data and Large Edges chunks are\n not repeated, though not that all required chunks are present\n\n> +\tFREE_AND_NULL(graph_name);\n\nAll right, graph_name is not used further, so we free it and set it to\nNULL.\n\nNote however that if load_commit_graph_one() finds errors, it would exit\nwith error code 1... but then exiting the process would free its\nresources anyway.\n\n> +\n> +\tif (!graph)\n> +\t\treturn 0;\n\nThe only case where load_commit_graph_one() returns NULL instead of\nexiting is if the file cannot be opened for reading (e.g. it does not\nexists, or cannot be opened for reading), or its status cannot be read.\n\nSide question: this is nice defensive programming, but could it happen\nthat file can be opened for reading, but its status cannot be read?\n\n> +\n> +\treturn verify_commit_graph(graph);\n\nHere we leak graph->fd (it is neither unmapped, nor closed) but that may\nnot matter as we are exiting anyway. Well, at least until it gets\nlibified, then maybe.\n\n> +}\n> +\n> static int graph_read(int argc, const char **argv)\n> {\n> \tstruct commit_graph *graph = NULL;\n> @@ -50,10 +86,10 @@ static int graph_read(int argc, const char **argv)\n> \n> \tgraph_name = get_commit_graph_filename(opts.obj_dir);\n> \tgraph = load_commit_graph_one(graph_name);\n> +\tFREE_AND_NULL(graph_name);\n> \n> \tif (!graph)\n> \t\tdie(\"graph file %s does not exist\", graph_name);\n> -\tFREE_AND_NULL(graph_name);\n\nHere we use graph_name, which has been just freed and set to NULL.\n\nThis should probably be (but I may be wrong):\n\n\n\tif (!graph) {\n\t\tUNLEAK(graph_name);\n\t\tdie(\"graph file %s does not exist\", graph_name);\n\t}\n\tFREE_AND_NULL(graph_name);\n\n\n> \n> \tprintf(\"header: %08x %d %d %d %d\\n\",\n> \t\tntohl(*(uint32_t*)graph->data),\n> @@ -160,6 +196,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n> \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n> \n> \tif (argc > 0) {\n> +\t\tif (!strcmp(argv[0], \"verify\"))\n> +\t\t\treturn graph_verify(argc, argv);\n> \t\tif (!strcmp(argv[0], \"read\"))\n> \t\t\treturn graph_read(argc, argv);\n> \t\tif (!strcmp(argv[0], \"write\"))\n\nAll right, straightforward adding of a new subcommand.\n\n> diff --git a/commit-graph.c b/commit-graph.c\n> index a8c337dd77..b25aaed128 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -817,3 +817,8 @@ void write_commit_graph(const char *obj_dir,\n> \toids.alloc = 0;\n> \toids.nr = 0;\n> }\n> +\n> +int verify_commit_graph(struct commit_graph *g)\n> +{\n> +\treturn !g;\n> +}\n\nAll right, nice placeholder.\n\n> diff --git a/commit-graph.h b/commit-graph.h\n> index 96cccb10f3..71a39c5a57 100644\n> --- a/commit-graph.h\n> +++ b/commit-graph.h\n> @@ -53,4 +53,6 @@ void write_commit_graph(const char *obj_dir,\n> \t\t\tint nr_commits,\n> \t\t\tint append);\n> \n> +int verify_commit_graph(struct commit_graph *g);\n> +\n> #endif\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 77d85aefe7..6ca451dfd2 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -11,6 +11,11 @@ test_expect_success 'setup full repo' '\n> \tobjdir=\".git/objects\"\n> '\n> \n> +test_expect_success 'verify graph with no graph file' '\n> +\tcd \"$TRASH_DIRECTORY/full\" &&\n> +\tgit commit-graph verify\n> +'\n\nNice to have this test here, but as it is now it is not an independent\ntest. It depends on the fact that other earlier tests did not generate\ngraph file.\n\nPerhaps it would be better to have a separate directory with repository\nwithout commit graph file, and separate directory with repository with\ncommit graph file. Or rename commit graph file if it exists, renaming\nit back after tests finishes. Or something like that.\n\n> +\n> test_expect_success 'write graph with no packs' '\n> \tcd \"$TRASH_DIRECTORY/full\" &&\n> \tgit commit-graph write --object-dir . &&\n> @@ -230,4 +235,9 @@ test_expect_success 'perform fast-forward merge in full repo' '\n> \ttest_cmp expect output\n> '\n> \n> +test_expect_success 'git commit-graph verify' '\n> +\tcd \"$TRASH_DIRECTORY/full\" &&\n> +\tgit commit-graph verify >output\n> +'\n\nThis is amost the same tests as the one added earlier, the only\ndifference is its place in t/t5318-commit-graph.sh. This test is not\nindependent.\n\nThough I'm not sure if that would matter much.\n\n> +\n> test_done\n"},{"id":"348130","messageId":"867enydrmz.fsf@gmail.com","threadId":"48311","inReplyTo":"20180511211504.79877-3-dstolee@microsoft.com","subject":"Re: [PATCH v2 02/12] commit-graph: verify file header information","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-05-20T20:00:20Z","receivedAt":"2018-05-20T20:00:28Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> During a run of 'git commit-graph verify', list the issues with the\n> header information in the commit-graph file. Some of this information\n> is inferred from the loaded 'struct commit_graph'. Some header\n> information is checked as part of load_commit_graph_one().\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 32 +++++++++++++++++++++++++++++++-\n> 1 file changed, 31 insertions(+), 1 deletion(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index b25aaed128..d2db20e49a 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -818,7 +818,37 @@ void write_commit_graph(const char *obj_dir,\n> \toids.nr = 0;\n> }\n> \n> +static int verify_commit_graph_error;\n> +\n> +static void graph_report(const char *fmt, ...)\n> +{\n> +\tva_list ap;\n> +\tstruct strbuf sb = STRBUF_INIT;\n> +\tverify_commit_graph_error = 1;\n> +\n> +\tva_start(ap, fmt);\n> +\tstrbuf_vaddf(&sb, fmt, ap);\n> +\n> +\tfprintf(stderr, \"%s\\n\", sb.buf);\n> +\tstrbuf_release(&sb);\n> +\tva_end(ap);\n> +}\n> +\n> int verify_commit_graph(struct commit_graph *g)\n> {\n> -\treturn !g;\n> +\tif (!g) {\n> +\t\tgraph_report(\"no commit-graph file loaded\");\n> +\t\treturn 1;\n> +\t}\n\nI won't be repeating what Martin said, but I agree with it. Well, that\nor make it a separate patch.\n\n> +\n> +\tverify_commit_graph_error = 0;\n> +\n\nA quick reminder for myself. The load_commit_graph_one() that is used\nto fill the commit_graph parameter alreaady verifies that:\n - file is not too small, i.e. smaller than GRAPH_MIN_SIZE\n - it has correct signature\n - it has correct graph version\n - it has correct hash version\n - chunks [offsets] all fit within file\n - that OID Fanout, OID Lookup, Commit Data and Large Edges chunks are\n not repeated, though not that all required chunks are present\n\n> +\tif (!g->chunk_oid_fanout)\n> +\t\tgraph_report(\"commit-graph is missing the OID Fanout chunk\");\n> +\tif (!g->chunk_oid_lookup)\n> +\t\tgraph_report(\"commit-graph is missing the OID Lookup chunk\");\n> +\tif (!g->chunk_commit_data)\n> +\t\tgraph_report(\"commit-graph is missing the Commit Data chunk\");\n\nThis one checks that all chunks that needs to be present are present.\nNice.\n\nThere are a few more things that we can check about CHUNK LOOKUP part.\nFor example we would detect if file was truncated because the offset of\nsome chunk would be pointing outside the file... unless the truncation\nfalls within the last chunk. We don't check that terminating label\n(chunk \"\\0\\0\\0\\0\" offset) is outside file, I think.\n\nWe also don't check that positions of subsequent chunks are sorted\n(increasing offsets), so that each chunk length is positive.\n\n\nI also wonder if we shouldn't at least _warn_ about unknown chunks.\n\n> +\n> +\treturn verify_commit_graph_error;\n> }\n\nNice trick to be able to check as much as possible without segfaulting,\nwhile still returning correct error result.\n\n\nTesting newly intruduced functionality would be hard, unless relying on\nhand-crafted files, or on some helper to produce invalid commit-graph\nfiles.\n\n-- \nJakub Narębski\n"},{"id":"348240","messageId":"86wovwdemu.fsf@gmail.com","threadId":"48311","inReplyTo":"20180511211504.79877-4-dstolee@microsoft.com","subject":"Re: [PATCH v2 03/12] commit-graph: test that 'verify' finds corruption","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-05-21T18:53:29Z","receivedAt":"2018-05-21T18:53:37Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> Add test cases to t5318-commit-graph.sh that corrupt the commit-graph\n> file and check that the 'git commit-graph verify' command fails. These\n> tests verify the header and chunk information is checked carefully.\n>\n> Helped-by: Martin Ågren \n> Signed-off-by: Derrick Stolee \n> ---\n> t/t5318-commit-graph.sh | 53 +++++++++++++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 53 insertions(+)\n>\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 6ca451dfd2..0cb88232fa 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -240,4 +240,57 @@ test_expect_success 'git commit-graph verify' '\n> \tgit commit-graph verify >output\n> '\n> \n> +# usage: corrupt_data []\n> +corrupt_data() {\n> +\tfile=$1\n> +\tpos=$2\n> +\tdata=\"${3:-\\0}\"\n> +\tprintf \"$data\" | dd of=\"$file\" bs=1 seek=\"$pos\" conv=notrunc\n> +}\n\nFirst, if we do this that way (and not by adding a test helper), the use\nof this function should be, I think, protected using appropriate test\nprerequisite. Not everyone has 'dd' tool installed, for example on\nMS Windows.\n\nSecond, the commit-graph file format has H-byte HASH-checksum of all of\nthe contents excluding checksum trailer. It feels like any corruption\nshould have been caught by checksum test; thus to actually test that\ncontents is verified we should adjust checksum too, e.g. with sha1sum if\navailable or with test helper... oh, actually we have t/helper/test-sha1.\nUnfortulately, it looks like it has no docs (beside commit message).\n\n> +\n> +test_expect_success 'detect bad signature' '\n> +\tcd \"$TRASH_DIRECTORY/full\" &&\n\nThis 'cd' outside subshell and withou accompanying change back feels a\nbit strange to me.\n\n> +\tcp $objdir/info/commit-graph commit-graph-backup &&\n> +\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n> +\tcorrupt_data $objdir/info/commit-graph 0 \"\\0\" &&\n\nSo 'CGPH' signature is currupted into '\\0GPH'.\n\n> +\ttest_must_fail git commit-graph verify 2>err &&\n> +\tgrep -v \"^\\+\" err > verify-errors &&\n\nMinor nit: redirection should be cuddled to the file, i.e.:\n\n +\tgrep -v \"^\\+\" err >verify-errors &&\n\nA question: why do you filter-out lines starting with \"+\" here?\n\n> +\ttest_line_count = 1 verify-errors &&\n> +\tgrep \"graph signature\" verify-errors\n\nIf messages from 'git commit-graph verify' can be localized (are\ntranslatable), then it should be i18n_grep, isn't it?\n\n> +'\n> +\n> +test_expect_success 'detect bad version number' '\n> +\tcd \"$TRASH_DIRECTORY/full\" &&\n> +\tcp $objdir/info/commit-graph commit-graph-backup &&\n> +\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n> +\tcorrupt_data $objdir/info/commit-graph 4 \"\\02\" &&\n\nAll right, so we replace commit-graph format version 1 (\"\\01\") with\nversion 2 (\"\\02\"). First, why 2 and not 0? Second, is \"\\02\" portable?\n\n> +\ttest_must_fail git commit-graph verify 2>err &&\n> +\tgrep -v \"^\\+\" err > verify-errors &&\n> +\ttest_line_count = 1 verify-errors &&\n\nThe above three lines is common across all test cases; I wonder if it\nwould be possible to extract it into function, to avoid code\nduplication.\n\n> +\tgrep \"graph version\" verify-errors\n> +'\n> +\n> +test_expect_success 'detect bad hash version' '\n> +\tcd \"$TRASH_DIRECTORY/full\" &&\n> +\tcp $objdir/info/commit-graph commit-graph-backup &&\n> +\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n> +\tcorrupt_data $objdir/info/commit-graph 5 \"\\02\" &&\n\nAll right, so we change / corrupt hash version from value of 1, which\nmeans SHA-1, to value of 2... which would soon meen NewHash. Why not\n\"\\777\" (i.e. 0xff)?\n\n> +\ttest_must_fail git commit-graph verify 2>err &&\n> +\tgrep -v \"^\\+\" err > verify-errors &&\n> +\ttest_line_count = 1 verify-errors &&\n> +\tgrep \"hash version\" verify-errors\n> +'\n\n\nNote: all of the above tests check in load_commit_graph_one(), not the\none in verify_commit_graph(). Just FYI.\n\n> +\n> +test_expect_success 'detect too small chunk-count' '\n> +\tcd \"$TRASH_DIRECTORY/full\" &&\n> +\tcp $objdir/info/commit-graph commit-graph-backup &&\n> +\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n> +\tcorrupt_data $objdir/info/commit-graph 6 \"\\01\" &&\n> +\ttest_must_fail git commit-graph verify 2>err &&\n> +\tgrep -v \"^\\+\" err > verify-errors &&\n> +\ttest_line_count = 2 verify-errors &&\n> +\tgrep \"missing the OID Lookup chunk\" verify-errors &&\n> +\tgrep \"missing the Commit Data chunk\" verify-errors\n\nThis feels too implementation specific. We should have at least two\nchunks missing (there are 3 required chunks, and number of chunks was\nchanged to 1), but commit-graph format specification does not state that\nOID Fanout must be first, and thus it is two remaining required chunks\nthat would be missing.\n\n> +'\n> +\n> test_done\n\nOne test that I would like to see that 'git commit-grph verify'\ncorrectly detects without crashing is if commit-graph file gets\ntruncated at various lengths: shorter than smallest possible\ncommit-graph file size, in the middle of fixed header, in the middle of\nchunk lookup part, in the middle of chunk, just the trailer chopped off.\n\nBest regards,\n-- \nJakub Narębski\n"},{"id":"348437","messageId":"20180524162504.158394-1-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180511211504.79877-1-dstolee@microsoft.com","subject":"[PATCH v3 00/20] Integrate commit-graph into 'fsck' and 'gc'","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:25:23Z","receivedAt":"2018-05-24T16:25:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Thanks for all the feedback on v2. I've tried to make this round's\nreview a bit easier by splitting up the commits into smaller pieces.\nAlso, the test script now has less boilerplate and uses variables and\nclear arithmetic to explain which bytes are being modified.\n\nOne other change worth mentioning: in \"commit-graph: add '--reachable'\noption\" I put the ref-iteration into a new external\n'write_commit_graph_reachable()' method inside commit-graph.c. This\nmakes the 'gc: automatically write commit-graph files' a simpler change.\n\nThanks,\n-Stolee\n\nDerrick Stolee (20):\n commit-graph: UNLEAK before die()\n commit-graph: fix GRAPH_MIN_SIZE\n commit-graph: parse commit from chosen graph\n commit: force commit to parse from object database\n commit-graph: load a root tree from specific graph\n commit-graph: add 'verify' subcommand\n commit-graph: verify catches corrupt signature\n commit-graph: verify required chunks are present\n commit-graph: verify corrupt OID fanout and lookup\n commit-graph: verify objects exist\n commit-graph: verify root tree OIDs\n commit-graph: verify parent list\n commit-graph: verify generation number\n commit-graph: verify commit date\n commit-graph: test for corrupted octopus edge\n commit-graph: verify contents match checksum\n fsck: verify commit-graph\n commit-graph: add '--reachable' option\n gc: automatically write commit-graph files\n commit-graph: update design document\n\n Documentation/config.txt | 6 +\n Documentation/git-commit-graph.txt | 14 +-\n Documentation/git-fsck.txt | 3 +\n Documentation/git-gc.txt | 4 +\n Documentation/technical/commit-graph.txt | 22 ---\n builtin/commit-graph.c | 59 +++++++-\n builtin/fsck.c | 21 +++\n builtin/gc.c | 6 +\n commit-graph.c | 234 +++++++++++++++++++++++++++++--\n commit-graph.h | 3 +\n commit.c | 9 +-\n commit.h | 1 +\n t/t5318-commit-graph.sh | 196 ++++++++++++++++++++++++++\n 13 files changed, 539 insertions(+), 39 deletions(-)\n\n\nbase-commit: 34fdd433396ee0e3ef4de02eb2189f8226eafe4e\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348438","messageId":"20180524162504.158394-2-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"[PATCH v3 01/20] commit-graph: UNLEAK before die()","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:25:27Z","receivedAt":"2018-05-24T16:25:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 37420ae0fd..f0875b8bf3 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -51,8 +51,11 @@ static int graph_read(int argc, const char **argv)\n \tgraph_name = get_commit_graph_filename(opts.obj_dir);\n \tgraph = load_commit_graph_one(graph_name);\n \n-\tif (!graph)\n+\tif (!graph) {\n+\t\tUNLEAK(graph_name);\n \t\tdie(\"graph file %s does not exist\", graph_name);\n+\t}\n+\n \tFREE_AND_NULL(graph_name);\n \n \tprintf(\"header: %08x %d %d %d %d\\n\",\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348439","messageId":"20180524162504.158394-5-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"[PATCH v3 04/20] commit: force commit to parse from object database","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:25:33Z","receivedAt":"2018-05-24T16:25:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In anticipation of verifying commit-graph file contents against the\nobject database, create parse_commit_internal() to allow side-stepping\nthe commit-graph file and parse directly from the object database.\n\nDue to the use of generation numbers, this method should not be called\nunless the intention is explicit in avoiding commits from the\ncommit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 9 +++++++--\n commit.h | 1 +\n 2 files changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/commit.c b/commit.c\nindex 1d28677dfb..6eaed0174c 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -392,7 +392,7 @@ int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long s\n \treturn 0;\n }\n \n-int parse_commit_gently(struct commit *item, int quiet_on_missing)\n+int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph)\n {\n \tenum object_type type;\n \tvoid *buffer;\n@@ -403,7 +403,7 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \t\treturn -1;\n \tif (item->object.parsed)\n \t\treturn 0;\n-\tif (parse_commit_in_graph(item))\n+\tif (use_commit_graph && parse_commit_in_graph(item))\n \t\treturn 0;\n \tbuffer = read_sha1_file(item->object.oid.hash, &type, &size);\n \tif (!buffer)\n@@ -424,6 +424,11 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \treturn ret;\n }\n \n+int parse_commit_gently(struct commit *item, int quiet_on_missing)\n+{\n+\treturn parse_commit_internal(item, quiet_on_missing, 1);\n+}\n+\n void parse_commit_or_die(struct commit *item)\n {\n \tif (parse_commit(item))\ndiff --git a/commit.h b/commit.h\nindex b5afde1ae9..5fde74fcd7 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -73,6 +73,7 @@ struct commit *lookup_commit_reference_by_name(const char *name);\n struct commit *lookup_commit_or_die(const struct object_id *oid, const char *ref_name);\n \n int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph);\n+int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph);\n int parse_commit_gently(struct commit *item, int quiet_on_missing);\n static inline int parse_commit(struct commit *item)\n {\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348440","messageId":"20180524162504.158394-4-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"[PATCH v3 03/20] commit-graph: parse commit from chosen graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:25:31Z","receivedAt":"2018-05-24T16:25:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Before verifying a commit-graph file against the object database, we\nneed to parse all commits from the given commit-graph file. Create\nparse_commit_in_graph_one() to target a given struct commit_graph.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 18 +++++++++++++++---\n 1 file changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 82295f0975..78ba0edc80 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -310,7 +310,7 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \t}\n }\n \n-int parse_commit_in_graph(struct commit *item)\n+static int parse_commit_in_graph_one(struct commit_graph *g, struct commit *item)\n {\n \tuint32_t pos;\n \n@@ -318,9 +318,21 @@ int parse_commit_in_graph(struct commit *item)\n \t\treturn 0;\n \tif (item->object.parsed)\n \t\treturn 1;\n+\n+\tif (find_commit_in_graph(item, g, &pos))\n+\t\treturn fill_commit_in_graph(item, g, pos);\n+\n+\treturn 0;\n+}\n+\n+int parse_commit_in_graph(struct commit *item)\n+{\n+\tif (!core_commit_graph)\n+\t\treturn 0;\n+\n \tprepare_commit_graph();\n-\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n-\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n+\tif (commit_graph)\n+\t\treturn parse_commit_in_graph_one(commit_graph, item);\n \treturn 0;\n }\n \n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348441","messageId":"20180524162504.158394-7-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"[PATCH v3 06/20] commit-graph: add 'verify' subcommand","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:25:37Z","receivedAt":"2018-05-24T16:25:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"If the commit-graph file becomes corrupt, we need a way to verify\nthat its contents match the object database. In the manner of\n'git fsck' we will implement a 'git commit-graph verify' subcommand\nto report all issues with the file.\n\nAdd the 'verify' subcommand to the 'commit-graph' builtin and its\ndocumentation. The subcommand is currently a no-op except for\nloading the commit-graph into memory, which may trigger run-time\nerrors that would be caught by normal use. Add a simple test that\nensures the command returns a zero error code.\n\nIf no commit-graph file exists, this is an acceptable state. Do\nnot report any errors.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 6 ++++++\n builtin/commit-graph.c | 38 ++++++++++++++++++++++++++++++++++++++\n commit-graph.c | 26 ++++++++++++++++++++++++++\n commit-graph.h | 2 ++\n t/t5318-commit-graph.sh | 10 ++++++++++\n 5 files changed, 82 insertions(+)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 4c97b555cc..a222cfab08 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git commit-graph read' [--object-dir ]\n+'git commit-graph verify' [--object-dir ]\n 'git commit-graph write' [--object-dir ]\n \n \n@@ -52,6 +53,11 @@ existing commit-graph file.\n Read a graph file given by the commit-graph file and output basic\n details about the graph file. Used for debugging purposes.\n \n+'verify'::\n+\n+Read the commit-graph file and verify its contents against the object\n+database. Used to check for corrupted data.\n+\n \n EXAMPLES\n --------\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex f0875b8bf3..0433dd6e20 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -8,10 +8,16 @@\n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n+\tN_(\"git commit-graph verify [--object-dir ]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n+static const char * const builtin_commit_graph_verify_usage[] = {\n+\tN_(\"git commit-graph verify [--object-dir ]\"),\n+\tNULL\n+};\n+\n static const char * const builtin_commit_graph_read_usage[] = {\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tNULL\n@@ -29,6 +35,36 @@ static struct opts_commit_graph {\n \tint append;\n } opts;\n \n+\n+static int graph_verify(int argc, const char **argv)\n+{\n+\tstruct commit_graph *graph = 0;\n+\tchar *graph_name;\n+\n+\tstatic struct option builtin_commit_graph_verify_options[] = {\n+\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n+\t\t\t N_(\"dir\"),\n+\t\t\t N_(\"The object directory to store the graph\")),\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t builtin_commit_graph_verify_options,\n+\t\t\t builtin_commit_graph_verify_usage, 0);\n+\n+\tif (!opts.obj_dir)\n+\t\topts.obj_dir = get_object_directory();\n+\n+\tgraph_name = get_commit_graph_filename(opts.obj_dir);\n+\tgraph = load_commit_graph_one(graph_name);\n+\tFREE_AND_NULL(graph_name);\n+\n+\tif (!graph)\n+\t\treturn 0;\n+\n+\treturn verify_commit_graph(graph);\n+}\n+\n static int graph_read(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n@@ -163,6 +199,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n \tif (argc > 0) {\n+\t\tif (!strcmp(argv[0], \"verify\"))\n+\t\t\treturn graph_verify(argc, argv);\n \t\tif (!strcmp(argv[0], \"read\"))\n \t\t\treturn graph_read(argc, argv);\n \t\tif (!strcmp(argv[0], \"write\"))\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 25893ec096..55b41664ee 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -836,3 +836,29 @@ void write_commit_graph(const char *obj_dir,\n \toids.alloc = 0;\n \toids.nr = 0;\n }\n+\n+static int verify_commit_graph_error;\n+\n+static void graph_report(const char *fmt, ...)\n+{\n+\tva_list ap;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tverify_commit_graph_error = 1;\n+\n+\tva_start(ap, fmt);\n+\tstrbuf_vaddf(&sb, fmt, ap);\n+\n+\tfprintf(stderr, \"%s\\n\", sb.buf);\n+\tstrbuf_release(&sb);\n+\tva_end(ap);\n+}\n+\n+int verify_commit_graph(struct commit_graph *g)\n+{\n+\tif (!g) {\n+\t\tgraph_report(\"no commit-graph file loaded\");\n+\t\treturn 1;\n+\t}\n+\n+\treturn verify_commit_graph_error;\n+}\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 96cccb10f3..71a39c5a57 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -53,4 +53,6 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tint nr_commits,\n \t\t\tint append);\n \n+int verify_commit_graph(struct commit_graph *g);\n+\n #endif\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 77d85aefe7..6ca451dfd2 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -11,6 +11,11 @@ test_expect_success 'setup full repo' '\n \tobjdir=\".git/objects\"\n '\n \n+test_expect_success 'verify graph with no graph file' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph verify\n+'\n+\n test_expect_success 'write graph with no packs' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write --object-dir . &&\n@@ -230,4 +235,9 @@ test_expect_success 'perform fast-forward merge in full repo' '\n \ttest_cmp expect output\n '\n \n+test_expect_success 'git commit-graph verify' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph verify >output\n+'\n+\n test_done\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348442","messageId":"20180524162504.158394-8-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"[PATCH v3 07/20] commit-graph: verify catches corrupt signature","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:25:40Z","receivedAt":"2018-05-24T16:25:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is the first of several commits that add a test to check that\n'git commit-graph verify' catches corruption in the commit-graph\nfile. The first test checks that the command catches an error in\nthe file signature. This is a check that exists in the existing\ncommit-graph reading code.\n\nAdd a helper method 'corrupt_graph_and_verify' to the test script\nt5318-commit-graph.sh. This helper corrupts the commit-graph file\nat a certain location, runs 'git commit-graph verify', and reports\nthe output to the 'err' file. This data is filtered to remove the\nlines added by 'test_must_fail' when the test is run verbosely.\nThen, the output is checked to contain a specific error message.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5318-commit-graph.sh | 43 +++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 43 insertions(+)\n\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 6ca451dfd2..bd64481c7a 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -235,9 +235,52 @@ test_expect_success 'perform fast-forward merge in full repo' '\n \ttest_cmp expect output\n '\n \n+# the verify tests below expect the commit-graph to contain\n+# exactly the commits reachable from the commits/8 branch.\n+# If the file changes the set of commits in the list, then the\n+# offsets into the binary file will result in different edits\n+# and the tests will likely break.\n+\n test_expect_success 'git commit-graph verify' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit rev-parse commits/8 | git commit-graph write --stdin-commits &&\n \tgit commit-graph verify >output\n '\n \n+GRAPH_BYTE_VERSION=4\n+GRAPH_BYTE_HASH=5\n+\n+# usage: corrupt_graph_and_verify \n+# Manipulates the commit-graph file at the position\n+# by inserting the data, then runs 'git commit-graph verify'\n+# and places the output in the file 'err'. Test 'err' for\n+# the given string.\n+corrupt_graph_and_verify() {\n+\tpos=$1\n+\tdata=\"${2:-\\0}\"\n+\tgrepstr=$3\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n+\tcp $objdir/info/commit-graph commit-graph-backup &&\n+\tprintf \"$data\" | dd of=\"$objdir/info/commit-graph\" bs=1 seek=\"$pos\" conv=notrunc &&\n+\ttest_must_fail git commit-graph verify 2>test_err &&\n+\tgrep -v \"^+\" test_err >err\n+\tgrep \"$grepstr\" err\n+}\n+\n+test_expect_success 'detect bad signature' '\n+\tcorrupt_graph_and_verify 0 \"\\0\" \\\n+\t\t\"graph signature\"\n+'\n+\n+test_expect_success 'detect bad version' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\02\" \\\n+\t\t\"graph version\"\n+'\n+\n+test_expect_success 'detect bad hash version' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_HASH \"\\02\" \\\n+\t\t\"hash version\"\n+'\n+\n test_done\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348443","messageId":"20180524162504.158394-6-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"[PATCH v3 05/20] commit-graph: load a root tree from specific graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:25:35Z","receivedAt":"2018-05-24T16:26:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When lazy-loading a tree for a commit, it will be important to select\nthe tree from a specific struct commit_graph. Create a new method that\nspecifies the commit-graph file and use that in\nget_commit_tree_in_graph().\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 12 +++++++++---\n 1 file changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 78ba0edc80..25893ec096 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -358,14 +358,20 @@ static struct tree *load_tree_for_commit(struct commit_graph *g, struct commit *\n \treturn c->maybe_tree;\n }\n \n-struct tree *get_commit_tree_in_graph(const struct commit *c)\n+static struct tree *get_commit_tree_in_graph_one(struct commit_graph *g,\n+\t\t\t\t\t\t const struct commit *c)\n {\n \tif (c->maybe_tree)\n \t\treturn c->maybe_tree;\n \tif (c->graph_pos == COMMIT_NOT_FROM_GRAPH)\n-\t\tBUG(\"get_commit_tree_in_graph called from non-commit-graph commit\");\n+\t\tBUG(\"get_commit_tree_in_graph_one called from non-commit-graph commit\");\n+\n+\treturn load_tree_for_commit(g, (struct commit *)c);\n+}\n \n-\treturn load_tree_for_commit(commit_graph, (struct commit *)c);\n+struct tree *get_commit_tree_in_graph(const struct commit *c)\n+{\n+\treturn get_commit_tree_in_graph_one(commit_graph, c);\n }\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348444","messageId":"20180524162504.158394-9-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"[PATCH v3 08/20] commit-graph: verify required chunks are present","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:25:42Z","receivedAt":"2018-05-24T16:26:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file requires the following three chunks:\n\n* OID Fanout\n* OID Lookup\n* Commit Data\n\nIf any of these are missing, then the 'verify' subcommand should\nreport a failure. This includes the chunk IDs malformed or the\nchunk count is truncated.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 9 +++++++++\n t/t5318-commit-graph.sh | 29 +++++++++++++++++++++++++++++\n 2 files changed, 38 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 55b41664ee..06e3e4f9ba 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -860,5 +860,14 @@ int verify_commit_graph(struct commit_graph *g)\n \t\treturn 1;\n \t}\n \n+\tverify_commit_graph_error = 0;\n+\n+\tif (!g->chunk_oid_fanout)\n+\t\tgraph_report(\"commit-graph is missing the OID Fanout chunk\");\n+\tif (!g->chunk_oid_lookup)\n+\t\tgraph_report(\"commit-graph is missing the OID Lookup chunk\");\n+\tif (!g->chunk_commit_data)\n+\t\tgraph_report(\"commit-graph is missing the Commit Data chunk\");\n+\n \treturn verify_commit_graph_error;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex bd64481c7a..4ef3fe3dc2 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -249,6 +249,15 @@ test_expect_success 'git commit-graph verify' '\n \n GRAPH_BYTE_VERSION=4\n GRAPH_BYTE_HASH=5\n+GRAPH_BYTE_CHUNK_COUNT=6\n+GRAPH_CHUNK_LOOKUP_OFFSET=8\n+GRAPH_CHUNK_LOOKUP_WIDTH=12\n+GRAPH_CHUNK_LOOKUP_ROWS=5\n+GRAPH_BYTE_OID_FANOUT_ID=$GRAPH_CHUNK_LOOKUP_OFFSET\n+GRAPH_BYTE_OID_LOOKUP_ID=`expr $GRAPH_CHUNK_LOOKUP_OFFSET + \\\n+\t\t\t 1 \\* $GRAPH_CHUNK_LOOKUP_WIDTH`\n+GRAPH_BYTE_COMMIT_DATA_ID=`expr $GRAPH_CHUNK_LOOKUP_OFFSET + \\\n+\t\t\t\t2 \\* $GRAPH_CHUNK_LOOKUP_WIDTH`\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -283,4 +292,24 @@ test_expect_success 'detect bad hash version' '\n \t\t\"hash version\"\n '\n \n+test_expect_success 'detect bad chunk count' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_CHUNK_COUNT \"\\02\" \\\n+\t\t\"missing the Commit Data chunk\"\n+'\n+\n+test_expect_success 'detect missing OID fanout chunk' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_FANOUT_ID \"\\0\" \\\n+\t\t\"missing the OID Fanout chunk\"\n+'\n+\n+test_expect_success 'detect missing OID lookup chunk' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_LOOKUP_ID \"\\0\" \\\n+\t\t\"missing the OID Lookup chunk\"\n+'\n+\n+test_expect_success 'detect missing commit data chunk' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_DATA_ID \"\\0\" \\\n+\t\t\"missing the Commit Data chunk\"\n+'\n+\n test_done\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348445","messageId":"20180524162504.158394-3-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"[PATCH v3 02/20] commit-graph: fix GRAPH_MIN_SIZE","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:25:29Z","receivedAt":"2018-05-24T16:26:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The GRAPH_MIN_SIZE macro should be the smallest size of a parsable\ncommit-graph file. However, the minimum number of chunks was wrong.\nIt is possible to write a commit-graph file with zero commits, and\nthat violates this macro's value.\n\nRewrite the macro, and use extra macros to better explain the magic\nconstants.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a8c337dd77..82295f0975 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -33,10 +33,11 @@\n \n #define GRAPH_LAST_EDGE 0x80000000\n \n+#define GRAPH_HEADER_SIZE 8\n #define GRAPH_FANOUT_SIZE (4 * 256)\n #define GRAPH_CHUNKLOOKUP_WIDTH 12\n-#define GRAPH_MIN_SIZE (5 * GRAPH_CHUNKLOOKUP_WIDTH + GRAPH_FANOUT_SIZE + \\\n-\t\t\tGRAPH_OID_LEN + 8)\n+#define GRAPH_MIN_SIZE (GRAPH_HEADER_SIZE + 4 * GRAPH_CHUNKLOOKUP_WIDTH \\\n+\t\t\t+ GRAPH_FANOUT_SIZE + GRAPH_OID_LEN)\n \n char *get_commit_graph_filename(const char *obj_dir)\n {\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348446","messageId":"20180524162504.158394-10-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"[PATCH v3 09/20] commit-graph: verify corrupt OID fanout and lookup","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:25:45Z","receivedAt":"2018-05-24T16:26:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In the commit-graph file, the OID fanout chunk provides an index into\nthe OID lookup. The 'verify' subcommand should find incorrect values\nin the fanout.\n\nSimilarly, the 'verify' subcommand should find out-of-order values in\nthe OID lookup.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 36 ++++++++++++++++++++++++++++++++++++\n t/t5318-commit-graph.sh | 22 ++++++++++++++++++++++\n 2 files changed, 58 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 06e3e4f9ba..cbd1aae514 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -855,6 +855,9 @@ static void graph_report(const char *fmt, ...)\n \n int verify_commit_graph(struct commit_graph *g)\n {\n+\tuint32_t i, cur_fanout_pos = 0;\n+\tstruct object_id prev_oid, cur_oid;\n+\n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n \t\treturn 1;\n@@ -869,5 +872,38 @@ int verify_commit_graph(struct commit_graph *g)\n \tif (!g->chunk_commit_data)\n \t\tgraph_report(\"commit-graph is missing the Commit Data chunk\");\n \n+\tif (verify_commit_graph_error)\n+\t\treturn verify_commit_graph_error;\n+\n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n+\n+\t\tif (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n+\t\t\tgraph_report(\"commit-graph has incorrect OID order: %s then %s\",\n+\t\t\t\t oid_to_hex(&prev_oid),\n+\t\t\t\t oid_to_hex(&cur_oid));\n+\n+\t\toidcpy(&prev_oid, &cur_oid);\n+\n+\t\twhile (cur_oid.hash[0] > cur_fanout_pos) {\n+\t\t\tuint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n+\t\t\tif (i != fanout_value)\n+\t\t\t\tgraph_report(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\",\n+\t\t\t\t\t cur_fanout_pos, fanout_value, i);\n+\n+\t\t\tcur_fanout_pos++;\n+\t\t}\n+\t}\n+\n+\twhile (cur_fanout_pos < 256) {\n+\t\tuint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n+\n+\t\tif (g->num_commits != fanout_value)\n+\t\t\tgraph_report(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\",\n+\t\t\t\t cur_fanout_pos, fanout_value, i);\n+\n+\t\tcur_fanout_pos++;\n+\t}\n+\n \treturn verify_commit_graph_error;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 4ef3fe3dc2..c050ef980b 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -247,6 +247,7 @@ test_expect_success 'git commit-graph verify' '\n \tgit commit-graph verify >output\n '\n \n+HASH_LEN=20\n GRAPH_BYTE_VERSION=4\n GRAPH_BYTE_HASH=5\n GRAPH_BYTE_CHUNK_COUNT=6\n@@ -258,6 +259,12 @@ GRAPH_BYTE_OID_LOOKUP_ID=`expr $GRAPH_CHUNK_LOOKUP_OFFSET + \\\n \t\t\t 1 \\* $GRAPH_CHUNK_LOOKUP_WIDTH`\n GRAPH_BYTE_COMMIT_DATA_ID=`expr $GRAPH_CHUNK_LOOKUP_OFFSET + \\\n \t\t\t\t2 \\* $GRAPH_CHUNK_LOOKUP_WIDTH`\n+GRAPH_FANOUT_OFFSET=`expr $GRAPH_CHUNK_LOOKUP_OFFSET + \\\n+\t\t\t $GRAPH_CHUNK_LOOKUP_WIDTH \\* $GRAPH_CHUNK_LOOKUP_ROWS`\n+GRAPH_BYTE_FANOUT1=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 4`\n+GRAPH_BYTE_FANOUT2=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 255`\n+GRAPH_OID_LOOKUP_OFFSET=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 256`\n+GRAPH_BYTE_OID_LOOKUP_ORDER=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8`\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -312,4 +319,19 @@ test_expect_success 'detect missing commit data chunk' '\n \t\t\"missing the Commit Data chunk\"\n '\n \n+test_expect_success 'detect incorrect fanout' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_FANOUT1 \"\\01\" \\\n+\t\t\"fanout value\"\n+'\n+\n+test_expect_success 'detect incorrect fanout' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_FANOUT2 \"\\01\" \\\n+\t\t\"fanout value\"\n+'\n+\n+test_expect_success 'detect incorrect OID order' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_LOOKUP_ORDER \"\\01\" \\\n+\t\t\"incorrect OID order\"\n+'\n+\n test_done\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348447","messageId":"20180524162504.158394-11-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"[PATCH v3 10/20] commit-graph: verify objects exist","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:25:47Z","receivedAt":"2018-05-24T16:26:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"In the 'verify' subcommand, load commits directly from the object\ndatabase to ensure they exist. Parse by skipping the commit-graph.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 20 ++++++++++++++++++++\n t/t5318-commit-graph.sh | 7 +++++++\n 2 files changed, 27 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex cbd1aae514..0420ebcd87 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -238,6 +238,10 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n {\n \tstruct commit *c;\n \tstruct object_id oid;\n+\n+\tif (pos >= g->num_commits)\n+\t\tdie(\"invalid parent position %\"PRIu64, pos);\n+\n \thashcpy(oid.hash, g->chunk_oid_lookup + g->hash_len * pos);\n \tc = lookup_commit(&oid);\n \tif (!c)\n@@ -905,5 +909,21 @@ int verify_commit_graph(struct commit_graph *g)\n \t\tcur_fanout_pos++;\n \t}\n \n+\tif (verify_commit_graph_error)\n+\t\treturn verify_commit_graph_error;\n+\n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct commit *odb_commit;\n+\n+\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n+\n+\t\todb_commit = (struct commit *)create_object(cur_oid.hash, alloc_commit_node());\n+\t\tif (parse_commit_internal(odb_commit, 0, 0)) {\n+\t\t\tgraph_report(\"failed to parse %s from object database\",\n+\t\t\t\t oid_to_hex(&cur_oid));\n+\t\t\tcontinue;\n+\t\t}\n+\t}\n+\n \treturn verify_commit_graph_error;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex c050ef980b..996a016239 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -247,6 +247,7 @@ test_expect_success 'git commit-graph verify' '\n \tgit commit-graph verify >output\n '\n \n+NUM_COMMITS=9\n HASH_LEN=20\n GRAPH_BYTE_VERSION=4\n GRAPH_BYTE_HASH=5\n@@ -265,6 +266,7 @@ GRAPH_BYTE_FANOUT1=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 4`\n GRAPH_BYTE_FANOUT2=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 255`\n GRAPH_OID_LOOKUP_OFFSET=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 256`\n GRAPH_BYTE_OID_LOOKUP_ORDER=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8`\n+GRAPH_BYTE_OID_LOOKUP_MISSING=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 4 + 10`\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -334,4 +336,9 @@ test_expect_success 'detect incorrect OID order' '\n \t\t\"incorrect OID order\"\n '\n \n+test_expect_success 'detect OID not in object database' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_LOOKUP_MISSING \"\\01\" \\\n+\t\t\"from object database\"\n+'\n+\n test_done\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348448","messageId":"20180524162504.158394-12-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"[PATCH v3 11/20] commit-graph: verify root tree OIDs","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:25:49Z","receivedAt":"2018-05-24T16:26:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The 'verify' subcommand must compare the commit content parsed from the\ncommit-graph and compare it against the content in the object database.\nUse lookup_commit() and parse_commit_in_graph_one() to parse the commits\nfrom the graph and compare against a commit that is loaded separately\nand parsed directly from the object database.\n\nAdd checks for the root tree OID.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 17 ++++++++++++++++-\n t/t5318-commit-graph.sh | 7 +++++++\n 2 files changed, 23 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0420ebcd87..19ea369fc6 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -880,6 +880,8 @@ int verify_commit_graph(struct commit_graph *g)\n \t\treturn verify_commit_graph_error;\n \n \tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct commit *graph_commit;\n+\n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n \t\tif (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n@@ -897,6 +899,11 @@ int verify_commit_graph(struct commit_graph *g)\n \n \t\t\tcur_fanout_pos++;\n \t\t}\n+\n+\t\tgraph_commit = lookup_commit(&cur_oid);\n+\t\tif (!parse_commit_in_graph_one(g, graph_commit))\n+\t\t\tgraph_report(\"failed to parse %s from commit-graph\",\n+\t\t\t\t oid_to_hex(&cur_oid));\n \t}\n \n \twhile (cur_fanout_pos < 256) {\n@@ -913,16 +920,24 @@ int verify_commit_graph(struct commit_graph *g)\n \t\treturn verify_commit_graph_error;\n \n \tfor (i = 0; i < g->num_commits; i++) {\n-\t\tstruct commit *odb_commit;\n+\t\tstruct commit *graph_commit, *odb_commit;\n \n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n+\t\tgraph_commit = lookup_commit(&cur_oid);\n \t\todb_commit = (struct commit *)create_object(cur_oid.hash, alloc_commit_node());\n \t\tif (parse_commit_internal(odb_commit, 0, 0)) {\n \t\t\tgraph_report(\"failed to parse %s from object database\",\n \t\t\t\t oid_to_hex(&cur_oid));\n \t\t\tcontinue;\n \t\t}\n+\n+\t\tif (oidcmp(&get_commit_tree_in_graph_one(g, graph_commit)->object.oid,\n+\t\t\t get_commit_tree_oid(odb_commit)))\n+\t\t\tgraph_report(\"root tree OID for commit %s in commit-graph is %s != %s\",\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t oid_to_hex(get_commit_tree_oid(graph_commit)),\n+\t\t\t\t oid_to_hex(get_commit_tree_oid(odb_commit)));\n \t}\n \n \treturn verify_commit_graph_error;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 996a016239..21cc8e82f3 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -267,6 +267,8 @@ GRAPH_BYTE_FANOUT2=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 255`\n GRAPH_OID_LOOKUP_OFFSET=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 256`\n GRAPH_BYTE_OID_LOOKUP_ORDER=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8`\n GRAPH_BYTE_OID_LOOKUP_MISSING=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 4 + 10`\n+GRAPH_COMMIT_DATA_OFFSET=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* $NUM_COMMITS`\n+GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -341,4 +343,9 @@ test_expect_success 'detect OID not in object database' '\n \t\t\"from object database\"\n '\n \n+test_expect_success 'detect incorrect tree OID' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_TREE \"\\01\" \\\n+\t\t\"root tree OID for commit\"\n+'\n+\n test_done\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348449","messageId":"20180524162504.158394-13-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"[PATCH v3 12/20] commit-graph: verify parent list","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:25:52Z","receivedAt":"2018-05-24T16:26:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file stores parents in a two-column portion of the\ncommit data chunk. If there is only one parent, then the second column\nstores 0xFFFFFFFF to indicate no second parent.\n\nThe 'verify' subcommand checks the parent list for the commit loaded\nfrom the commit-graph and the one parsed from the object database. Test\nthese checks for corrupt parents, too many parents, and wrong parents.\n\nThe octopus merge will be tested in a later commit.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 25 +++++++++++++++++++++++++\n t/t5318-commit-graph.sh | 18 ++++++++++++++++++\n 2 files changed, 43 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 19ea369fc6..fff22dc0c3 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -921,6 +921,7 @@ int verify_commit_graph(struct commit_graph *g)\n \n \tfor (i = 0; i < g->num_commits; i++) {\n \t\tstruct commit *graph_commit, *odb_commit;\n+\t\tstruct commit_list *graph_parents, *odb_parents;\n \n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n@@ -938,6 +939,30 @@ int verify_commit_graph(struct commit_graph *g)\n \t\t\t\t oid_to_hex(&cur_oid),\n \t\t\t\t oid_to_hex(get_commit_tree_oid(graph_commit)),\n \t\t\t\t oid_to_hex(get_commit_tree_oid(odb_commit)));\n+\n+\t\tgraph_parents = graph_commit->parents;\n+\t\todb_parents = odb_commit->parents;\n+\n+\t\twhile (graph_parents) {\n+\t\t\tif (odb_parents == NULL) {\n+\t\t\t\tgraph_report(\"commit-graph parent list for commit %s is too long\",\n+\t\t\t\t\t oid_to_hex(&cur_oid));\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tif (oidcmp(&graph_parents->item->object.oid, &odb_parents->item->object.oid))\n+\t\t\t\tgraph_report(\"commit-graph parent for %s is %s != %s\",\n+\t\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid),\n+\t\t\t\t\t oid_to_hex(&odb_parents->item->object.oid));\n+\n+\t\t\tgraph_parents = graph_parents->next;\n+\t\t\todb_parents = odb_parents->next;\n+\t\t}\n+\n+\t\tif (odb_parents != NULL)\n+\t\t\tgraph_report(\"commit-graph parent list for commit %s terminates early\",\n+\t\t\t\t oid_to_hex(&cur_oid));\n \t}\n \n \treturn verify_commit_graph_error;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 21cc8e82f3..12f0d7f54d 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -269,6 +269,9 @@ GRAPH_BYTE_OID_LOOKUP_ORDER=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8`\n GRAPH_BYTE_OID_LOOKUP_MISSING=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 4 + 10`\n GRAPH_COMMIT_DATA_OFFSET=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* $NUM_COMMITS`\n GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n+GRAPH_BYTE_COMMIT_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN`\n+GRAPH_BYTE_COMMIT_EXTRA_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4`\n+GRAPH_BYTE_COMMIT_WRONG_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3`\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -348,4 +351,19 @@ test_expect_success 'detect incorrect tree OID' '\n \t\t\"root tree OID for commit\"\n '\n \n+test_expect_success 'detect incorrect parent int-id' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_PARENT \"\\01\" \\\n+\t\t\"invalid parent\"\n+'\n+\n+test_expect_success 'detect extra parent int-id' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_EXTRA_PARENT \"\\00\" \\\n+\t\t\"is too long\"\n+'\n+\n+test_expect_success 'detect incorrect tree OID' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_WRONG_PARENT \"\\01\" \\\n+\t\t\"commit-graph parent for\"\n+'\n+\n test_done\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348450","messageId":"20180524162504.158394-15-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"[PATCH v3 14/20] commit-graph: verify commit date","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:25:57Z","receivedAt":"2018-05-24T16:26:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Signed-off-by: Derrick Stolee \n---\n commit-graph.c | 6 ++++++\n t/t5318-commit-graph.sh | 6 ++++++\n 2 files changed, 12 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex ead92460c1..d2b291aca2 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -981,6 +981,12 @@ int verify_commit_graph(struct commit_graph *g)\n \t\t\t\t oid_to_hex(&cur_oid),\n \t\t\t\t graph_commit->generation,\n \t\t\t\t max_generation + 1);\n+\n+\t\tif (graph_commit->date != odb_commit->date)\n+\t\t\tgraph_report(\"commit date for commit %s in commit-graph is %\"PRItime\" != %\"PRItime,\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t graph_commit->date,\n+\t\t\t\t odb_commit->date);\n \t}\n \n \treturn verify_commit_graph_error;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 673b0d37d5..58adb8246d 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -273,6 +273,7 @@ GRAPH_BYTE_COMMIT_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN`\n GRAPH_BYTE_COMMIT_EXTRA_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4`\n GRAPH_BYTE_COMMIT_WRONG_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3`\n GRAPH_BYTE_COMMIT_GENERATION=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 8`\n+GRAPH_BYTE_COMMIT_DATE=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 12`\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -372,4 +373,9 @@ test_expect_success 'detect incorrect generation number' '\n \t\t\"generation\"\n '\n \n+test_expect_success 'detect incorrect commit date' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_DATE \"\\01\" \\\n+\t\t\"commit date\"\n+'\n+\n test_done\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348451","messageId":"20180524162504.158394-17-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"[PATCH v3 16/20] commit-graph: verify contents match checksum","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:26:01Z","receivedAt":"2018-05-24T16:26:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file ends with a SHA1 hash of the previous contents. If\na commit-graph file has errors but the checksum hash is correct, then we\nknow that the problem is a bug in Git and not simply file corruption\nafter-the-fact.\n\nCompute the checksum right away so it is the first error that appears,\nand make the message translatable since this error can be \"corrected\" by\na user by simply deleting the file and recomputing. The rest of the\nerrors are useful only to developers.\n\nBe sure to continue checking the rest of the file data if the checksum\nis wrong. This is important for our tests, as we break the checksum as\nwe modify bytes of the commit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 16 ++++++++++++++--\n t/t5318-commit-graph.sh | 6 ++++++\n 2 files changed, 20 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d2b291aca2..a33600c584 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -841,6 +841,7 @@ void write_commit_graph(const char *obj_dir,\n \toids.nr = 0;\n }\n \n+#define VERIFY_COMMIT_GRAPH_ERROR_HASH 2\n static int verify_commit_graph_error;\n \n static void graph_report(const char *fmt, ...)\n@@ -860,7 +861,9 @@ static void graph_report(const char *fmt, ...)\n int verify_commit_graph(struct commit_graph *g)\n {\n \tuint32_t i, cur_fanout_pos = 0;\n-\tstruct object_id prev_oid, cur_oid;\n+\tstruct object_id prev_oid, cur_oid, checksum;\n+\tstruct hashfile *f;\n+\tint devnull;\n \n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n@@ -879,6 +882,15 @@ int verify_commit_graph(struct commit_graph *g)\n \tif (verify_commit_graph_error)\n \t\treturn verify_commit_graph_error;\n \n+\tdevnull = open(\"/dev/null\", O_WRONLY);\n+\tf = hashfd(devnull, NULL);\n+\thashwrite(f, g->data, g->data_len - g->hash_len);\n+\tfinalize_hashfile(f, checksum.hash, CSUM_CLOSE);\n+\tif (hashcmp(checksum.hash, g->data + g->data_len - g->hash_len)) {\n+\t\tgraph_report(_(\"the commit-graph file has incorrect checksum and is likely corrupt\"));\n+\t\tverify_commit_graph_error = VERIFY_COMMIT_GRAPH_ERROR_HASH;\n+\t}\n+\n \tfor (i = 0; i < g->num_commits; i++) {\n \t\tstruct commit *graph_commit;\n \n@@ -916,7 +928,7 @@ int verify_commit_graph(struct commit_graph *g)\n \t\tcur_fanout_pos++;\n \t}\n \n-\tif (verify_commit_graph_error)\n+\tif (verify_commit_graph_error & ~VERIFY_COMMIT_GRAPH_ERROR_HASH)\n \t\treturn verify_commit_graph_error;\n \n \tfor (i = 0; i < g->num_commits; i++) {\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 240aef6add..2680a2ebff 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -279,6 +279,7 @@ GRAPH_COMMIT_DATA_WIDTH=`expr $HASH_LEN + 16`\n GRAPH_OCTOPUS_DATA_OFFSET=`expr $GRAPH_COMMIT_DATA_OFFSET + \\\n \t\t\t\t$GRAPH_COMMIT_DATA_WIDTH \\* $NUM_COMMITS`\n GRAPH_BYTE_OCTOPUS=`expr $GRAPH_OCTOPUS_DATA_OFFSET + 4`\n+GRAPH_BYTE_FOOTER=`expr $GRAPH_OCTOPUS_DATA_OFFSET + 4 \\* $NUM_OCTOPUS_EDGES`\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -388,4 +389,9 @@ test_expect_success 'detect incorrect parent for octopus merge' '\n \t\t\"invalid parent\"\n '\n \n+test_expect_success 'detect invalid checksum hash' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_FOOTER \"\\00\" \\\n+\t\t\"incorrect checksum\"\n+'\n+\n test_done\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348452","messageId":"20180524162504.158394-20-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"[PATCH v3 19/20] gc: automatically write commit-graph files","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:26:09Z","receivedAt":"2018-05-24T16:26:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file is a very helpful feature for speeding up git\noperations. In order to make it more useful, write the commit-graph file\nby default during standard garbage collection operations.\n\nAdd a 'gc.commitGraph' config setting that triggers writing a\ncommit-graph file after any non-trivial 'git gc' command. Defaults to\nfalse while the commit-graph feature matures. We specifically do not\nwant to turn this on by default until the commit-graph feature is fully\nintegrated with history-modifying features like shallow clones.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 6 ++++++\n Documentation/git-gc.txt | 4 ++++\n builtin/gc.c | 6 ++++++\n t/t5318-commit-graph.sh | 14 ++++++++++++++\n 4 files changed, 30 insertions(+)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 11f027194e..9a3abd87e7 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -1553,6 +1553,12 @@ gc.autoDetach::\n \tMake `git gc --auto` return immediately and run in background\n \tif the system supports it. Default is true.\n \n+gc.commitGraph::\n+\tIf true, then gc will rewrite the commit-graph file after any\n+\tchange to the object database. If '--auto' is used, then the\n+\tcommit-graph will not be updated unless the threshold is met.\n+\tSee linkgit:git-commit-graph[1] for details.\n+\n gc.logExpiry::\n \tIf the file gc.log exists, then `git gc --auto` won't run\n \tunless that file is more than 'gc.logExpiry' old. Default is\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex 571b5a7e3c..17dd654a59 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -119,6 +119,10 @@ The optional configuration variable `gc.packRefs` determines if\n it within all non-bare repos or it can be set to a boolean value.\n This defaults to true.\n \n+The optional configuration variable 'gc.commitGraph' determines if\n+'git gc' runs 'git commit-graph write'. This can be set to a boolean\n+value. This defaults to false.\n+\n The optional configuration variable `gc.aggressiveWindow` controls how\n much time is spent optimizing the delta compression of the objects in\n the repository when the --aggressive option is specified. The larger\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 77fa720bd0..efd214a59f 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -20,6 +20,7 @@\n #include \"argv-array.h\"\n #include \"commit.h\"\n #include \"packfile.h\"\n+#include \"commit-graph.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -34,6 +35,7 @@ static int aggressive_depth = 50;\n static int aggressive_window = 250;\n static int gc_auto_threshold = 6700;\n static int gc_auto_pack_limit = 50;\n+static int gc_commit_graph = 0;\n static int detach_auto = 1;\n static timestamp_t gc_log_expire_time;\n static const char *gc_log_expire = \"1.day.ago\";\n@@ -121,6 +123,7 @@ static void gc_config(void)\n \tgit_config_get_int(\"gc.aggressivedepth\", &aggressive_depth);\n \tgit_config_get_int(\"gc.auto\", &gc_auto_threshold);\n \tgit_config_get_int(\"gc.autopacklimit\", &gc_auto_pack_limit);\n+\tgit_config_get_bool(\"gc.commitgraph\", &gc_commit_graph);\n \tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n \tgit_config_get_expiry(\"gc.pruneexpire\", &prune_expire);\n \tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n@@ -480,6 +483,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tif (pack_garbage.nr > 0)\n \t\tclean_pack_garbage();\n \n+\tif (gc_commit_graph)\n+\t\twrite_commit_graph_reachable(get_object_directory(), 0);\n+\n \tif (auto_gc && too_many_loose_objects())\n \t\twarning(_(\"There are too many unreachable loose objects; \"\n \t\t\t\"run 'git prune' to remove them.\"));\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex a659620332..d20b17586f 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -245,6 +245,20 @@ test_expect_success 'perform fast-forward merge in full repo' '\n \ttest_cmp expect output\n '\n \n+test_expect_success 'check that gc clears commit-graph' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit --allow-empty -m \"blank\" &&\n+\tgit commit-graph write --reachable &&\n+\tcp $objdir/info/commit-graph commit-graph-before-gc &&\n+\tgit reset --hard HEAD~1 &&\n+\tgit config gc.commitGraph true &&\n+\tgit gc &&\n+\tcp $objdir/info/commit-graph commit-graph-after-gc &&\n+\t! test_cmp commit-graph-before-gc commit-graph-after-gc &&\n+\tgit commit-graph write --reachable &&\n+\ttest_cmp commit-graph-after-gc $objdir/info/commit-graph\n+'\n+\n # the verify tests below expect the commit-graph to contain\n # exactly the commits reachable from the commits/8 branch.\n # If the file changes the set of commits in the list, then the\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348453","messageId":"20180524162504.158394-21-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"[PATCH v3 20/20] commit-graph: update design document","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:26:11Z","receivedAt":"2018-05-24T16:26:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph feature is now integrated with 'fsck' and 'gc',\nso remove those items from the \"Future Work\" section of the\ncommit-graph design document.\n\nAlso remove the section on lazy-loading trees, as that was completed\nin an earlier patch series.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 22 ----------------------\n 1 file changed, 22 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex e1a883eb46..c664acbd76 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -118,9 +118,6 @@ Future Work\n - The commit graph feature currently does not honor commit grafts. This can\n be remedied by duplicating or refactoring the current graft logic.\n \n-- The 'commit-graph' subcommand does not have a \"verify\" mode that is\n- necessary for integration with fsck.\n-\n - After computing and storing generation numbers, we must make graph\n walks aware of generation numbers to gain the performance benefits they\n enable. This will mostly be accomplished by swapping a commit-date-ordered\n@@ -130,25 +127,6 @@ Future Work\n - 'log --topo-order'\n - 'tag --merged'\n \n-- Currently, parse_commit_gently() requires filling in the root tree\n- object for a commit. This passes through lookup_tree() and consequently\n- lookup_object(). Also, it calls lookup_commit() when loading the parents.\n- These method calls check the ODB for object existence, even if the\n- consumer does not need the content. For example, we do not need the\n- tree contents when computing merge bases. Now that commit parsing is\n- removed from the computation time, these lookup operations are the\n- slowest operations keeping graph walks from being fast. Consider\n- loading these objects without verifying their existence in the ODB and\n- only loading them fully when consumers need them. Consider a method\n- such as \"ensure_tree_loaded(commit)\" that fully loads a tree before\n- using commit->tree.\n-\n-- The current design uses the 'commit-graph' subcommand to generate the graph.\n- When this feature stabilizes enough to recommend to most users, we should\n- add automatic graph writes to common operations that create many commits.\n- For example, one could compute a graph on 'clone', 'fetch', or 'repack'\n- commands.\n-\n - A server could provide a commit graph file as part of the network protocol\n to avoid extra calculations by clients. This feature is only of benefit if\n the user is willing to trust the file, because verifying the file is correct\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348454","messageId":"20180524162504.158394-18-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"[PATCH v3 17/20] fsck: verify commit-graph","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:26:03Z","receivedAt":"2018-05-24T16:26:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"If core.commitGraph is true, verify the contents of the commit-graph\nduring 'git fsck' using the 'git commit-graph verify' subcommand. Run\nthis check on all alternates, as well.\n\nWe use a new process for two reasons:\n\n1. The subcommand decouples the details of loading and verifying a\n commit-graph file from the other fsck details.\n\n2. The commit-graph verification requires the commits to be loaded\n in a specific order to guarantee we parse from the commit-graph\n file for some objects and from the object database for others.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-fsck.txt | 3 +++\n builtin/fsck.c | 21 +++++++++++++++++++++\n t/t5318-commit-graph.sh | 8 ++++++++\n 3 files changed, 32 insertions(+)\n\ndiff --git a/Documentation/git-fsck.txt b/Documentation/git-fsck.txt\nindex b9f060e3b2..ab9a93fb9b 100644\n--- a/Documentation/git-fsck.txt\n+++ b/Documentation/git-fsck.txt\n@@ -110,6 +110,9 @@ Any corrupt objects you will have to find in backups or other archives\n (i.e., you can just remove them and do an 'rsync' with some other site in\n the hopes that somebody else has the object you have corrupted).\n \n+If core.commitGraph is true, the commit-graph file will also be inspected\n+using 'git commit-graph verify'. See linkgit:git-commit-graph[1].\n+\n Extracted Diagnostics\n ---------------------\n \ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex ef78c6c00c..a6d5045b77 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -16,6 +16,7 @@\n #include \"streaming.h\"\n #include \"decorate.h\"\n #include \"packfile.h\"\n+#include \"run-command.h\"\n \n #define REACHABLE 0x0001\n #define SEEN 0x0002\n@@ -45,6 +46,7 @@ static int name_objects;\n #define ERROR_REACHABLE 02\n #define ERROR_PACK 04\n #define ERROR_REFS 010\n+#define ERROR_COMMIT_GRAPH 020\n \n static const char *describe_object(struct object *obj)\n {\n@@ -815,5 +817,24 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t}\n \n \tcheck_connectivity();\n+\n+\tif (core_commit_graph) {\n+\t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n+\t\tconst char *verify_argv[] = { \"commit-graph\", \"verify\", NULL, NULL, NULL, NULL };\n+\t\tcommit_graph_verify.argv = verify_argv;\n+\t\tcommit_graph_verify.git_cmd = 1;\n+\n+\t\tif (run_command(&commit_graph_verify))\n+\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n+\n+\t\tprepare_alt_odb();\n+\t\tfor (alt = alt_odb_list; alt; alt = alt->next) {\n+\t\t\tverify_argv[2] = \"--object-dir\";\n+\t\t\tverify_argv[3] = alt->path;\n+\t\t\tif (run_command(&commit_graph_verify))\n+\t\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n+\t\t}\n+\t}\n+\n \treturn errors_found;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 2680a2ebff..4941937163 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -394,4 +394,12 @@ test_expect_success 'detect invalid checksum hash' '\n \t\t\"incorrect checksum\"\n '\n \n+test_expect_success 'git fsck (checks commit-graph)' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit fsck &&\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_FOOTER \"\\00\" \\\n+\t\t\"incorrect checksum\" &&\n+\ttest_must_fail git fsck\n+'\n+\n test_done\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348455","messageId":"20180524162504.158394-16-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"[PATCH v3 15/20] commit-graph: test for corrupted octopus edge","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:25:59Z","receivedAt":"2018-05-24T16:26:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file has an extra chunk to store the parent int-ids for\nparents beyond the first parent for octopus merges. Our test repo has a\nsingle octopus merge that we can manipulate to demonstrate the 'verify'\nsubcommand detects incorrect values in that chunk.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5318-commit-graph.sh | 10 ++++++++++\n 1 file changed, 10 insertions(+)\n\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 58adb8246d..240aef6add 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -248,6 +248,7 @@ test_expect_success 'git commit-graph verify' '\n '\n \n NUM_COMMITS=9\n+NUM_OCTOPUS_EDGES=2\n HASH_LEN=20\n GRAPH_BYTE_VERSION=4\n GRAPH_BYTE_HASH=5\n@@ -274,6 +275,10 @@ GRAPH_BYTE_COMMIT_EXTRA_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4`\n GRAPH_BYTE_COMMIT_WRONG_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3`\n GRAPH_BYTE_COMMIT_GENERATION=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 8`\n GRAPH_BYTE_COMMIT_DATE=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 12`\n+GRAPH_COMMIT_DATA_WIDTH=`expr $HASH_LEN + 16`\n+GRAPH_OCTOPUS_DATA_OFFSET=`expr $GRAPH_COMMIT_DATA_OFFSET + \\\n+\t\t\t\t$GRAPH_COMMIT_DATA_WIDTH \\* $NUM_COMMITS`\n+GRAPH_BYTE_OCTOPUS=`expr $GRAPH_OCTOPUS_DATA_OFFSET + 4`\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -378,4 +383,9 @@ test_expect_success 'detect incorrect commit date' '\n \t\t\"commit date\"\n '\n \n+test_expect_success 'detect incorrect parent for octopus merge' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_OCTOPUS \"\\01\" \\\n+\t\t\"invalid parent\"\n+'\n+\n test_done\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348456","messageId":"20180524162504.158394-19-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"[PATCH v3 18/20] commit-graph: add '--reachable' option","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:26:06Z","receivedAt":"2018-05-24T16:27:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When writing commit-graph files, it can be convenient to ask for all\nreachable commits (starting at the ref set) in the resulting file. This\nis particularly helpful when writing to stdin is complicated, such as a\nfuture integration with 'git gc' which will call\nwrite_commit_graph_reachable() after performing cleanup of the object\ndatabase.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 8 ++++++--\n builtin/commit-graph.c | 16 ++++++++++++----\n commit-graph.c | 32 ++++++++++++++++++++++++++++++++\n commit-graph.h | 1 +\n t/t5318-commit-graph.sh | 10 ++++++++++\n 5 files changed, 61 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex a222cfab08..dececb79d7 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -38,12 +38,16 @@ Write a commit graph file based on the commits found in packfiles.\n +\n With the `--stdin-packs` option, generate the new commit graph by\n walking objects only in the specified pack-indexes. (Cannot be combined\n-with --stdin-commits.)\n+with `--stdin-commits` or `--reachable`.)\n +\n With the `--stdin-commits` option, generate the new commit graph by\n walking commits starting at the commits specified in stdin as a list\n of OIDs in hex, one OID per line. (Cannot be combined with\n---stdin-packs.)\n+`--stdin-packs` or `--reachable`.)\n++\n+With the `--reachable` option, generate the new commit graph by walking\n+commits starting at all refs. (Cannot be combined with `--stdin-commits`\n+or `--stdin-packs`.)\n +\n With the `--append` option, include all commits that are present in the\n existing commit-graph file.\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 0433dd6e20..20ce6437ae 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -9,7 +9,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph verify [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -24,12 +24,13 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n static struct opts_commit_graph {\n \tconst char *obj_dir;\n+\tint reachable;\n \tint stdin_packs;\n \tint stdin_commits;\n \tint append;\n@@ -130,6 +131,8 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n \t\t\tN_(\"dir\"),\n \t\t\tN_(\"The object directory to store the graph\")),\n+\t\tOPT_BOOL(0, \"reachable\", &opts.reachable,\n+\t\t\tN_(\"start walk at all refs\")),\n \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n \t\t\tN_(\"scan pack-indexes listed by stdin for commits\")),\n \t\tOPT_BOOL(0, \"stdin-commits\", &opts.stdin_commits,\n@@ -143,11 +146,16 @@ static int graph_write(int argc, const char **argv)\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\n \n-\tif (opts.stdin_packs && opts.stdin_commits)\n-\t\tdie(_(\"cannot use both --stdin-commits and --stdin-packs\"));\n+\tif (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)\n+\t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n \n+\tif (opts.reachable) {\n+\t\twrite_commit_graph_reachable(opts.obj_dir, opts.append);\n+\t\treturn 0;\n+\t}\n+\n \tif (opts.stdin_packs || opts.stdin_commits) {\n \t\tstruct strbuf buf = STRBUF_INIT;\n \t\tlines_nr = 0;\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a33600c584..057d734926 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -6,6 +6,7 @@\n #include \"packfile.h\"\n #include \"commit.h\"\n #include \"object.h\"\n+#include \"refs.h\"\n #include \"revision.h\"\n #include \"sha1-lookup.h\"\n #include \"commit-graph.h\"\n@@ -651,6 +652,37 @@ static void compute_generation_numbers(struct packed_commit_list* commits)\n \t}\n }\n \n+struct hex_list {\n+\tchar **hex_strs;\n+\tint hex_nr;\n+\tint hex_alloc;\n+};\n+\n+static int add_ref_to_list(const char *refname,\n+\t\t\t const struct object_id *oid,\n+\t\t\t int flags, void *cb_data)\n+{\n+\tstruct hex_list *list = (struct hex_list*)cb_data;\n+\n+\tALLOC_GROW(list->hex_strs, list->hex_nr + 1, list->hex_alloc);\n+\tlist->hex_strs[list->hex_nr] = xcalloc(GIT_MAX_HEXSZ + 1, 1);\n+\tstrcpy(list->hex_strs[list->hex_nr], oid_to_hex(oid));\n+\tlist->hex_nr++;\n+\treturn 0;\n+}\n+\n+void write_commit_graph_reachable(const char *obj_dir, int append)\n+{\n+\tstruct hex_list list;\n+\tlist.hex_nr = 0;\n+\tlist.hex_alloc = 128;\n+\tALLOC_ARRAY(list.hex_strs, list.hex_alloc);\n+\n+\tfor_each_ref(add_ref_to_list, &list);\n+\n+\twrite_commit_graph(obj_dir, NULL, 0, (const char **)list.hex_strs, list.hex_nr, append);\n+}\n+\n void write_commit_graph(const char *obj_dir,\n \t\t\tconst char **pack_indexes,\n \t\t\tint nr_packs,\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 71a39c5a57..9a06a5f188 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -46,6 +46,7 @@ struct commit_graph {\n \n struct commit_graph *load_commit_graph_one(const char *graph_file);\n \n+void write_commit_graph_reachable(const char *obj_dir, int append);\n void write_commit_graph(const char *obj_dir,\n \t\t\tconst char **pack_indexes,\n \t\t\tint nr_packs,\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 4941937163..a659620332 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -205,6 +205,16 @@ test_expect_success 'build graph from commits with append' '\n graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n \n+test_expect_success 'build graph using --reachable' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph write --reachable &&\n+\ttest_path_is_file $objdir/info/commit-graph &&\n+\tgraph_read_expect \"11\" \"large_edges\"\n+'\n+\n+graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n+graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n+\n test_expect_success 'setup bare repo' '\n \tcd \"$TRASH_DIRECTORY\" &&\n \tgit clone --bare --no-local full bare &&\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348457","messageId":"20180524162504.158394-14-dstolee@microsoft.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"[PATCH v3 13/20] commit-graph: verify generation number","fromName":"Derrick Stolee","fromEmail":"dstolee@microsoft.com","sentAt":"2018-05-24T16:25:54Z","receivedAt":"2018-05-24T16:27:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"While iterating through the commit parents, perform the generation\nnumber calculation and compare against the value stored in the\ncommit-graph.\n\nThe tests demonstrate that having a different set of parents affects\nthe generation number calculation, and this value propagates to\ndescendants. Hence, we drop the single-line condition on the output.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 18 ++++++++++++++++++\n t/t5318-commit-graph.sh | 6 ++++++\n 2 files changed, 24 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex fff22dc0c3..ead92460c1 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -922,6 +922,7 @@ int verify_commit_graph(struct commit_graph *g)\n \tfor (i = 0; i < g->num_commits; i++) {\n \t\tstruct commit *graph_commit, *odb_commit;\n \t\tstruct commit_list *graph_parents, *odb_parents;\n+\t\tuint32_t max_generation = 0;\n \n \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n \n@@ -956,6 +957,9 @@ int verify_commit_graph(struct commit_graph *g)\n \t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid),\n \t\t\t\t\t oid_to_hex(&odb_parents->item->object.oid));\n \n+\t\t\tif (graph_parents->item->generation > max_generation)\n+\t\t\t\tmax_generation = graph_parents->item->generation;\n+\n \t\t\tgraph_parents = graph_parents->next;\n \t\t\todb_parents = odb_parents->next;\n \t\t}\n@@ -963,6 +967,20 @@ int verify_commit_graph(struct commit_graph *g)\n \t\tif (odb_parents != NULL)\n \t\t\tgraph_report(\"commit-graph parent list for commit %s terminates early\",\n \t\t\t\t oid_to_hex(&cur_oid));\n+\n+\t\t/*\n+\t\t * If one of our parents has generation GENERATION_NUMBER_MAX, then\n+\t\t * our generation is also GENERATION_NUMBER_MAX. Decrement to avoid\n+\t\t * extra logic in the following condition.\n+\t\t */\n+\t\tif (max_generation == GENERATION_NUMBER_MAX)\n+\t\t\tmax_generation--;\n+\n+\t\tif (graph_commit->generation != max_generation + 1)\n+\t\t\tgraph_report(\"commit-graph generation for commit %s is %u != %u\",\n+\t\t\t\t oid_to_hex(&cur_oid),\n+\t\t\t\t graph_commit->generation,\n+\t\t\t\t max_generation + 1);\n \t}\n \n \treturn verify_commit_graph_error;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 12f0d7f54d..673b0d37d5 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -272,6 +272,7 @@ GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n GRAPH_BYTE_COMMIT_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN`\n GRAPH_BYTE_COMMIT_EXTRA_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4`\n GRAPH_BYTE_COMMIT_WRONG_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3`\n+GRAPH_BYTE_COMMIT_GENERATION=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 8`\n \n # usage: corrupt_graph_and_verify \n # Manipulates the commit-graph file at the position\n@@ -366,4 +367,9 @@ test_expect_success 'detect incorrect tree OID' '\n \t\t\"commit-graph parent for\"\n '\n \n+test_expect_success 'detect incorrect generation number' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_GENERATION \"\\01\" \\\n+\t\t\"generation\"\n+'\n+\n test_done\n-- \n2.16.2.329.gfb62395de6\n\n"},{"id":"348458","messageId":"33480b33-1baf-e822-981e-c7d4b392aeac@gmail.com","threadId":"48311","inReplyTo":"86wovwdemu.fsf@gmail.com","subject":"Re: [PATCH v2 03/12] commit-graph: test that 'verify' finds corruption","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-24T16:28:14Z","receivedAt":"2018-05-24T16:28:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/21/2018 2:53 PM, Jakub Narebski wrote:\n>> +corrupt_data() {\n>> +\tfile=$1\n>> +\tpos=$2\n>> +\tdata=\"${3:-\\0}\"\n>> +\tprintf \"$data\" | dd of=\"$file\" bs=1 seek=\"$pos\" conv=notrunc\n>> +}\n> First, if we do this that way (and not by adding a test helper), the use\n> of this function should be, I think, protected using appropriate test\n> prerequisite. Not everyone has 'dd' tool installed, for example on\n> MS Windows.\n\nWindows does not, but it is also missing many things this test suite \nneeds. 'dd' is included in the Git for Windows SDK. I rebased this \nseries onto Git for Windows and the tests passed when run in an SDK shell.\n\nThanks,\n-Stolee\n"},{"id":"348492","messageId":"87po1k7o1u.fsf@evledraar.gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"Re: [PATCH v3 00/20] Integrate commit-graph into 'fsck' and 'gc'","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-05-24T21:15:41Z","receivedAt":"2018-05-24T21:15:48Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, May 24 2018, Derrick Stolee wrote:\n\n> Thanks for all the feedback on v2. I've tried to make this round's\n> review a bit easier by splitting up the commits into smaller pieces.\n> Also, the test script now has less boilerplate and uses variables and\n> clear arithmetic to explain which bytes are being modified.\n\nThanks. it's a lot easier.\n\n> One other change worth mentioning: in \"commit-graph: add '--reachable'\n> option\" I put the ref-iteration into a new external\n> 'write_commit_graph_reachable()' method inside commit-graph.c. This\n> makes the 'gc: automatically write commit-graph files' a simpler change.\n\nMaybe you want this, maybe not, but I came up with this to squash:\n\n diff --git a/Documentation/config.txt b/Documentation/config.txt\n index 9a3abd87e7..2665522385 100644\n --- a/Documentation/config.txt\n +++ b/Documentation/config.txt\n @@ -900,7 +900,8 @@ the `GIT_NOTES_REF` environment variable. See linkgit:git-notes[1].\n\n core.commitGraph::\n Enable git commit graph feature. Allows reading from the\n - commit-graph file.\n + commit-graph file. See `gc.commitGraph` for automatically\n + maintaining the file.\n\n core.sparseCheckout::\n Enable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n @@ -1554,10 +1555,10 @@ gc.autoDetach::\n if the system supports it. Default is true.\n\n gc.commitGraph::\n - If true, then gc will rewrite the commit-graph file after any\n - change to the object database. If '--auto' is used, then the\n - commit-graph will not be updated unless the threshold is met.\n - See linkgit:git-commit-graph[1] for details.\n + If true, then gc will rewrite the commit-graph file when\n + linkgit:git-gc[1] is run. When using linkgit:git-gc[1]\n + '--auto' the commit-graph will be updated if housekeeping is\n + required. See linkgit:git-commit-graph[1] for details.\n\n gc.logExpiry::\n If the file gc.log exists, then `git gc --auto` won't run\n\nI.e. let's mention the new gc.commitGraph in core.commitGraph, and I\nthink the \"any change to the object database\" line in gc.commitGraph is\nneedlessly confusing, let's just say \"when git-gc is run\".\n"},{"id":"348495","messageId":"CAGZ79kYaC3VsH2kGMvsv0Xn-ghu5t7ENe+yebQCS5nvW5ms1_g@mail.gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-2-dstolee@microsoft.com","subject":"Re: [PATCH v3 01/20] commit-graph: UNLEAK before die()","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-05-24T22:47:16Z","receivedAt":"2018-05-24T22:47:21Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, May 24, 2018 at 9:25 AM, Derrick Stolee wrote:\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/commit-graph.c | 5 ++++-\n> 1 file changed, 4 insertions(+), 1 deletion(-)\n>\n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index 37420ae0fd..f0875b8bf3 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -51,8 +51,11 @@ static int graph_read(int argc, const char **argv)\n> graph_name = get_commit_graph_filename(opts.obj_dir);\n> graph = load_commit_graph_one(graph_name);\n>\n> - if (!graph)\n> + if (!graph) {\n> + UNLEAK(graph_name);\n> die(\"graph file %s does not exist\", graph_name);\n\nUnrelated to this patch: Is the command that ends up die()ing here\na plumbing or porcelain, or: Do we want to translate the message here?\n\nIn a lot of commands that show paths we single quote them '%s',\n(speaking from experience with a lot of submodule path code)\n"},{"id":"348501","messageId":"67f7c4ef-d82e-fe66-b4cb-d118e6366c54@gmail.com","threadId":"48311","inReplyTo":"CAGZ79kYaC3VsH2kGMvsv0Xn-ghu5t7ENe+yebQCS5nvW5ms1_g@mail.gmail.com","subject":"Re: [PATCH v3 01/20] commit-graph: UNLEAK before die()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-25T00:08:17Z","receivedAt":"2018-05-25T00:10:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/24/2018 6:47 PM, Stefan Beller wrote:\n> On Thu, May 24, 2018 at 9:25 AM, Derrick Stolee wrote:\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> builtin/commit-graph.c | 5 ++++-\n>> 1 file changed, 4 insertions(+), 1 deletion(-)\n>>\n>> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n>> index 37420ae0fd..f0875b8bf3 100644\n>> --- a/builtin/commit-graph.c\n>> +++ b/builtin/commit-graph.c\n>> @@ -51,8 +51,11 @@ static int graph_read(int argc, const char **argv)\n>> graph_name = get_commit_graph_filename(opts.obj_dir);\n>> graph = load_commit_graph_one(graph_name);\n>>\n>> - if (!graph)\n>> + if (!graph) {\n>> + UNLEAK(graph_name);\n>> die(\"graph file %s does not exist\", graph_name);\n> Unrelated to this patch: Is the command that ends up die()ing here\n> a plumbing or porcelain, or: Do we want to translate the message here?\n>\n> In a lot of commands that show paths we single quote them '%s',\n> (speaking from experience with a lot of submodule path code)\n\nThis is for the 'git commit-graph read' command, which is plumbing (and \n'read' is really only for testing). I don't think this message requires \ntranslation.\n\nI'll keep the quotes in mind for the future.\n\nThanks,\n\n-Stolee\n\n"},{"id":"348515","messageId":"xmqqtvqwnzmi.fsf@gitster-ct.c.googlers.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"Re: [PATCH v3 00/20] Integrate commit-graph into 'fsck' and 'gc'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-05-25T04:11:17Z","receivedAt":"2018-05-25T04:11:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> One other change worth mentioning: in \"commit-graph: add '--reachable'\n> option\" I put the ref-iteration into a new external\n> 'write_commit_graph_reachable()' method inside commit-graph.c. This\n> makes the 'gc: automatically write commit-graph files' a simpler change.\n\n;-).\n"},{"id":"348605","messageId":"868t861ci6.fsf@gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-3-dstolee@microsoft.com","subject":"Re: [PATCH v3 02/20] commit-graph: fix GRAPH_MIN_SIZE","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-05-26T18:46:09Z","receivedAt":"2018-05-26T18:46:15Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> The GRAPH_MIN_SIZE macro should be the smallest size of a parsable\n> commit-graph file. However, the minimum number of chunks was wrong.\n> It is possible to write a commit-graph file with zero commits, and\n> that violates this macro's value.\n>\n> Rewrite the macro, and use extra macros to better explain the magic\n> constants.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 5 +++--\n> 1 file changed, 3 insertions(+), 2 deletions(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index a8c337dd77..82295f0975 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -33,10 +33,11 @@\n> \n> #define GRAPH_LAST_EDGE 0x80000000\n> \n> +#define GRAPH_HEADER_SIZE 8\n\nNice.\n\n> #define GRAPH_FANOUT_SIZE (4 * 256)\n> #define GRAPH_CHUNKLOOKUP_WIDTH 12\n> -#define GRAPH_MIN_SIZE (5 * GRAPH_CHUNKLOOKUP_WIDTH + GRAPH_FANOUT_SIZE + \\\n> -\t\t\tGRAPH_OID_LEN + 8)\n> +#define GRAPH_MIN_SIZE (GRAPH_HEADER_SIZE + 4 * GRAPH_CHUNKLOOKUP_WIDTH \\\n> +\t\t\t+ GRAPH_FANOUT_SIZE + GRAPH_OID_LEN)\n\nSo in this case we have file header (4-byte signature, 1-byte version\nnumber, 1-byte oid/hash version, 1-byte number of chunks, 1-byte\nreserved: 4+1+1+1+1 = 8 bytes), chunk lookup: 3 required chunks plus\nterminating label = 4 entries, constant-size fanout chunks, and\nchecksum. Two remaining required chunks (OID Lookup and Commit Data)\ncan have length of 0.\n\n\nOne issue: in the future when Git moves to NewHash, it could encounter\nthen both commit-graph files using SHA-1 and using NewHash. What about\nGRPH_OID_LEN then: for one of those it would be incorrect. Unless it is\nabout minimal length of checksum, that is we assume that NewHash would\nbe longer than SHA-1, but ten why name it GRAPH_OID_LEN?\n\nThis may be going too much in the future; there is no need to borrow\ntrouble now, where we have only SHA-1 supported as OID. Still...\n\n> \n> char *get_commit_graph_filename(const char *obj_dir)\n> {\n\nBest,\n-- \nJakub Narębski\n"},{"id":"348607","messageId":"20180526203036.GL652292@genre.crustytoothpaste.net","threadId":"48311","inReplyTo":"868t861ci6.fsf@gmail.com","subject":"Re: [PATCH v3 02/20] commit-graph: fix GRAPH_MIN_SIZE","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-05-26T20:30:36Z","receivedAt":"2018-05-26T20:30:49Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Sat, May 26, 2018 at 08:46:09PM +0200, Jakub Narebski wrote:\n> One issue: in the future when Git moves to NewHash, it could encounter\n> then both commit-graph files using SHA-1 and using NewHash. What about\n> GRPH_OID_LEN then: for one of those it would be incorrect. Unless it is\n> about minimal length of checksum, that is we assume that NewHash would\n> be longer than SHA-1, but ten why name it GRAPH_OID_LEN?\n\nMy proposal is that whatever we're using in the .git directory is\nconsistent. If we're using SHA-1 for objects, then everything is SHA-1.\nIf we're using NewHash for objects, then all data is stored in NewHash\n(except translation tables and such). Any conversions between SHA-1 and\nNewHash require a lookup through the standard techniques.\n\nI agree that here it would be more helpful if it were a reference to\nthe_hash_algo, and I've applied a patch to my object-id-part14 series to\nmake that conversion.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"348626","messageId":"86y3g5z9ah.fsf@gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-4-dstolee@microsoft.com","subject":"Re: [PATCH v3 03/20] commit-graph: parse commit from chosen graph","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-05-27T10:23:50Z","receivedAt":"2018-05-27T10:25:13Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> Before verifying a commit-graph file against the object database, we\n> need to parse all commits from the given commit-graph file. Create\n> parse_commit_in_graph_one() to target a given struct commit_graph.\n\nIf I understand it properly the problem is that when verifying against\nthe object database we want to check one single commit-graph file, not\nconcatenation of data from commit-graph file for the repository and\ncommit-graph files from its alternates -- like prepare_commit_graph()\ndoes; which is called by parse_commit_in_graph().\n\n>\n> Signed-off-by: Derrick Stolee \n\nO.K., so you introduce here a layer of indirection; parse_commit_in_graph()\nnow just uses parse_commit_in_graph_one(), passing core_commit_graph\n(or the_commit_graph) to it, after checking that core_commit_graph is set\n(which handles the case when feature is not turned off) and loading\ncommit-graph file.\n\nNice and simple 'split function' refactoring, with new function taking\nover when there is commit graph file prepared.\n\n\nSo, after the changes:\n* parse_commit_in_graph() is responsible for checking whether to use\n commit-graph feature and ensuring that data from commit-graph is\n loaded, where it passes the control to parse_commit_in_graph_one()\n* parse_commit_in_graph_one() checks whether commit-graph feature is\n turned on, whether commit we are interested in was already parsed,\n and then uses fill_commit_in_graph() to actually get the data\n* fill_commit_in_graph() gets data out of commit-graph file, extracting\n it from commit data chunk (and if needed large edges chunk).\n\nAll those functions return 1 if they got data from commit-graph, and 0\nif they didn't.\n\n\nOne minor nitpick / complaint / question is about naming of global\nvariables used here, namely:\n* static struct commit_graph *commit_graph\n from commit-graph.c for global storage of commit-graph[s] data\n* int core_commit_graph\n from environment.c for storing core.commitGraph config\n\nBut I see that at least the latter is common convention in Git source\ncode; I guess that the former maybe follows convention as used for \"the\nindex\" and \"the repository\" - additionally it is static / file-local.\n\n> ---\n> commit-graph.c | 18 +++++++++++++++---\n> 1 file changed, 15 insertions(+), 3 deletions(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 82295f0975..78ba0edc80 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -310,7 +310,7 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n> \t}\n> }\n> \n> -int parse_commit_in_graph(struct commit *item)\n> +static int parse_commit_in_graph_one(struct commit_graph *g, struct commit *item)\n> {\n> \tuint32_t pos;\n> \n> @@ -318,9 +318,21 @@ static int parse_commit_in_graph_one(struct commit_graph *g, struct commit *item)\n> \tif (!core_commit_graph)\n> \t\treturn 0;\n\nAll right, we check that commit-graph feature is enabled because\nparse_commit_in_graph_one() will be used standalone, not by being\ninvoked from parse_commit_in_graph().\n\nThis check is fast.\n\n> \tif (item->object.parsed)\n> \t\treturn 1;\n\nSidenote: I just wonder why object.parsed and not for example\nobject.graph_pos is used to checck whether the object was filled if\npossible with commit-graph data...\n\n> +\n> +\tif (find_commit_in_graph(item, g, &pos))\n> +\t\treturn fill_commit_in_graph(item, g, pos);\n> +\n> +\treturn 0;\n> +}\n> +\n> +int parse_commit_in_graph(struct commit *item)\n> +{\n> +\tif (!core_commit_graph)\n> +\t\treturn 0;\n\nAll right, this check is here to short-circuit and make it so git does\nnot even try to lod commit-graph file[s] if the feature is disabled.\n\n> +\n> \tprepare_commit_graph();\n> -\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n> -\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n> +\tif (commit_graph)\n> +\t\treturn parse_commit_in_graph_one(commit_graph, item);\n> \treturn 0;\n> }\n"},{"id":"348637","messageId":"86o9h1ynyu.fsf@gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-5-dstolee@microsoft.com","subject":"Re: [PATCH v3 04/20] commit: force commit to parse from object database","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-05-27T18:04:25Z","receivedAt":"2018-05-27T18:04:32Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> In anticipation of verifying commit-graph file contents against the\n> object database, create parse_commit_internal() to allow side-stepping\n> the commit-graph file and parse directly from the object database.\n>\n> Due to the use of generation numbers, this method should not be called\n> unless the intention is explicit in avoiding commits from the\n> commit-graph file.\n\nA straightforward addition of a parameter to parse_commit() and renaming\nit to parse_commit_internal(), while changing parse_commit() to be a\nsimple wrapper around newly introduced parse_commit_internal(), passing\nthe default arguments.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit.c | 9 +++++++--\n> commit.h | 1 +\n> 2 files changed, 8 insertions(+), 2 deletions(-)\n\nNice and simple refactoring in preparation for future changes.\n\n>\n> diff --git a/commit.c b/commit.c\n> index 1d28677dfb..6eaed0174c 100644\n> --- a/commit.c\n> +++ b/commit.c\n> @@ -392,7 +392,7 @@ int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long s\n> \treturn 0;\n> }\n> \n> -int parse_commit_gently(struct commit *item, int quiet_on_missing)\n> +int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph)\n\nI guess that the \"new\" parse_commit_internal() function was not made\nstatic despite the *_internal() in the name because it would need to be\nused from commit-graph.c, isn't it?\n\nI don't think we would need more similar options, so *_with_flags()\nwould be YAGNI overkill.\n\n> {\n> \tenum object_type type;\n> \tvoid *buffer;\n> @@ -403,7 +403,7 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n> \t\treturn -1;\n> \tif (item->object.parsed)\n> \t\treturn 0;\n> -\tif (parse_commit_in_graph(item))\n> +\tif (use_commit_graph && parse_commit_in_graph(item))\n> \t\treturn 0;\n> \tbuffer = read_sha1_file(item->object.oid.hash, &type, &size);\n> \tif (!buffer)\n> @@ -424,6 +424,11 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n> \treturn ret;\n> }\n> \n> +int parse_commit_gently(struct commit *item, int quiet_on_missing)\n> +{\n> +\treturn parse_commit_internal(item, quiet_on_missing, 1);\n> +}\n> +\n> void parse_commit_or_die(struct commit *item)\n> {\n> \tif (parse_commit(item))\n> diff --git a/commit.h b/commit.h\n> index b5afde1ae9..5fde74fcd7 100644\n> --- a/commit.h\n> +++ b/commit.h\n> @@ -73,6 +73,7 @@ struct commit *lookup_commit_reference_by_name(const char *name);\n> struct commit *lookup_commit_or_die(const struct object_id *oid, const char *ref_name);\n> \n> int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size, int check_graph);\n> +int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph);\n> int parse_commit_gently(struct commit *item, int quiet_on_missing);\n> static inline int parse_commit(struct commit *item)\n> {\n"},{"id":"348639","messageId":"86h8mszzdy.fsf@gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-6-dstolee@microsoft.com","subject":"Re: [PATCH v3 05/20] commit-graph: load a root tree from specific graph","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-05-27T19:12:25Z","receivedAt":"2018-05-27T19:12:32Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> When lazy-loading a tree for a commit, it will be important to select\n> the tree from a specific struct commit_graph. Create a new method that\n> specifies the commit-graph file and use that in\n> get_commit_tree_in_graph().\n\nIs this for the same reason why parse_commit_in_graph_one() was created\nin ptch 03/20? Why it would be important?\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 12 +++++++++---\n> 1 file changed, 9 insertions(+), 3 deletions(-)\n\nSimple and straightforward refactoring in the same vein as\nparse_commit_in_graph_one() one.\n\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 78ba0edc80..25893ec096 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -358,14 +358,20 @@ static struct tree *load_tree_for_commit(struct commit_graph *g, struct commit *\n> \treturn c->maybe_tree;\n> }\n> \n> -struct tree *get_commit_tree_in_graph(const struct commit *c)\n> +static struct tree *get_commit_tree_in_graph_one(struct commit_graph *g,\n> +\t\t\t\t\t\t const struct commit *c)\n> {\n> \tif (c->maybe_tree)\n> \t\treturn c->maybe_tree;\n> \tif (c->graph_pos == COMMIT_NOT_FROM_GRAPH)\n> -\t\tBUG(\"get_commit_tree_in_graph called from non-commit-graph commit\");\n> +\t\tBUG(\"get_commit_tree_in_graph_one called from non-commit-graph commit\");\n\nSidenote: I wonder if it would be better or worse to use __func__ magic\ncostant variable here (part of C99 and C++11 standards).\n\n> +\n> +\treturn load_tree_for_commit(g, (struct commit *)c);\n> +}\n> \n> -\treturn load_tree_for_commit(commit_graph, (struct commit *)c);\n> +struct tree *get_commit_tree_in_graph(const struct commit *c)\n> +{\n> +\treturn get_commit_tree_in_graph_one(commit_graph, c);\n> }\n> \n> static void write_graph_chunk_fanout(struct hashfile *f,\n\nLooks good to me.\n"},{"id":"348641","messageId":"86y3g4yahm.fsf@gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-7-dstolee@microsoft.com","subject":"Re: [PATCH v3 06/20] commit-graph: add 'verify' subcommand","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-05-27T22:55:33Z","receivedAt":"2018-05-27T22:55:40Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> If the commit-graph file becomes corrupt, we need a way to verify\n> that its contents match the object database. In the manner of\n> 'git fsck' we will implement a 'git commit-graph verify' subcommand\n> to report all issues with the file.\n>\n> Add the 'verify' subcommand to the 'commit-graph' builtin and its\n> documentation. The subcommand is currently a no-op except for\n> loading the commit-graph into memory, which may trigger run-time\n> errors that would be caught by normal use.\n\nSo this commit is simply getting the boilerplate out of the way for\nimplementing 'git commit-graph verify' subcommand. Good.\n\n> Add a simple test that\n> ensures the command returns a zero error code.\n\nNice.\n\n>\n> If no commit-graph file exists, this is an acceptable state. Do\n> not report any errors.\n\nAll right. I assume that as it is explicit verification call, it does\nignore core.commitGraph setting, isn't it?\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-commit-graph.txt | 6 ++++++\n> builtin/commit-graph.c | 38 ++++++++++++++++++++++++++++++++++++++\n> commit-graph.c | 26 ++++++++++++++++++++++++++\n> commit-graph.h | 2 ++\n> t/t5318-commit-graph.sh | 10 ++++++++++\n> 5 files changed, 82 insertions(+)\n>\n> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> index 4c97b555cc..a222cfab08 100644\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -10,6 +10,7 @@ SYNOPSIS\n> --------\n> [verse]\n> 'git commit-graph read' [--object-dir ]\n> +'git commit-graph verify' [--object-dir ]\n> 'git commit-graph write' [--object-dir ]\n\nIn alphabetical order, good.\n\n> \n> \n> @@ -52,6 +53,11 @@ existing commit-graph file.\n> Read a graph file given by the commit-graph file and output basic\n> details about the graph file. Used for debugging purposes.\n> \n> +'verify'::\n> +\n> +Read the commit-graph file and verify its contents against the object\n> +database. Used to check for corrupted data.\n> +\n\nAll right, good enough description.\n\n> \n> EXAMPLES\n> --------\n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index f0875b8bf3..0433dd6e20 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -8,10 +8,16 @@\n> static char const * const builtin_commit_graph_usage[] = {\n> \tN_(\"git commit-graph [--object-dir ]\"),\n> \tN_(\"git commit-graph read [--object-dir ]\"),\n> +\tN_(\"git commit-graph verify [--object-dir ]\"),\n> \tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n> \tNULL\n> };\n\nIn alphabetical order, same as in the manpage for git-commit-graph.\n\n> \n> +static const char * const builtin_commit_graph_verify_usage[] = {\n> +\tN_(\"git commit-graph verify [--object-dir ]\"),\n> +\tNULL\n> +};\n> +\n> static const char * const builtin_commit_graph_read_usage[] = {\n> \tN_(\"git commit-graph read [--object-dir ]\"),\n> \tNULL\n> @@ -29,6 +35,36 @@ static struct opts_commit_graph {\n> \tint append;\n> } opts;\n> \n> +\n> +static int graph_verify(int argc, const char **argv)\n> +{\n> +\tstruct commit_graph *graph = 0;\n> +\tchar *graph_name;\n> +\n> +\tstatic struct option builtin_commit_graph_verify_options[] = {\n> +\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n> +\t\t\t N_(\"dir\"),\n> +\t\t\t N_(\"The object directory to store the graph\")),\n> +\t\tOPT_END(),\n> +\t};\n> +\n> +\targc = parse_options(argc, argv, NULL,\n> +\t\t\t builtin_commit_graph_verify_options,\n> +\t\t\t builtin_commit_graph_verify_usage, 0);\n> +\n> +\tif (!opts.obj_dir)\n> +\t\topts.obj_dir = get_object_directory();\n\nGetting the boilerplate of implementing the command mostly out of the\nway. Good.\n\n> +\n> +\tgraph_name = get_commit_graph_filename(opts.obj_dir);\n> +\tgraph = load_commit_graph_one(graph_name);\n\nSo we are verifying only the commit-graph file belonging directly to\ncurrent repository, as I have expected. This is needed to for warnings\nand error messages from the 'verify' action, to be able to tell in which\nfile there are problems.\n\nThis means that it is possible that there would be problems with\ncommit-graph files that running 'git commit-graph verify' would not\nfind, because they are in commit-graph file in one of the alternates.\n\nIt is very easy, though, to check all commit-graph files that would be\nread and its data concatenated when using commit-graph feature\n(e.g. 'git commit-graph read', IIRC):\n\n $ git commit-graph verify\n $ for obj_dir in $(cat .git/objects/info/alternates) do;\n git commit-graph --object-dir=\"$obj_dir\";\n done\n\nNote: I have not checked the above that it works.\n\n> +\tFREE_AND_NULL(graph_name);\n\nFreeing the resources, always nice to have.\n\n> +\n> +\tif (!graph)\n> +\t\treturn 0;\n\nDS> If no commit-graph file exists, this is an acceptable state. Do\nDS> not report any errors.\n\nRight, non existant commit-graph file is certainly valid ;-)\n\n> +\n> +\treturn verify_commit_graph(graph);\n> +}\n\nI guess that graph_verify() would not change much, if at all, in\nsubsequent commits in this patch series.\n\n> +\n> static int graph_read(int argc, const char **argv)\n> {\n> \tstruct commit_graph *graph = NULL;\n> @@ -163,6 +199,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n> \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n> \n> \tif (argc > 0) {\n> +\t\tif (!strcmp(argv[0], \"verify\"))\n> +\t\t\treturn graph_verify(argc, argv);\n> \t\tif (!strcmp(argv[0], \"read\"))\n> \t\t\treturn graph_read(argc, argv);\n> \t\tif (!strcmp(argv[0], \"write\"))\n\nNot in alphabetical order... is there a reason for that?\n\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 25893ec096..55b41664ee 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -836,3 +836,29 @@ void write_commit_graph(const char *obj_dir,\n> \toids.alloc = 0;\n> \toids.nr = 0;\n> }\n> +\n> +static int verify_commit_graph_error;\n> +\n> +static void graph_report(const char *fmt, ...)\n> +{\n> +\tva_list ap;\n> +\tstruct strbuf sb = STRBUF_INIT;\n> +\tverify_commit_graph_error = 1;\n> +\n> +\tva_start(ap, fmt);\n> +\tstrbuf_vaddf(&sb, fmt, ap);\n> +\n> +\tfprintf(stderr, \"%s\\n\", sb.buf);\n> +\tstrbuf_release(&sb);\n> +\tva_end(ap);\n\nWhy do you use strbuf_vaddf + fprintf instead of straighforward\nvfprintf (or function instead of variable-level macro)?\n\nIs it because of [string] safety?\n\n> +}\n> +\n> +int verify_commit_graph(struct commit_graph *g)\n> +{\n> +\tif (!g) {\n> +\t\tgraph_report(\"no commit-graph file loaded\");\n> +\t\treturn 1;\n> +\t}\n\nAll right, this is just a placeholder - we should not ever get this\nmessage because in this case we exit with error code of 0 (EXIT_SUCCESS)\nif there is no commit-graph file loaded before invoking\nverify_commit_graph().\n\n> +\n> +\treturn verify_commit_graph_error;\n\nAll right, this is for the future. Good.\n\n> +}\n> diff --git a/commit-graph.h b/commit-graph.h\n> index 96cccb10f3..71a39c5a57 100644\n> --- a/commit-graph.h\n> +++ b/commit-graph.h\n> @@ -53,4 +53,6 @@ void write_commit_graph(const char *obj_dir,\n> \t\t\tint nr_commits,\n> \t\t\tint append);\n> \n> +int verify_commit_graph(struct commit_graph *g);\n> +\n\nWhy does this need to be exported? I think it is not used outside of\ncommit-graph.c, isn't it?\n\n> #endif\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 77d85aefe7..6ca451dfd2 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -11,6 +11,11 @@ test_expect_success 'setup full repo' '\n> \tobjdir=\".git/objects\"\n> '\n> \n> +test_expect_success 'verify graph with no graph file' '\n> +\tcd \"$TRASH_DIRECTORY/full\" &&\n\nIs sich bare `cd`, without corresponding `cd` back or using subshell\nsafe?\n\n> +\tgit commit-graph verify\n> +'\n> +\n> test_expect_success 'write graph with no packs' '\n> \tcd \"$TRASH_DIRECTORY/full\" &&\n> \tgit commit-graph write --object-dir . &&\n> @@ -230,4 +235,9 @@ test_expect_success 'perform fast-forward merge in full repo' '\n> \ttest_cmp expect output\n> '\n> \n> +test_expect_success 'git commit-graph verify' '\n> +\tcd \"$TRASH_DIRECTORY/full\" &&\n> +\tgit commit-graph verify >output\n> +'\n\nThose are tests with nearly the same code, but they are (by their\ndescriptions) testing different things. This means that they rely on\nside effects of earlier tests.\n\nThis is suboptimal, as it means that it would be impossible or very\ndifficult to run individual tests (e.g. with GIT_SKIP_TESTS environment\nvariable, or with an individual test suite --run option), unless you\nknow which tests setup the repository state for later tests.\n\nIt also means that running only failed tests with prove\n--state=failed,save or equivalently with\n\n $ make DEFAULT_TEST_TARGET=prove GIT_PROVE_OPTS='--state=failed,save' test\n\nwouldn't work correctly.\n\nAs Johannes Schindelin (alias Dscho) said in latest Git Rev News\ninterview: https://git.github.io/rev_news/2018/05/16/edition-39/\n\nJS> We have a test suite where debugging a regression may mean that you\nJS> have to run 98 test cases before the failing one every single time in\nJS> the edit/compile/debug cycle, because the 99th test case may depend on\nJS> a side effect of at least one of the preceding test cases. Git’s test\nJS> suite is so not [21st century best practices][1].\nJS>\nJS> [1]: https://www.slideshare.net/BuckHodges/lessons-learned-doing-devops-at-scale-at-microsoft\n\n\nI think can be solved quite efficiently by creating and using shell\nfunction, or two shell functions, which would either:\n\n * rename commit-graph file to some other temporary name if it exists,\n and move it back after the test.\n * create commit-graph file if it does not exist.\n\nFor example (untested):\n\n prepare_no_commit_graph() {\n \tmv .git/info/commit-graph .git/info/commit-graph.away &&\n \ttest_when_finished \"mv .git/info/commit-graph.away .git/info/commit-graph\"\n }\n\n prepare_commit_graph() {\n \tif ! test -f \".git/info/commit-graph\"\n \tthen\n \t\tgit commit-graph write\n \tfi\n }\n\nOr something like that.\n\n> +\n> test_done\n\nRegards,\n-- \nJakub Narębski\n"},{"id":"348671","messageId":"86muwjyixo.fsf@gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-8-dstolee@microsoft.com","subject":"Re: [PATCH v3 07/20] commit-graph: verify catches corrupt signature","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-05-28T14:05:23Z","receivedAt":"2018-05-28T14:07:46Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> This is the first of several commits that add a test to check that\n> 'git commit-graph verify' catches corruption in the commit-graph\n> file. The first test checks that the command catches an error in\n> the file signature. This is a check that exists in the existing\n> commit-graph reading code.\n\nGood start.\n\nThis handles 3 out of 5 checks in load_commit_graph_one(). The\nremaining are:\n* too short file (length smaller than minimal commit-graph size)\n* more than one chunk of one of 4 defined types\n\n> Add a helper method 'corrupt_graph_and_verify' to the test script\n> t5318-commit-graph.sh. This helper corrupts the commit-graph file\n> at a certain location, runs 'git commit-graph verify', and reports\n> the output to the 'err' file. This data is filtered to remove the\n> lines added by 'test_must_fail' when the test is run verbosely.\n> Then, the output is checked to contain a specific error message.\n\nThanks for an explanation.\n\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t5318-commit-graph.sh | 43 +++++++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 43 insertions(+)\n>\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 6ca451dfd2..bd64481c7a 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -235,9 +235,52 @@ test_expect_success 'perform fast-forward merge in full repo' '\n> \ttest_cmp expect output\n> '\n> \n> +# the verify tests below expect the commit-graph to contain\n> +# exactly the commits reachable from the commits/8 branch.\n> +# If the file changes the set of commits in the list, then the\n> +# offsets into the binary file will result in different edits\n> +# and the tests will likely break.\n> +\n> test_expect_success 'git commit-graph verify' '\n> \tcd \"$TRASH_DIRECTORY/full\" &&\n> +\tgit rev-parse commits/8 | git commit-graph write --stdin-commits &&\n> \tgit commit-graph verify >output\n> '\n\nI don't quite understand what the change is meant to do.\n\nAlso, as I said earlier, I'd prefer if tests were as indpendent of each\nother as possible, to make running individual tests (e.g. only\npreviously falling tests) easier.\n\nI especially do not like mixing running actual test with setting up the\nrepository for future tests, as here.\n\n> \n> +GRAPH_BYTE_VERSION=4\n> +GRAPH_BYTE_HASH=5\n> +\n> +# usage: corrupt_graph_and_verify \n> +# Manipulates the commit-graph file at the position\n> +# by inserting the data, then runs 'git commit-graph verify'\n> +# and places the output in the file 'err'. Test 'err' for\n> +# the given string.\n\nVery nice to have this description.\n\n> +corrupt_graph_and_verify() {\n> +\tpos=$1\n> +\tdata=\"${2:-\\0}\"\n> +\tgrepstr=$3\n> +\tcd \"$TRASH_DIRECTORY/full\" &&\n> +\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n> +\tcp $objdir/info/commit-graph commit-graph-backup &&\n> +\tprintf \"$data\" | dd of=\"$objdir/info/commit-graph\" bs=1 seek=\"$pos\" conv=notrunc &&\n\nUsing 'printf' with octal is much more portable than relying on 'echo'\nsupporting octal escape sequences (or supporting escape sequences at\nall).\n\n> +\ttest_must_fail git commit-graph verify 2>test_err &&\n> +\tgrep -v \"^+\" test_err >err\n> +\tgrep \"$grepstr\" err\n\nShouldn't this last 'grep' be 'test_i18ngrep' instead, to allow for\ntranslated messages from 'git commit-graph verify' / 'git fsck'?\n\n> +}\n\nThis function makes actual tests short and simple, without duplicated\ncode. Very good.\n\n> +\n> +test_expect_success 'detect bad signature' '\n> +\tcorrupt_graph_and_verify 0 \"\\0\" \\\n> +\t\t\"graph signature\"\n> +'\n> +\n> +test_expect_success 'detect bad version' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\02\" \\\n> +\t\t\"graph version\"\n> +'\n> +\n> +test_expect_success 'detect bad hash version' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_HASH \"\\02\" \\\n\nWhen we move from SHA-1 (hash version 1) to NewHash (hash version 2),\nthis test would start failing... which is actually not a bad idea.\n\n> +\t\t\"hash version\"\n> +'\n> +\n> test_done\n"},{"id":"348677","messageId":"867ennyaax.fsf@gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-9-dstolee@microsoft.com","subject":"Re: [PATCH v3 08/20] commit-graph: verify required chunks are present","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-05-28T17:11:50Z","receivedAt":"2018-05-28T17:14:34Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> The commit-graph file requires the following three chunks:\n>\n> * OID Fanout\n> * OID Lookup\n> * Commit Data\n>\n> If any of these are missing, then the 'verify' subcommand should\n> report a failure. This includes the chunk IDs malformed or the\n> chunk count is truncated.\n\nMinor nit: it should IMVHO either be \"or the chunk count truncated\", or\n\"or when the chunk count is truncated\".\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 9 +++++++++\n> t/t5318-commit-graph.sh | 29 +++++++++++++++++++++++++++++\n> 2 files changed, 38 insertions(+)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 55b41664ee..06e3e4f9ba 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -860,5 +860,14 @@ int verify_commit_graph(struct commit_graph *g)\n> \t\treturn 1;\n> \t}\n> \n> +\tverify_commit_graph_error = 0;\n> +\n\nBy the way, if chunk count is less than 3, then by pigeonhole principle\nat least one required chunk is missing.\n\n> +\tif (!g->chunk_oid_fanout)\n> +\t\tgraph_report(\"commit-graph is missing the OID Fanout chunk\");\n> +\tif (!g->chunk_oid_lookup)\n> +\t\tgraph_report(\"commit-graph is missing the OID Lookup chunk\");\n> +\tif (!g->chunk_commit_data)\n> +\t\tgraph_report(\"commit-graph is missing the Commit Data chunk\");\n\nNice and simple. Good.\n\n> +\n> \treturn verify_commit_graph_error;\n> }\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index bd64481c7a..4ef3fe3dc2 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -249,6 +249,15 @@ test_expect_success 'git commit-graph verify' '\n> \n> GRAPH_BYTE_VERSION=4\n> GRAPH_BYTE_HASH=5\n> +GRAPH_BYTE_CHUNK_COUNT=6\n> +GRAPH_CHUNK_LOOKUP_OFFSET=8\n> +GRAPH_CHUNK_LOOKUP_WIDTH=12\n> +GRAPH_CHUNK_LOOKUP_ROWS=5\n> +GRAPH_BYTE_OID_FANOUT_ID=$GRAPH_CHUNK_LOOKUP_OFFSET\n> +GRAPH_BYTE_OID_LOOKUP_ID=`expr $GRAPH_CHUNK_LOOKUP_OFFSET + \\\n> +\t\t\t 1 \\* $GRAPH_CHUNK_LOOKUP_WIDTH`\n> +GRAPH_BYTE_COMMIT_DATA_ID=`expr $GRAPH_CHUNK_LOOKUP_OFFSET + \\\n> +\t\t\t\t2 \\* $GRAPH_CHUNK_LOOKUP_WIDTH`\n> \n> # usage: corrupt_graph_and_verify \n> # Manipulates the commit-graph file at the position\n> @@ -283,4 +292,24 @@ test_expect_success 'detect bad hash version' '\n> \t\t\"hash version\"\n> '\n> \n> +test_expect_success 'detect bad chunk count' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_CHUNK_COUNT \"\\02\" \\\n> +\t\t\"missing the Commit Data chunk\"\n> +'\n\nAs I wrote before, this test assumes that the last chunk (the one not\ncounted because of changed / corrupted chunk count) is the Commit Data\nchunk. This may be true for corrent implementation, but it is not\nrequired by the format.\n\nBetter solution would be to check for \"missing the .* chunk\"; as I\nunderstand you can pass the regexp to grep, not only strings.\n\n\nAnother thing would be to check if there are gaps in the file, or if the\nwhole file is being used. Changing chunk count to a smaller number\nwould mean that chunks would not cover the rest of files.\n\nBy the way, would the following be detected:\n\n \tcorrupt_graph_and_verify $GRAPH_BYTE_CHUNK_COUNT \"\\05\"\n\nthat is corrupting the chunk count to be larger than the number of\nactual chunks? Or is it left for later?\n\n> +\n> +test_expect_success 'detect missing OID fanout chunk' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_FANOUT_ID \"\\0\" \\\n\nWe could have used \"X\" or \" \" in place of \"\\0\", but admittedly the\nlatter is a better check - it also checks if there are problems with\nhandling of NUL character (\"\\0\") in chunk names.\n\n> +\t\t\"missing the OID Fanout chunk\"\n> +'\n> +\n> +test_expect_success 'detect missing OID lookup chunk' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_LOOKUP_ID \"\\0\" \\\n> +\t\t\"missing the OID Lookup chunk\"\n> +'\n> +\n> +test_expect_success 'detect missing commit data chunk' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_DATA_ID \"\\0\" \\\n> +\t\t\"missing the Commit Data chunk\"\n> +'\n\nWhat happens if the terminating pseudo-chunk name \"\\0\\0\\0\\0\" gets\ncorrupted? Would it be detected (or maybe it is handled by later patch\nin the series)?\n\n> +\n> test_done\n"},{"id":"348687","messageId":"xmqqzi0jgk70.fsf@gitster-ct.c.googlers.com","threadId":"48311","inReplyTo":"20180524162504.158394-1-dstolee@microsoft.com","subject":"Re: [PATCH v3 00/20] Integrate commit-graph into 'fsck' and 'gc'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-05-29T04:27:47Z","receivedAt":"2018-05-29T04:27:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> Thanks for all the feedback on v2. I've tried to make this round's\n> review a bit easier by splitting up the commits into smaller pieces.\n> Also, the test script now has less boilerplate and uses variables and\n> clear arithmetic to explain which bytes are being modified.\n>\n> One other change worth mentioning: in \"commit-graph: add '--reachable'\n> option\" I put the ref-iteration into a new external\n> 'write_commit_graph_reachable()' method inside commit-graph.c. This\n> makes the 'gc: automatically write commit-graph files' a simpler change.\n\nI finally managed to find time to resolve conflicts this topic has\nwith other topics (of your own included, if I am not mistaken).\nPlease double check the resolution when I push out the day's\nintegration result later today.\n\nThanks.\n"},{"id":"348698","messageId":"83acfbf4-dc09-377d-212a-24579c4d7703@gmail.com","threadId":"48311","inReplyTo":"86y3g5z9ah.fsf@gmail.com","subject":"Re: [PATCH v3 03/20] commit-graph: parse commit from chosen graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-29T12:31:32Z","receivedAt":"2018-05-29T12:31:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/27/2018 6:23 AM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> Before verifying a commit-graph file against the object database, we\n>> need to parse all commits from the given commit-graph file. Create\n>> parse_commit_in_graph_one() to target a given struct commit_graph.\n> If I understand it properly the problem is that when verifying against\n> the object database we want to check one single commit-graph file, not\n> concatenation of data from commit-graph file for the repository and\n> commit-graph files from its alternates -- like prepare_commit_graph()\n> does; which is called by parse_commit_in_graph().\n>\n>> Signed-off-by: Derrick Stolee \n> O.K., so you introduce here a layer of indirection; parse_commit_in_graph()\n> now just uses parse_commit_in_graph_one(), passing core_commit_graph\n> (or the_commit_graph) to it, after checking that core_commit_graph is set\n> (which handles the case when feature is not turned off) and loading\n> commit-graph file.\n>\n> Nice and simple 'split function' refactoring, with new function taking\n> over when there is commit graph file prepared.\n>\n>\n> So, after the changes:\n> * parse_commit_in_graph() is responsible for checking whether to use\n> commit-graph feature and ensuring that data from commit-graph is\n> loaded, where it passes the control to parse_commit_in_graph_one()\n> * parse_commit_in_graph_one() checks whether commit-graph feature is\n> turned on, whether commit we are interested in was already parsed,\n> and then uses fill_commit_in_graph() to actually get the data\n> * fill_commit_in_graph() gets data out of commit-graph file, extracting\n> it from commit data chunk (and if needed large edges chunk).\n>\n> All those functions return 1 if they got data from commit-graph, and 0\n> if they didn't.\n>\n>\n> One minor nitpick / complaint / question is about naming of global\n> variables used here, namely:\n> * static struct commit_graph *commit_graph\n> from commit-graph.c for global storage of commit-graph[s] data\n> * int core_commit_graph\n> from environment.c for storing core.commitGraph config\n>\n> But I see that at least the latter is common convention in Git source\n> code; I guess that the former maybe follows convention as used for \"the\n> index\" and \"the repository\" - additionally it is static / file-local.\n\nSee also `struct packed_git *packed_git;` from cache.h.\n\n>\n>> ---\n>> commit-graph.c | 18 +++++++++++++++---\n>> 1 file changed, 15 insertions(+), 3 deletions(-)\n>>\n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index 82295f0975..78ba0edc80 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> @@ -310,7 +310,7 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n>> \t}\n>> }\n>> \n>> -int parse_commit_in_graph(struct commit *item)\n>> +static int parse_commit_in_graph_one(struct commit_graph *g, struct commit *item)\n>> {\n>> \tuint32_t pos;\n>> \n>> @@ -318,9 +318,21 @@ static int parse_commit_in_graph_one(struct commit_graph *g, struct commit *item)\n>> \tif (!core_commit_graph)\n>> \t\treturn 0;\n> All right, we check that commit-graph feature is enabled because\n> parse_commit_in_graph_one() will be used standalone, not by being\n> invoked from parse_commit_in_graph().\n>\n> This check is fast.\n>\n>> \tif (item->object.parsed)\n>> \t\treturn 1;\n> Sidenote: I just wonder why object.parsed and not for example\n> object.graph_pos is used to checck whether the object was filled if\n> possible with commit-graph data...\n\nHere we are filling all data, including commit date and parents. We use \nload_commit_graph_info() when we only need the graph_pos and generation \nvalues.\n\n>\n>> +\n>> +\tif (find_commit_in_graph(item, g, &pos))\n>> +\t\treturn fill_commit_in_graph(item, g, pos);\n>> +\n>> +\treturn 0;\n>> +}\n>> +\n>> +int parse_commit_in_graph(struct commit *item)\n>> +{\n>> +\tif (!core_commit_graph)\n>> +\t\treturn 0;\n> All right, this check is here to short-circuit and make it so git does\n> not even try to lod commit-graph file[s] if the feature is disabled.\n>\n>> +\n>> \tprepare_commit_graph();\n>> -\tif (commit_graph && find_commit_in_graph(item, commit_graph, &pos))\n>> -\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n>> +\tif (commit_graph)\n>> +\t\treturn parse_commit_in_graph_one(commit_graph, item);\n>> \treturn 0;\n>> }\n\n"},{"id":"348699","messageId":"c5a11465-67ac-fbae-dbea-d20e6958bf1a@gmail.com","threadId":"48311","inReplyTo":"xmqqzi0jgk70.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3 00/20] Integrate commit-graph into 'fsck' and 'gc'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-29T12:37:13Z","receivedAt":"2018-05-29T12:37:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/29/2018 12:27 AM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> Thanks for all the feedback on v2. I've tried to make this round's\n>> review a bit easier by splitting up the commits into smaller pieces.\n>> Also, the test script now has less boilerplate and uses variables and\n>> clear arithmetic to explain which bytes are being modified.\n>>\n>> One other change worth mentioning: in \"commit-graph: add '--reachable'\n>> option\" I put the ref-iteration into a new external\n>> 'write_commit_graph_reachable()' method inside commit-graph.c. This\n>> makes the 'gc: automatically write commit-graph files' a simpler change.\n> I finally managed to find time to resolve conflicts this topic has\n> with other topics (of your own included, if I am not mistaken).\n> Please double check the resolution when I push out the day's\n> integration result later today.\n\nSorry about the confusion. I was operating on my copy of \nds/generation-numbers (34fdd43339) but fetching just now I see you \nupdated that branch to 1472978ec6. I reset my local branch to \nds/commit-graph-fsk (53dd1e6600). The only diff I see between my v3 \nbranch and that commit are the changes from ds/commit-graph-lockfile-fix \n(33286dcd6d).\n\nThanks,\n-Stolee\n"},{"id":"348700","messageId":"6d9d346c-b178-26f2-fdeb-bb0ffb070774@gmail.com","threadId":"48311","inReplyTo":"86muwjyixo.fsf@gmail.com","subject":"Re: [PATCH v3 07/20] commit-graph: verify catches corrupt signature","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-29T12:43:48Z","receivedAt":"2018-05-29T12:43:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/28/2018 10:05 AM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> This is the first of several commits that add a test to check that\n>> 'git commit-graph verify' catches corruption in the commit-graph\n>> file. The first test checks that the command catches an error in\n>> the file signature. This is a check that exists in the existing\n>> commit-graph reading code.\n> Good start.\n>\n> This handles 3 out of 5 checks in load_commit_graph_one(). The\n> remaining are:\n> * too short file (length smaller than minimal commit-graph size)\n> * more than one chunk of one of 4 defined types\n>\n>> Add a helper method 'corrupt_graph_and_verify' to the test script\n>> t5318-commit-graph.sh. This helper corrupts the commit-graph file\n>> at a certain location, runs 'git commit-graph verify', and reports\n>> the output to the 'err' file. This data is filtered to remove the\n>> lines added by 'test_must_fail' when the test is run verbosely.\n>> Then, the output is checked to contain a specific error message.\n> Thanks for an explanation.\n>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> t/t5318-commit-graph.sh | 43 +++++++++++++++++++++++++++++++++++++++++++\n>> 1 file changed, 43 insertions(+)\n>>\n>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n>> index 6ca451dfd2..bd64481c7a 100755\n>> --- a/t/t5318-commit-graph.sh\n>> +++ b/t/t5318-commit-graph.sh\n>> @@ -235,9 +235,52 @@ test_expect_success 'perform fast-forward merge in full repo' '\n>> \ttest_cmp expect output\n>> '\n>> \n>> +# the verify tests below expect the commit-graph to contain\n>> +# exactly the commits reachable from the commits/8 branch.\n>> +# If the file changes the set of commits in the list, then the\n>> +# offsets into the binary file will result in different edits\n>> +# and the tests will likely break.\n>> +\n>> test_expect_success 'git commit-graph verify' '\n>> \tcd \"$TRASH_DIRECTORY/full\" &&\n>> +\tgit rev-parse commits/8 | git commit-graph write --stdin-commits &&\n>> \tgit commit-graph verify >output\n>> '\n> I don't quite understand what the change is meant to do.\n\nThis gives us a constant commit-graph file to work with in the later tests.\n\nTo get the \"independent test\" structure you want for the tests that are \ncoming, we need to do one of the following:\n\n1. Write a new commit-graph file for every test (slows things down).\n2. Do all corruption/verify checks in a single test (reduces the \ninformation from a failed test, as it only reports the first failure).\n\nI don't like either of these options, so I went with this \"prepare\" step.\n\n> Also, as I said earlier, I'd prefer if tests were as indpendent of each\n> other as possible, to make running individual tests (e.g. only\n> previously falling tests) easier.\n>\n> I especially do not like mixing running actual test with setting up the\n> repository for future tests, as here.\n>\n>> \n>> +GRAPH_BYTE_VERSION=4\n>> +GRAPH_BYTE_HASH=5\n>> +\n>> +# usage: corrupt_graph_and_verify \n>> +# Manipulates the commit-graph file at the position\n>> +# by inserting the data, then runs 'git commit-graph verify'\n>> +# and places the output in the file 'err'. Test 'err' for\n>> +# the given string.\n> Very nice to have this description.\n>\n>> +corrupt_graph_and_verify() {\n>> +\tpos=$1\n>> +\tdata=\"${2:-\\0}\"\n>> +\tgrepstr=$3\n>> +\tcd \"$TRASH_DIRECTORY/full\" &&\n>> +\ttest_when_finished mv commit-graph-backup $objdir/info/commit-graph &&\n>> +\tcp $objdir/info/commit-graph commit-graph-backup &&\n>> +\tprintf \"$data\" | dd of=\"$objdir/info/commit-graph\" bs=1 seek=\"$pos\" conv=notrunc &&\n> Using 'printf' with octal is much more portable than relying on 'echo'\n> supporting octal escape sequences (or supporting escape sequences at\n> all).\n>\n>> +\ttest_must_fail git commit-graph verify 2>test_err &&\n>> +\tgrep -v \"^+\" test_err >err\n>> +\tgrep \"$grepstr\" err\n> Shouldn't this last 'grep' be 'test_i18ngrep' instead, to allow for\n> translated messages from 'git commit-graph verify' / 'git fsck'?\n>\n>> +}\n> This function makes actual tests short and simple, without duplicated\n> code. Very good.\n>\n>> +\n>> +test_expect_success 'detect bad signature' '\n>> +\tcorrupt_graph_and_verify 0 \"\\0\" \\\n>> +\t\t\"graph signature\"\n>> +'\n>> +\n>> +test_expect_success 'detect bad version' '\n>> +\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\02\" \\\n>> +\t\t\"graph version\"\n>> +'\n>> +\n>> +test_expect_success 'detect bad hash version' '\n>> +\tcorrupt_graph_and_verify $GRAPH_BYTE_HASH \"\\02\" \\\n> When we move from SHA-1 (hash version 1) to NewHash (hash version 2),\n> this test would start failing... which is actually not a bad idea.\n>\n>> +\t\t\"hash version\"\n>> +'\n>> +\n>> test_done\n\n"},{"id":"348701","messageId":"xmqq6036h94z.fsf@gitster-ct.c.googlers.com","threadId":"48311","inReplyTo":"c5a11465-67ac-fbae-dbea-d20e6958bf1a@gmail.com","subject":"Re: [PATCH v3 00/20] Integrate commit-graph into 'fsck' and 'gc'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-05-29T13:41:16Z","receivedAt":"2018-05-29T13:41:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 5/29/2018 12:27 AM, Junio C Hamano wrote:\n>> Derrick Stolee writes:\n>>\n>>> Thanks for all the feedback on v2. I've tried to make this round's\n>>> review a bit easier by splitting up the commits into smaller pieces.\n>>> Also, the test script now has less boilerplate and uses variables and\n>>> clear arithmetic to explain which bytes are being modified.\n>>>\n>>> One other change worth mentioning: in \"commit-graph: add '--reachable'\n>>> option\" I put the ref-iteration into a new external\n>>> 'write_commit_graph_reachable()' method inside commit-graph.c. This\n>>> makes the 'gc: automatically write commit-graph files' a simpler change.\n>> I finally managed to find time to resolve conflicts this topic has\n>> with other topics (of your own included, if I am not mistaken).\n>> Please double check the resolution when I push out the day's\n>> integration result later today.\n>\n> Sorry about the confusion. I was operating on my copy of\n> ds/generation-numbers (34fdd43339) but fetching just now I see you\n> updated that branch to 1472978ec6. I reset my local branch to\n> ds/commit-graph-fsk (53dd1e6600). The only diff I see between my v3\n> branch and that commit are the changes from\n> ds/commit-graph-lockfile-fix (33286dcd6d).\n\nSorry for a confusing/confused comment. The topic I had in mind\nthat I saw interactions with this one was the per-in-core-repo\nallocation (sb/object-store-alloc), which was not yours.\n\nIn any case, what I wanted to see sanity checked was the result of\nthe merge into 'pu' (which needed some semantic adjustment), not\nindividual patches on the topic branch. Relative to what we see at\nthe tip of 'pu' right now, what I'll be pushing out in 3 or 4 hours\nwill gain yet another semantic adjustment, so you may want to wait\na bit more to avoid duplicated work.\n\nThanks.\n"},{"id":"348805","messageId":"20180530123529.32090-1-szeder.dev@gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-17-dstolee@microsoft.com","subject":"Re: [PATCH v3 16/20] commit-graph: verify contents match checksum","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-05-30T12:35:29Z","receivedAt":"2018-05-30T12:35:39Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"\n> diff --git a/commit-graph.c b/commit-graph.c\n> index d2b291aca2..a33600c584 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -841,6 +841,7 @@ void write_commit_graph(const char *obj_dir,\n> \toids.nr =3D 0;\n> }\n> =20\n> +#define VERIFY_COMMIT_GRAPH_ERROR_HASH 2\n> static int verify_commit_graph_error;\n> =20\n> static void graph_report(const char *fmt, ...)\n> @@ -860,7 +861,9 @@ static void graph_report(const char *fmt, ...)\n> int verify_commit_graph(struct commit_graph *g)\n> {\n> \tuint32_t i, cur_fanout_pos =3D 0;\n> -\tstruct object_id prev_oid, cur_oid;\n> +\tstruct object_id prev_oid, cur_oid, checksum;\n> +\tstruct hashfile *f;\n> +\tint devnull;\n> =20\n> \tif (!g) {\n> \t\tgraph_report(\"no commit-graph file loaded\");\n> @@ -879,6 +882,15 @@ int verify_commit_graph(struct commit_graph *g)\n> \tif (verify_commit_graph_error)\n> \t\treturn verify_commit_graph_error;\n> =20\n> +\tdevnull =3D open(\"/dev/null\", O_WRONLY);\n> +\tf =3D hashfd(devnull, NULL);\n> +\thashwrite(f, g->data, g->data_len - g->hash_len);\n> +\tfinalize_hashfile(f, checksum.hash, CSUM_CLOSE);\n> +\tif (hashcmp(checksum.hash, g->data + g->data_len - g->hash_len)) {\n> +\t\tgraph_report(_(\"the commit-graph file has incorrect checksum and is likely corrupt\"));\n\nThis error message is translated ...\n\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 240aef6add..2680a2ebff 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n\n> @@ -388,4 +389,9 @@ test_expect_success 'detect incorrect parent for octopu=\n> s merge' '\n> \t\t\"invalid parent\"\n> '\n> =20\n> +test_expect_success 'detect invalid checksum hash' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_FOOTER \"\\00\" \\\n> +\t\t\"incorrect checksum\"\n\n... but here in 'corrupt_graph_and_verify' you look for \"incorrect\nchecksum\" with plain 'grep' (as opposed to 'test_i18ngrep', which\nwon't find that string in a GETTEXT_POISON build, and ultimately\ncauses the test to fail.\n\n"},{"id":"348807","messageId":"86in75w9m9.fsf@gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-10-dstolee@microsoft.com","subject":"Re: [PATCH v3 09/20] commit-graph: verify corrupt OID fanout and lookup","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-05-30T13:34:06Z","receivedAt":"2018-05-30T13:34:14Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> In the commit-graph file, the OID fanout chunk provides an index into\n> the OID lookup. The 'verify' subcommand should find incorrect values\n> in the fanout.\n>\n> Similarly, the 'verify' subcommand should find out-of-order values in\n> the OID lookup.\n\nO.K., the OID Lookup chunk is checked together with OID Fanout chunk\nbecause those two chunks are tightly connected: OID Fanout is fanout\ninto OID Lookup.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 36 ++++++++++++++++++++++++++++++++++++\n> t/t5318-commit-graph.sh | 22 ++++++++++++++++++++++\n> 2 files changed, 58 insertions(+)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 06e3e4f9ba..cbd1aae514 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -855,6 +855,9 @@ static void graph_report(const char *fmt, ...)\n> \n> int verify_commit_graph(struct commit_graph *g)\n> {\n> +\tuint32_t i, cur_fanout_pos = 0;\n> +\tstruct object_id prev_oid, cur_oid;\n\nMinor nitpick about the naming convention: why cur_*, and not curr_*?\n\n> +\n> \tif (!g) {\n> \t\tgraph_report(\"no commit-graph file loaded\");\n> \t\treturn 1;\n> @@ -869,5 +872,38 @@ int verify_commit_graph(struct commit_graph *g)\n> \tif (!g->chunk_commit_data)\n> \t\tgraph_report(\"commit-graph is missing the Commit Data chunk\");\n> \n> +\tif (verify_commit_graph_error)\n> +\t\treturn verify_commit_graph_error;\n> +\n\nBefore checking that commits in OID Lookup are sorted, and that OID\nFanout correctly points into OID Lookup (thus also checking that OID\nLookup is sorted), it would be good to verify that sizes of those chunks\nare correct.\n\nOut of 4 standrd chunks, 1 of them (OIDF) has constant size, and 2 of\nthem have size given by number of commits and hash size\n - OID Fanout is (256 * 4 bytes)\n - OID Lookup is (N * H bytes),\n where N is number of commits, and H is hash size\n\nThe one that is more significant is if number of commits gets corrupted\nupwards, and walking through OID Lookup would lead us out of bounds,\noutside the file size.\n\nIIRC we have checked that all chunks fit within file size, isn't it?\n\n> +\tfor (i = 0; i < g->num_commits; i++) {\n> +\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n\nWhy do you use hashcpy() here, but oidcpy() below?\n\n> +\n> +\t\tif (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n\nAll right, OIDs needs to be sorted in ascending lexicographical order,\nand the above condition checks that this constraint is fullfilled. \n\n> +\t\t\tgraph_report(\"commit-graph has incorrect OID order: %s then %s\",\n> +\t\t\t\t oid_to_hex(&prev_oid),\n> +\t\t\t\t oid_to_hex(&cur_oid));\n\nNice informative error message.\n\n> +\n> +\t\toidcpy(&prev_oid, &cur_oid);\n> +\n> +\t\twhile (cur_oid.hash[0] > cur_fanout_pos) {\n> +\t\t\tuint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n> +\t\t\tif (i != fanout_value)\n> +\t\t\t\tgraph_report(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\",\n> +\t\t\t\t\t cur_fanout_pos, fanout_value, i);\n> +\n> +\t\t\tcur_fanout_pos++;\n> +\t\t}\n\nThe k-th entry of fanout, F[k], stores the number of OIDs with first\nbyte at most k.\n\nLet's examine it in detail on a simple example:\n\n fanout lookup\n ------ ------\n 0 : 0 ---------------> 0: 1xxxx....\n 1 : 2 -----\\ 1: 1yyyy....\n 2 : 2 ------\\--------> 2: 3xxxx....\n 3 : 3 ---------------> 3: 4xxxx....\n\nWe are checking that after most significant byte (first byte) changes,\nthen all fanout position up to the current first byte value (exclusive)\npoint to current position in OID Lookup chunk.\n\nSeems all right; it would be nice to come up with rigorous proof that it\nis all right.\n\n> +\t}\n> +\n> +\twhile (cur_fanout_pos < 256) {\n> +\t\tuint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n> +\n> +\t\tif (g->num_commits != fanout_value)\n> +\t\t\tgraph_report(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\",\n> +\t\t\t\t cur_fanout_pos, fanout_value, i);\n> +\n> +\t\tcur_fanout_pos++;\n> +\t}\n\nAll right, this checks that all remaining fanout entries, up and\nincluding the 255-ith entry store the total number of commits, N.\n\n> +\n> \treturn verify_commit_graph_error;\n> }\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 4ef3fe3dc2..c050ef980b 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -247,6 +247,7 @@ test_expect_success 'git commit-graph verify' '\n> \tgit commit-graph verify >output\n> '\n> \n> +HASH_LEN=20\n> GRAPH_BYTE_VERSION=4\n> GRAPH_BYTE_HASH=5\n> GRAPH_BYTE_CHUNK_COUNT=6\n> @@ -258,6 +259,12 @@ GRAPH_BYTE_OID_LOOKUP_ID=`expr $GRAPH_CHUNK_LOOKUP_OFFSET + \\\n> \t\t\t 1 \\* $GRAPH_CHUNK_LOOKUP_WIDTH`\n> GRAPH_BYTE_COMMIT_DATA_ID=`expr $GRAPH_CHUNK_LOOKUP_OFFSET + \\\n> \t\t\t\t2 \\* $GRAPH_CHUNK_LOOKUP_WIDTH`\n> +GRAPH_FANOUT_OFFSET=`expr $GRAPH_CHUNK_LOOKUP_OFFSET + \\\n> +\t\t\t $GRAPH_CHUNK_LOOKUP_WIDTH \\* $GRAPH_CHUNK_LOOKUP_ROWS`\n> +GRAPH_BYTE_FANOUT1=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 4`\n> +GRAPH_BYTE_FANOUT2=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 255`\n> +GRAPH_OID_LOOKUP_OFFSET=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 256`\n> +GRAPH_BYTE_OID_LOOKUP_ORDER=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8`\n\nSomething that I forgot to write about in previous patch.\n\nPOSIX shell includes $(( ... )) for arithmetic expansion, there is\nnowadays no need to use $(expr ...), and even more no need to use\npre-POSIX `expr ...`.\n\nTLDR: use $(( ... )), not `expr ... `.\n\n> \n> # usage: corrupt_graph_and_verify \n> # Manipulates the commit-graph file at the position\n> @@ -312,4 +319,19 @@ test_expect_success 'detect missing commit data chunk' '\n> \t\t\"missing the Commit Data chunk\"\n> '\n> \n> +test_expect_success 'detect incorrect fanout' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_FANOUT1 \"\\01\" \\\n\nHow can you be sure that this value is incorrect?\n\n> +\t\t\"fanout value\"\n> +'\n> +\n> +test_expect_success 'detect incorrect fanout' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_FANOUT2 \"\\01\" \\\n> +\t\t\"fanout value\"\n> +'\n\nI would prefer if different tests had different description. Those two\nare both 'detect incorrect layout'. What is the difference between them?\n\n> +\n> +test_expect_success 'detect incorrect OID order' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_LOOKUP_ORDER \"\\01\" \\\n> +\t\t\"incorrect OID order\"\n> +'\n\nHow can you be sure that this value is incorrect, or rather that it\nwould be always incorrect?\n\n> +\n> test_done\n\n-- \nJakub Narębski\n"},{"id":"348810","messageId":"4343f593-5650-3b24-54ec-a0d97d5251e9@gmail.com","threadId":"48311","inReplyTo":"86y3g4yahm.fsf@gmail.com","subject":"Re: [PATCH v3 06/20] commit-graph: add 'verify' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-30T16:07:39Z","receivedAt":"2018-05-30T16:07:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/27/2018 6:55 PM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> If the commit-graph file becomes corrupt, we need a way to verify\n>> that its contents match the object database. In the manner of\n>> 'git fsck' we will implement a 'git commit-graph verify' subcommand\n>> to report all issues with the file.\n>>\n>> Add the 'verify' subcommand to the 'commit-graph' builtin and its\n>> documentation. The subcommand is currently a no-op except for\n>> loading the commit-graph into memory, which may trigger run-time\n>> errors that would be caught by normal use.\n> So this commit is simply getting the boilerplate out of the way for\n> implementing 'git commit-graph verify' subcommand. Good.\n>\n>> Add a simple test that\n>> ensures the command returns a zero error code.\n> Nice.\n>\n>> If no commit-graph file exists, this is an acceptable state. Do\n>> not report any errors.\n> All right. I assume that as it is explicit verification call, it does\n> ignore core.commitGraph setting, isn't it?\n>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> Documentation/git-commit-graph.txt | 6 ++++++\n>> builtin/commit-graph.c | 38 ++++++++++++++++++++++++++++++++++++++\n>> commit-graph.c | 26 ++++++++++++++++++++++++++\n>> commit-graph.h | 2 ++\n>> t/t5318-commit-graph.sh | 10 ++++++++++\n>> 5 files changed, 82 insertions(+)\n>>\n>> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n>> index 4c97b555cc..a222cfab08 100644\n>> --- a/Documentation/git-commit-graph.txt\n>> +++ b/Documentation/git-commit-graph.txt\n>> @@ -10,6 +10,7 @@ SYNOPSIS\n>> --------\n>> [verse]\n>> 'git commit-graph read' [--object-dir ]\n>> +'git commit-graph verify' [--object-dir ]\n>> 'git commit-graph write' [--object-dir ]\n> In alphabetical order, good.\n>\n>> \n>> \n>> @@ -52,6 +53,11 @@ existing commit-graph file.\n>> Read a graph file given by the commit-graph file and output basic\n>> details about the graph file. Used for debugging purposes.\n>> \n>> +'verify'::\n>> +\n>> +Read the commit-graph file and verify its contents against the object\n>> +database. Used to check for corrupted data.\n>> +\n> All right, good enough description.\n>\n>> \n>> EXAMPLES\n>> --------\n>> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n>> index f0875b8bf3..0433dd6e20 100644\n>> --- a/builtin/commit-graph.c\n>> +++ b/builtin/commit-graph.c\n>> @@ -8,10 +8,16 @@\n>> static char const * const builtin_commit_graph_usage[] = {\n>> \tN_(\"git commit-graph [--object-dir ]\"),\n>> \tN_(\"git commit-graph read [--object-dir ]\"),\n>> +\tN_(\"git commit-graph verify [--object-dir ]\"),\n>> \tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n>> \tNULL\n>> };\n> In alphabetical order, same as in the manpage for git-commit-graph.\n>\n>> \n>> +static const char * const builtin_commit_graph_verify_usage[] = {\n>> +\tN_(\"git commit-graph verify [--object-dir ]\"),\n>> +\tNULL\n>> +};\n>> +\n>> static const char * const builtin_commit_graph_read_usage[] = {\n>> \tN_(\"git commit-graph read [--object-dir ]\"),\n>> \tNULL\n>> @@ -29,6 +35,36 @@ static struct opts_commit_graph {\n>> \tint append;\n>> } opts;\n>> \n>> +\n>> +static int graph_verify(int argc, const char **argv)\n>> +{\n>> +\tstruct commit_graph *graph = 0;\n>> +\tchar *graph_name;\n>> +\n>> +\tstatic struct option builtin_commit_graph_verify_options[] = {\n>> +\t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n>> +\t\t\t N_(\"dir\"),\n>> +\t\t\t N_(\"The object directory to store the graph\")),\n>> +\t\tOPT_END(),\n>> +\t};\n>> +\n>> +\targc = parse_options(argc, argv, NULL,\n>> +\t\t\t builtin_commit_graph_verify_options,\n>> +\t\t\t builtin_commit_graph_verify_usage, 0);\n>> +\n>> +\tif (!opts.obj_dir)\n>> +\t\topts.obj_dir = get_object_directory();\n> Getting the boilerplate of implementing the command mostly out of the\n> way. Good.\n>\n>> +\n>> +\tgraph_name = get_commit_graph_filename(opts.obj_dir);\n>> +\tgraph = load_commit_graph_one(graph_name);\n> So we are verifying only the commit-graph file belonging directly to\n> current repository, as I have expected. This is needed to for warnings\n> and error messages from the 'verify' action, to be able to tell in which\n> file there are problems.\n>\n> This means that it is possible that there would be problems with\n> commit-graph files that running 'git commit-graph verify' would not\n> find, because they are in commit-graph file in one of the alternates.\n>\n> It is very easy, though, to check all commit-graph files that would be\n> read and its data concatenated when using commit-graph feature\n> (e.g. 'git commit-graph read', IIRC):\n>\n> $ git commit-graph verify\n> $ for obj_dir in $(cat .git/objects/info/alternates) do;\n> git commit-graph --object-dir=\"$obj_dir\";\n> done\n>\n> Note: I have not checked the above that it works.\n>\n>> +\tFREE_AND_NULL(graph_name);\n> Freeing the resources, always nice to have.\n>\n>> +\n>> +\tif (!graph)\n>> +\t\treturn 0;\n> DS> If no commit-graph file exists, this is an acceptable state. Do\n> DS> not report any errors.\n>\n> Right, non existant commit-graph file is certainly valid ;-)\n>\n>> +\n>> +\treturn verify_commit_graph(graph);\n>> +}\n> I guess that graph_verify() would not change much, if at all, in\n> subsequent commits in this patch series.\n>\n>> +\n>> static int graph_read(int argc, const char **argv)\n>> {\n>> \tstruct commit_graph *graph = NULL;\n>> @@ -163,6 +199,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n>> \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n>> \n>> \tif (argc > 0) {\n>> +\t\tif (!strcmp(argv[0], \"verify\"))\n>> +\t\t\treturn graph_verify(argc, argv);\n>> \t\tif (!strcmp(argv[0], \"read\"))\n>> \t\t\treturn graph_read(argc, argv);\n>> \t\tif (!strcmp(argv[0], \"write\"))\n> Not in alphabetical order... is there a reason for that?\nThis will be fixed in v4.\n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index 25893ec096..55b41664ee 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> @@ -836,3 +836,29 @@ void write_commit_graph(const char *obj_dir,\n>> \toids.alloc = 0;\n>> \toids.nr = 0;\n>> }\n>> +\n>> +static int verify_commit_graph_error;\n>> +\n>> +static void graph_report(const char *fmt, ...)\n>> +{\n>> +\tva_list ap;\n>> +\tstruct strbuf sb = STRBUF_INIT;\n>> +\tverify_commit_graph_error = 1;\n>> +\n>> +\tva_start(ap, fmt);\n>> +\tstrbuf_vaddf(&sb, fmt, ap);\n>> +\n>> +\tfprintf(stderr, \"%s\\n\", sb.buf);\n>> +\tstrbuf_release(&sb);\n>> +\tva_end(ap);\n> Why do you use strbuf_vaddf + fprintf instead of straighforward\n> vfprintf (or function instead of variable-level macro)?\n>\n> Is it because of [string] safety?\n\nIt's because I've never used this variable-parameter thing before and \nfound a different example.\n\nI'll use vfprintf() in v4, as it is simpler.\n\n\n>> +}\n>> +\n>> +int verify_commit_graph(struct commit_graph *g)\n>> +{\n>> +\tif (!g) {\n>> +\t\tgraph_report(\"no commit-graph file loaded\");\n>> +\t\treturn 1;\n>> +\t}\n> All right, this is just a placeholder - we should not ever get this\n> message because in this case we exit with error code of 0 (EXIT_SUCCESS)\n> if there is no commit-graph file loaded before invoking\n> verify_commit_graph().\n>\n>> +\n>> +\treturn verify_commit_graph_error;\n> All right, this is for the future. Good.\n>\n>> +}\n>> diff --git a/commit-graph.h b/commit-graph.h\n>> index 96cccb10f3..71a39c5a57 100644\n>> --- a/commit-graph.h\n>> +++ b/commit-graph.h\n>> @@ -53,4 +53,6 @@ void write_commit_graph(const char *obj_dir,\n>> \t\t\tint nr_commits,\n>> \t\t\tint append);\n>> \n>> +int verify_commit_graph(struct commit_graph *g);\n>> +\n> Why does this need to be exported? I think it is not used outside of\n> commit-graph.c, isn't it?\n\nUsed by builtin/commit-graph.c\n\n\n>\n>> #endif\n>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n>> index 77d85aefe7..6ca451dfd2 100755\n>> --- a/t/t5318-commit-graph.sh\n>> +++ b/t/t5318-commit-graph.sh\n>> @@ -11,6 +11,11 @@ test_expect_success 'setup full repo' '\n>> \tobjdir=\".git/objects\"\n>> '\n>> \n>> +test_expect_success 'verify graph with no graph file' '\n>> +\tcd \"$TRASH_DIRECTORY/full\" &&\n> Is sich bare `cd`, without corresponding `cd` back or using subshell\n> safe?\n>\n>> +\tgit commit-graph verify\n>> +'\n>> +\n>> test_expect_success 'write graph with no packs' '\n>> \tcd \"$TRASH_DIRECTORY/full\" &&\n>> \tgit commit-graph write --object-dir . &&\n>> @@ -230,4 +235,9 @@ test_expect_success 'perform fast-forward merge in full repo' '\n>> \ttest_cmp expect output\n>> '\n>> \n>> +test_expect_success 'git commit-graph verify' '\n>> +\tcd \"$TRASH_DIRECTORY/full\" &&\n>> +\tgit commit-graph verify >output\n>> +'\n> Those are tests with nearly the same code, but they are (by their\n> descriptions) testing different things. This means that they rely on\n> side effects of earlier tests.\n>\n> This is suboptimal, as it means that it would be impossible or very\n> difficult to run individual tests (e.g. with GIT_SKIP_TESTS environment\n> variable, or with an individual test suite --run option), unless you\n> know which tests setup the repository state for later tests.\n>\n> It also means that running only failed tests with prove\n> --state=failed,save or equivalently with\n>\n> $ make DEFAULT_TEST_TARGET=prove GIT_PROVE_OPTS='--state=failed,save' test\n>\n> wouldn't work correctly.\n>\n> As Johannes Schindelin (alias Dscho) said in latest Git Rev News\n> interview: https://git.github.io/rev_news/2018/05/16/edition-39/\n>\n> JS> We have a test suite where debugging a regression may mean that you\n> JS> have to run 98 test cases before the failing one every single time in\n> JS> the edit/compile/debug cycle, because the 99th test case may depend on\n> JS> a side effect of at least one of the preceding test cases. Git’s test\n> JS> suite is so not [21st century best practices][1].\n> JS>\n> JS> [1]: https://www.slideshare.net/BuckHodges/lessons-learned-doing-devops-at-scale-at-microsoft\n>\n>\n> I think can be solved quite efficiently by creating and using shell\n> function, or two shell functions, which would either:\n>\n> * rename commit-graph file to some other temporary name if it exists,\n> and move it back after the test.\n> * create commit-graph file if it does not exist.\n>\n> For example (untested):\n>\n> prepare_no_commit_graph() {\n> \tmv .git/info/commit-graph .git/info/commit-graph.away &&\n> \ttest_when_finished \"mv .git/info/commit-graph.away .git/info/commit-graph\"\n> }\n>\n> prepare_commit_graph() {\n> \tif ! test -f \".git/info/commit-graph\"\n> \tthen\n> \t\tgit commit-graph write\n> \tfi\n> }\n>\n> Or something like that.\n\nDo we have a way to run individual steps of the test suite? I am unfamiliar with that process.\n\nAdding the complexity of storing a copy of the commit-graph file for re-use in a later test is wasted energy right now, because we need to run the steps of the test that create the repo shape with the commits laid out as set earlier in the test. This shape changes as we test different states of the commit-graph (exists and contains all commits, exists and doesn't contain all commits, etc.)\n\nThanks,\n-Stolee\n\n"},{"id":"348814","messageId":"b34afb39-8df2-b825-773c-8526d93314b5@gmail.com","threadId":"48311","inReplyTo":"86in75w9m9.fsf@gmail.com","subject":"Re: [PATCH v3 09/20] commit-graph: verify corrupt OID fanout and lookup","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-30T16:18:26Z","receivedAt":"2018-05-30T16:18:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"\nOn 5/30/2018 9:34 AM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> In the commit-graph file, the OID fanout chunk provides an index into\n>> the OID lookup. The 'verify' subcommand should find incorrect values\n>> in the fanout.\n>>\n>> Similarly, the 'verify' subcommand should find out-of-order values in\n>> the OID lookup.\n> O.K., the OID Lookup chunk is checked together with OID Fanout chunk\n> because those two chunks are tightly connected: OID Fanout is fanout\n> into OID Lookup.\n>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> commit-graph.c | 36 ++++++++++++++++++++++++++++++++++++\n>> t/t5318-commit-graph.sh | 22 ++++++++++++++++++++++\n>> 2 files changed, 58 insertions(+)\n>>\n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index 06e3e4f9ba..cbd1aae514 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> @@ -855,6 +855,9 @@ static void graph_report(const char *fmt, ...)\n>> \n>> int verify_commit_graph(struct commit_graph *g)\n>> {\n>> +\tuint32_t i, cur_fanout_pos = 0;\n>> +\tstruct object_id prev_oid, cur_oid;\n> Minor nitpick about the naming convention: why cur_*, and not curr_*?\n>\n>> +\n>> \tif (!g) {\n>> \t\tgraph_report(\"no commit-graph file loaded\");\n>> \t\treturn 1;\n>> @@ -869,5 +872,38 @@ int verify_commit_graph(struct commit_graph *g)\n>> \tif (!g->chunk_commit_data)\n>> \t\tgraph_report(\"commit-graph is missing the Commit Data chunk\");\n>> \n>> +\tif (verify_commit_graph_error)\n>> +\t\treturn verify_commit_graph_error;\n>> +\n> Before checking that commits in OID Lookup are sorted, and that OID\n> Fanout correctly points into OID Lookup (thus also checking that OID\n> Lookup is sorted), it would be good to verify that sizes of those chunks\n> are correct.\n>\n> Out of 4 standrd chunks, 1 of them (OIDF) has constant size, and 2 of\n> them have size given by number of commits and hash size\n> - OID Fanout is (256 * 4 bytes)\n> - OID Lookup is (N * H bytes),\n> where N is number of commits, and H is hash size\n>\n> The one that is more significant is if number of commits gets corrupted\n> upwards, and walking through OID Lookup would lead us out of bounds,\n> outside the file size.\n>\n> IIRC we have checked that all chunks fit within file size, isn't it?\n>\n>> +\tfor (i = 0; i < g->num_commits; i++) {\n>> +\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n> Why do you use hashcpy() here, but oidcpy() below?\n\nWe are copying from a section of binary data, not from a struct object_id *.\n\n>\n>> +\n>> +\t\tif (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n> All right, OIDs needs to be sorted in ascending lexicographical order,\n> and the above condition checks that this constraint is fullfilled.\n>\n>> +\t\t\tgraph_report(\"commit-graph has incorrect OID order: %s then %s\",\n>> +\t\t\t\t oid_to_hex(&prev_oid),\n>> +\t\t\t\t oid_to_hex(&cur_oid));\n> Nice informative error message.\n>\n>> +\n>> +\t\toidcpy(&prev_oid, &cur_oid);\n>> +\n>> +\t\twhile (cur_oid.hash[0] > cur_fanout_pos) {\n>> +\t\t\tuint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n>> +\t\t\tif (i != fanout_value)\n>> +\t\t\t\tgraph_report(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\",\n>> +\t\t\t\t\t cur_fanout_pos, fanout_value, i);\n>> +\n>> +\t\t\tcur_fanout_pos++;\n>> +\t\t}\n> The k-th entry of fanout, F[k], stores the number of OIDs with first\n> byte at most k.\n>\n> Let's examine it in detail on a simple example:\n>\n> fanout lookup\n> ------ ------\n> 0 : 0 ---------------> 0: 1xxxx....\n> 1 : 2 -----\\ 1: 1yyyy....\n> 2 : 2 ------\\--------> 2: 3xxxx....\n> 3 : 3 ---------------> 3: 4xxxx....\n>\n> We are checking that after most significant byte (first byte) changes,\n> then all fanout position up to the current first byte value (exclusive)\n> point to current position in OID Lookup chunk.\n>\n> Seems all right; it would be nice to come up with rigorous proof that it\n> is all right.\n>\n>> +\t}\n>> +\n>> +\twhile (cur_fanout_pos < 256) {\n>> +\t\tuint32_t fanout_value = get_be32(g->chunk_oid_fanout + cur_fanout_pos);\n>> +\n>> +\t\tif (g->num_commits != fanout_value)\n>> +\t\t\tgraph_report(\"commit-graph has incorrect fanout value: fanout[%d] = %u != %u\",\n>> +\t\t\t\t cur_fanout_pos, fanout_value, i);\n>> +\n>> +\t\tcur_fanout_pos++;\n>> +\t}\n> All right, this checks that all remaining fanout entries, up and\n> including the 255-ith entry store the total number of commits, N.\n>\n>> +\n>> \treturn verify_commit_graph_error;\n>> }\n>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n>> index 4ef3fe3dc2..c050ef980b 100755\n>> --- a/t/t5318-commit-graph.sh\n>> +++ b/t/t5318-commit-graph.sh\n>> @@ -247,6 +247,7 @@ test_expect_success 'git commit-graph verify' '\n>> \tgit commit-graph verify >output\n>> '\n>> \n>> +HASH_LEN=20\n>> GRAPH_BYTE_VERSION=4\n>> GRAPH_BYTE_HASH=5\n>> GRAPH_BYTE_CHUNK_COUNT=6\n>> @@ -258,6 +259,12 @@ GRAPH_BYTE_OID_LOOKUP_ID=`expr $GRAPH_CHUNK_LOOKUP_OFFSET + \\\n>> \t\t\t 1 \\* $GRAPH_CHUNK_LOOKUP_WIDTH`\n>> GRAPH_BYTE_COMMIT_DATA_ID=`expr $GRAPH_CHUNK_LOOKUP_OFFSET + \\\n>> \t\t\t\t2 \\* $GRAPH_CHUNK_LOOKUP_WIDTH`\n>> +GRAPH_FANOUT_OFFSET=`expr $GRAPH_CHUNK_LOOKUP_OFFSET + \\\n>> +\t\t\t $GRAPH_CHUNK_LOOKUP_WIDTH \\* $GRAPH_CHUNK_LOOKUP_ROWS`\n>> +GRAPH_BYTE_FANOUT1=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 4`\n>> +GRAPH_BYTE_FANOUT2=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 255`\n>> +GRAPH_OID_LOOKUP_OFFSET=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 256`\n>> +GRAPH_BYTE_OID_LOOKUP_ORDER=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8`\n> Something that I forgot to write about in previous patch.\n>\n> POSIX shell includes $(( ... )) for arithmetic expansion, there is\n> nowadays no need to use $(expr ...), and even more no need to use\n> pre-POSIX `expr ...`.\n>\n> TLDR: use $(( ... )), not `expr ... `.\n\nI'll use this convention in v4. Thanks!\n\n>\n>> \n>> # usage: corrupt_graph_and_verify \n>> # Manipulates the commit-graph file at the position\n>> @@ -312,4 +319,19 @@ test_expect_success 'detect missing commit data chunk' '\n>> \t\t\"missing the Commit Data chunk\"\n>> '\n>> \n>> +test_expect_success 'detect incorrect fanout' '\n>> +\tcorrupt_graph_and_verify $GRAPH_BYTE_FANOUT1 \"\\01\" \\\n> How can you be sure that this value is incorrect?\n\nThe repo is created using constant information (the commit/author dates \nare set by environment variables in the test environment). The \ncommit-graph file that is generated by the test script is identical each \ntime.\n\n\n>\n>> +\t\t\"fanout value\"\n>> +'\n>> +\n>> +test_expect_success 'detect incorrect fanout' '\n>> +\tcorrupt_graph_and_verify $GRAPH_BYTE_FANOUT2 \"\\01\" \\\n>> +\t\t\"fanout value\"\n>> +'\n> I would prefer if different tests had different description. Those two\n> are both 'detect incorrect layout'. What is the difference between them?\nI'll specify this one is for the final value.\n>\n>> +\n>> +test_expect_success 'detect incorrect OID order' '\n>> +\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_LOOKUP_ORDER \"\\01\" \\\n>> +\t\t\"incorrect OID order\"\n>> +'\n> How can you be sure that this value is incorrect, or rather that it\n> would be always incorrect?\n>\n>> +\n>> test_done\n"},{"id":"348834","messageId":"86a7shvth6.fsf@gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-11-dstolee@microsoft.com","subject":"Re: [PATCH v3 10/20] commit-graph: verify objects exist","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-05-30T19:22:45Z","receivedAt":"2018-05-30T19:22:53Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> In the 'verify' subcommand, load commits directly from the object\n> database to ensure they exist. Parse by skipping the commit-graph.\n\nAll right, before we check that the commit data matches, we need to\ncheck that all the commits in cache (in the serialized commit graph) are\npresent in real data (in the object database of the repository).\n\nWhat's nice of this series is that the operation that actually removes\nunreachable commits from the object database, that is `git gc`, would\nalso update commit-gaph file.\n\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 20 ++++++++++++++++++++\n> t/t5318-commit-graph.sh | 7 +++++++\n> 2 files changed, 27 insertions(+)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index cbd1aae514..0420ebcd87 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -238,6 +238,10 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n> {\n> \tstruct commit *c;\n> \tstruct object_id oid;\n> +\n> +\tif (pos >= g->num_commits)\n> +\t\tdie(\"invalid parent position %\"PRIu64, pos);\n> +\n\nThis change is not described in the commit message.\n\n> \thashcpy(oid.hash, g->chunk_oid_lookup + g->hash_len * pos);\n> \tc = lookup_commit(&oid);\n> \tif (!c)\n> @@ -905,5 +909,21 @@ int verify_commit_graph(struct commit_graph *g)\n> \t\tcur_fanout_pos++;\n> \t}\n> \n> +\tif (verify_commit_graph_error)\n> +\t\treturn verify_commit_graph_error;\n\nAll right, so we by default short-circuit so that errors found earlier\nwouldn't cause crash when checking other things.\n\nIs it needed, though, in this case? Though I guess it is better to be\nconservative; lso by terminating after a batch of one type of errors we\ndon't get many different error messages from the same error (i.e. error\npropagation).\n\n> +\n> +\tfor (i = 0; i < g->num_commits; i++) {\n> +\t\tstruct commit *odb_commit;\n> +\n> +\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n> +\n> +\t\todb_commit = (struct commit *)create_object(cur_oid.hash, alloc_commit_node());\n\nDo we really need to keep all those commits from the object database in\nmemory (in the object::obj_hash hash table)? Perhaps using something\nlike Flywheel / Recycler pattern would be a better solution (if\npossible)?\n\nThough perhaps this doesn't matter much with respect to memory use.\n\n> +\t\tif (parse_commit_internal(odb_commit, 0, 0)) {\n\nJust a reminder to myself: the signature is\n\n int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph)\n\n\nHmmm... I wonder if with two boolean paramaters wouldn't it be better to\nuse flags parameter, i.e.\n\n int parse_commit_internal(struct commit *item, int flags)\n\n ...\n\n parse_commit_internal(commit, QUIET_ON_MISSING | USE_COMMIT_GRAPH)\n\nBut I guess that it is not worth it (especially for internal-ish\nfunction).\n\n> +\t\t\tgraph_report(\"failed to parse %s from object database\",\n> +\t\t\t\t oid_to_hex(&cur_oid));\n\nWouldn't parse_commit_internal() show it's own error message, in\naddition to the one above?\n\n> +\t\t\tcontinue;\n> +\t\t}\n> +\t}\n> +\n> \treturn verify_commit_graph_error;\n> }\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index c050ef980b..996a016239 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -247,6 +247,7 @@ test_expect_success 'git commit-graph verify' '\n> \tgit commit-graph verify >output\n> '\n> \n> +NUM_COMMITS=9\n> HASH_LEN=20\n> GRAPH_BYTE_VERSION=4\n> GRAPH_BYTE_HASH=5\n> @@ -265,6 +266,7 @@ GRAPH_BYTE_FANOUT1=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 4`\n> GRAPH_BYTE_FANOUT2=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 255`\n> GRAPH_OID_LOOKUP_OFFSET=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 256`\n> GRAPH_BYTE_OID_LOOKUP_ORDER=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8`\n> +GRAPH_BYTE_OID_LOOKUP_MISSING=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 4 + 10`\n\nAll right, so we modify 10-the byte of OID of 5-th commit out of 9,\nam I correct here?\n\n> \n> # usage: corrupt_graph_and_verify \n> # Manipulates the commit-graph file at the position\n> @@ -334,4 +336,9 @@ test_expect_success 'detect incorrect OID order' '\n> \t\t\"incorrect OID order\"\n> '\n> \n> +test_expect_success 'detect OID not in object database' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_LOOKUP_MISSING \"\\01\" \\\n> +\t\t\"from object database\"\n> +'\n\nI guess that if we ensure that OIDs are constant, you have chosen the\nchange to actually corrupt the OID in OID Lookup chunk to point to OID\nthat is not in the object database (while still not violating the\nconstraint that OID in OID Lookup chunk needs to be sorted).\n\n> +\n> test_done\n\nAll right (well, except for `expr ... ` --> $(( ... )) change).\n\n-- \nJakub Narębski\n"},{"id":"348850","messageId":"86zi0gvl1h.fsf@gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-12-dstolee@microsoft.com","subject":"Re: [PATCH v3 11/20] commit-graph: verify root tree OIDs","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-05-30T22:24:58Z","receivedAt":"2018-05-30T22:25:05Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> The 'verify' subcommand must compare the commit content parsed from the\n> commit-graph and compare it against the content in the object database.\n\nYou have \"compare\" twice in the above sentence.\n\n> Use lookup_commit() and parse_commit_in_graph_one() to parse the commits\n> from the graph and compare against a commit that is loaded separately\n> and parsed directly from the object database.\n\nAll right, that looks like a nice extension of what was done in previous\npatch. We want to check that cache (serialized commit graph) matches\nreality (object database).\n\n>\n> Add checks for the root tree OID.\n\nAll right; isn't it that now we check almost all information from\ncommit-graph that hs match in object database (with exception of commit\nparents, I think).\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 17 ++++++++++++++++-\n> t/t5318-commit-graph.sh | 7 +++++++\n> 2 files changed, 23 insertions(+), 1 deletion(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 0420ebcd87..19ea369fc6 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -880,6 +880,8 @@ int verify_commit_graph(struct commit_graph *g)\n> \t\treturn verify_commit_graph_error;\n\nNOTE: we will be checking Commit Data chunk; I think it would be good\nidea to verify that size of Commit Data chunk matches (N * (H + 16) bytes)\nthat format gives us, so that we don't accidentally red outside of\nmemory if something got screwed up (like number of commits being wrong,\nor file truncated).\n\n> \n> \tfor (i = 0; i < g->num_commits; i++) {\n> +\t\tstruct commit *graph_commit;\n> +\n> \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n> \n> \t\tif (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n> @@ -897,6 +899,11 @@ int verify_commit_graph(struct commit_graph *g)\n> \n> \t\t\tcur_fanout_pos++;\n> \t\t}\n> +\n> +\t\tgraph_commit = lookup_commit(&cur_oid);\n\nSo now I see why we add all commits to memory (to hash structure).\n\n> +\t\tif (!parse_commit_in_graph_one(g, graph_commit))\n> +\t\t\tgraph_report(\"failed to parse %s from commit-graph\",\n> +\t\t\t\t oid_to_hex(&cur_oid));\n\nAll right, this verifies that commit in OID Lookup chunk has parse-able\ndata in Commit Data chunk, isn't it?\n\n> \t}\n> \n> \twhile (cur_fanout_pos < 256) {\n> @@ -913,16 +920,24 @@ int verify_commit_graph(struct commit_graph *g)\n> \t\treturn verify_commit_graph_error;\n> \n> \tfor (i = 0; i < g->num_commits; i++) {\n> -\t\tstruct commit *odb_commit;\n> +\t\tstruct commit *graph_commit, *odb_commit;\n> \n> \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n> \n> +\t\tgraph_commit = lookup_commit(&cur_oid);\n> \t\todb_commit = (struct commit *)create_object(cur_oid.hash, alloc_commit_node());\n\nAll right, so we have commit data from graph, and commit data from the\nobject database.\n\n> \t\tif (parse_commit_internal(odb_commit, 0, 0)) {\n> \t\t\tgraph_report(\"failed to parse %s from object database\",\n> \t\t\t\t oid_to_hex(&cur_oid));\n> \t\t\tcontinue;\n> \t\t}\n> +\n> +\t\tif (oidcmp(&get_commit_tree_in_graph_one(g, graph_commit)->object.oid,\n> +\t\t\t get_commit_tree_oid(odb_commit)))\n> +\t\t\tgraph_report(\"root tree OID for commit %s in commit-graph is %s != %s\",\n> +\t\t\t\t oid_to_hex(&cur_oid),\n> +\t\t\t\t oid_to_hex(get_commit_tree_oid(graph_commit)),\n> +\t\t\t\t oid_to_hex(get_commit_tree_oid(odb_commit)));\n\nMaybe explicitly say that it doesn't match the value from the object\ndatabase; OTOH this may be too verbose.\n\n> \t}\n> \n> \treturn verify_commit_graph_error;\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 996a016239..21cc8e82f3 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -267,6 +267,8 @@ GRAPH_BYTE_FANOUT2=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 255`\n> GRAPH_OID_LOOKUP_OFFSET=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 256`\n> GRAPH_BYTE_OID_LOOKUP_ORDER=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8`\n> GRAPH_BYTE_OID_LOOKUP_MISSING=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 4 + 10`\n> +GRAPH_COMMIT_DATA_OFFSET=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* $NUM_COMMITS`\n> +GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n\nAll right, so the first is entry into record in Commit Data chunk, and\nthe latter points into tree entry in this record -- which entry is first\nfield:\n\n Commit Data (ID: {'C', 'G', 'E', 'T' }) (N * (H + 16) bytes)\n * The first H bytes are for the OID of the root tree.\n\n> \n> # usage: corrupt_graph_and_verify \n> # Manipulates the commit-graph file at the position\n> @@ -341,4 +343,9 @@ test_expect_success 'detect OID not in object database' '\n> \t\t\"from object database\"\n> '\n> \n> +test_expect_success 'detect incorrect tree OID' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_TREE \"\\01\" \\\n> +\t\t\"root tree OID for commit\"\n> +'\n\nAll right.\n\nI wonder if we can create a test for first check added (that Commit\nGraph data parses correctly), that is the one with the following error\nmessage:\n\n \"failed to parse from commit-graph file\".\n\n> +\n> test_done\n\nRegards,\n-- \nJakub Narębski\n"},{"id":"348896","messageId":"7876b157-6730-53ae-d1c9-102df1a16331@gmail.com","threadId":"48311","inReplyTo":"86a7shvth6.fsf@gmail.com","subject":"Re: [PATCH v3 10/20] commit-graph: verify objects exist","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-31T12:53:35Z","receivedAt":"2018-05-31T12:53:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/30/2018 3:22 PM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> In the 'verify' subcommand, load commits directly from the object\n>> database to ensure they exist. Parse by skipping the commit-graph.\n> All right, before we check that the commit data matches, we need to\n> check that all the commits in cache (in the serialized commit graph) are\n> present in real data (in the object database of the repository).\n>\n> What's nice of this series is that the operation that actually removes\n> unreachable commits from the object database, that is `git gc`, would\n> also update commit-gaph file.\n>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> commit-graph.c | 20 ++++++++++++++++++++\n>> t/t5318-commit-graph.sh | 7 +++++++\n>> 2 files changed, 27 insertions(+)\n>>\n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index cbd1aae514..0420ebcd87 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> @@ -238,6 +238,10 @@ static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n>> {\n>> \tstruct commit *c;\n>> \tstruct object_id oid;\n>> +\n>> +\tif (pos >= g->num_commits)\n>> +\t\tdie(\"invalid parent position %\"PRIu64, pos);\n>> +\n> This change is not described in the commit message.\nThis change should go in \"commit-graph: verify parent list\" which adds a \ntest that fails without it. Thanks.\n\n>> \thashcpy(oid.hash, g->chunk_oid_lookup + g->hash_len * pos);\n>> \tc = lookup_commit(&oid);\n>> \tif (!c)\n>> @@ -905,5 +909,21 @@ int verify_commit_graph(struct commit_graph *g)\n>> \t\tcur_fanout_pos++;\n>> \t}\n>> \n>> +\tif (verify_commit_graph_error)\n>> +\t\treturn verify_commit_graph_error;\n> All right, so we by default short-circuit so that errors found earlier\n> wouldn't cause crash when checking other things.\n>\n> Is it needed, though, in this case? Though I guess it is better to be\n> conservative; lso by terminating after a batch of one type of errors we\n> don't get many different error messages from the same error (i.e. error\n> propagation).\n>\n>> +\n>> +\tfor (i = 0; i < g->num_commits; i++) {\n>> +\t\tstruct commit *odb_commit;\n>> +\n>> +\t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n>> +\n>> +\t\todb_commit = (struct commit *)create_object(cur_oid.hash, alloc_commit_node());\n> Do we really need to keep all those commits from the object database in\n> memory (in the object::obj_hash hash table)? Perhaps using something\n> like Flywheel / Recycler pattern would be a better solution (if\n> possible)?\n>\n> Though perhaps this doesn't matter much with respect to memory use.\n>\n>> +\t\tif (parse_commit_internal(odb_commit, 0, 0)) {\n> Just a reminder to myself: the signature is\n>\n> int parse_commit_internal(struct commit *item, int quiet_on_missing, int use_commit_graph)\n>\n>\n> Hmmm... I wonder if with two boolean paramaters wouldn't it be better to\n> use flags parameter, i.e.\n>\n> int parse_commit_internal(struct commit *item, int flags)\n>\n> ...\n>\n> parse_commit_internal(commit, QUIET_ON_MISSING | USE_COMMIT_GRAPH)\n>\n> But I guess that it is not worth it (especially for internal-ish\n> function).\n>\n>> +\t\t\tgraph_report(\"failed to parse %s from object database\",\n>> +\t\t\t\t oid_to_hex(&cur_oid));\n> Wouldn't parse_commit_internal() show it's own error message, in\n> addition to the one above?\n>\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\t}\n>> +\n>> \treturn verify_commit_graph_error;\n>> }\n>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n>> index c050ef980b..996a016239 100755\n>> --- a/t/t5318-commit-graph.sh\n>> +++ b/t/t5318-commit-graph.sh\n>> @@ -247,6 +247,7 @@ test_expect_success 'git commit-graph verify' '\n>> \tgit commit-graph verify >output\n>> '\n>> \n>> +NUM_COMMITS=9\n>> HASH_LEN=20\n>> GRAPH_BYTE_VERSION=4\n>> GRAPH_BYTE_HASH=5\n>> @@ -265,6 +266,7 @@ GRAPH_BYTE_FANOUT1=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 4`\n>> GRAPH_BYTE_FANOUT2=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 255`\n>> GRAPH_OID_LOOKUP_OFFSET=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 256`\n>> GRAPH_BYTE_OID_LOOKUP_ORDER=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8`\n>> +GRAPH_BYTE_OID_LOOKUP_MISSING=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 4 + 10`\n> All right, so we modify 10-the byte of OID of 5-th commit out of 9,\n> am I correct here?\n>\n>> \n>> # usage: corrupt_graph_and_verify \n>> # Manipulates the commit-graph file at the position\n>> @@ -334,4 +336,9 @@ test_expect_success 'detect incorrect OID order' '\n>> \t\t\"incorrect OID order\"\n>> '\n>> \n>> +test_expect_success 'detect OID not in object database' '\n>> +\tcorrupt_graph_and_verify $GRAPH_BYTE_OID_LOOKUP_MISSING \"\\01\" \\\n>> +\t\t\"from object database\"\n>> +'\n> I guess that if we ensure that OIDs are constant, you have chosen the\n> change to actually corrupt the OID in OID Lookup chunk to point to OID\n> that is not in the object database (while still not violating the\n> constraint that OID in OID Lookup chunk needs to be sorted).\n>\n>> +\n>> test_done\n> All right (well, except for `expr ... ` --> $(( ... )) change).\n>\n\n"},{"id":"348897","messageId":"d59f5723-7fb6-56d6-af2b-b8263cfdf354@gmail.com","threadId":"48311","inReplyTo":"86zi0gvl1h.fsf@gmail.com","subject":"Re: [PATCH v3 11/20] commit-graph: verify root tree OIDs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-05-31T13:16:59Z","receivedAt":"2018-05-31T13:17:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/30/2018 6:24 PM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> The 'verify' subcommand must compare the commit content parsed from the\n>> commit-graph and compare it against the content in the object database.\n> You have \"compare\" twice in the above sentence.\n>\n>> Use lookup_commit() and parse_commit_in_graph_one() to parse the commits\n>> from the graph and compare against a commit that is loaded separately\n>> and parsed directly from the object database.\n> All right, that looks like a nice extension of what was done in previous\n> patch. We want to check that cache (serialized commit graph) matches\n> reality (object database).\n>\n>> Add checks for the root tree OID.\n> All right; isn't it that now we check almost all information from\n> commit-graph that hs match in object database (with exception of commit\n> parents, I think).\n>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> commit-graph.c | 17 ++++++++++++++++-\n>> t/t5318-commit-graph.sh | 7 +++++++\n>> 2 files changed, 23 insertions(+), 1 deletion(-)\n>>\n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index 0420ebcd87..19ea369fc6 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> @@ -880,6 +880,8 @@ int verify_commit_graph(struct commit_graph *g)\n>> \t\treturn verify_commit_graph_error;\n> NOTE: we will be checking Commit Data chunk; I think it would be good\n> idea to verify that size of Commit Data chunk matches (N * (H + 16) bytes)\n> that format gives us, so that we don't accidentally red outside of\n> memory if something got screwed up (like number of commits being wrong,\n> or file truncated).\n\nThis is actually how we calculate 'num_commits' during \nload_commit_graph_one():\n\n     if (last_chunk_id == GRAPH_CHUNKID_OIDLOOKUP)\n     {\n         graph->num_commits = (chunk_offset - last_chunk_offset)\n                              / graph->hash_len;\n     }\n\nSo, if the chunk doesn't match N*(H+16), we detect this because \nFANOUT[255] != N.\n\n(There is one caveat here: (chunk_offset - last_chunk_offset) may not be \na multiple of hash_len, and \"accidentally\" truncate to N in the \ndivision. I'll add more careful checks for this.)\n\nWe also check out-of-bounds offsets in that method.\n\n>\n>> \n>> \tfor (i = 0; i < g->num_commits; i++) {\n>> +\t\tstruct commit *graph_commit;\n>> +\n>> \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n>> \n>> \t\tif (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n>> @@ -897,6 +899,11 @@ int verify_commit_graph(struct commit_graph *g)\n>> \n>> \t\t\tcur_fanout_pos++;\n>> \t\t}\n>> +\n>> +\t\tgraph_commit = lookup_commit(&cur_oid);\n> So now I see why we add all commits to memory (to hash structure).\n>\n>> +\t\tif (!parse_commit_in_graph_one(g, graph_commit))\n>> +\t\t\tgraph_report(\"failed to parse %s from commit-graph\",\n>> +\t\t\t\t oid_to_hex(&cur_oid));\n> All right, this verifies that commit in OID Lookup chunk has parse-able\n> data in Commit Data chunk, isn't it?\n>\n>> \t}\n>> \n>> \twhile (cur_fanout_pos < 256) {\n>> @@ -913,16 +920,24 @@ int verify_commit_graph(struct commit_graph *g)\n>> \t\treturn verify_commit_graph_error;\n>> \n>> \tfor (i = 0; i < g->num_commits; i++) {\n>> -\t\tstruct commit *odb_commit;\n>> +\t\tstruct commit *graph_commit, *odb_commit;\n>> \n>> \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n>> \n>> +\t\tgraph_commit = lookup_commit(&cur_oid);\n>> \t\todb_commit = (struct commit *)create_object(cur_oid.hash, alloc_commit_node());\n> All right, so we have commit data from graph, and commit data from the\n> object database.\n>\n>> \t\tif (parse_commit_internal(odb_commit, 0, 0)) {\n>> \t\t\tgraph_report(\"failed to parse %s from object database\",\n>> \t\t\t\t oid_to_hex(&cur_oid));\n>> \t\t\tcontinue;\n>> \t\t}\n>> +\n>> +\t\tif (oidcmp(&get_commit_tree_in_graph_one(g, graph_commit)->object.oid,\n>> +\t\t\t get_commit_tree_oid(odb_commit)))\n>> +\t\t\tgraph_report(\"root tree OID for commit %s in commit-graph is %s != %s\",\n>> +\t\t\t\t oid_to_hex(&cur_oid),\n>> +\t\t\t\t oid_to_hex(get_commit_tree_oid(graph_commit)),\n>> +\t\t\t\t oid_to_hex(get_commit_tree_oid(odb_commit)));\n> Maybe explicitly say that it doesn't match the value from the object\n> database; OTOH this may be too verbose.\n>\n>> \t}\n>> \n>> \treturn verify_commit_graph_error;\n>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n>> index 996a016239..21cc8e82f3 100755\n>> --- a/t/t5318-commit-graph.sh\n>> +++ b/t/t5318-commit-graph.sh\n>> @@ -267,6 +267,8 @@ GRAPH_BYTE_FANOUT2=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 255`\n>> GRAPH_OID_LOOKUP_OFFSET=`expr $GRAPH_FANOUT_OFFSET + 4 \\* 256`\n>> GRAPH_BYTE_OID_LOOKUP_ORDER=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8`\n>> GRAPH_BYTE_OID_LOOKUP_MISSING=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 4 + 10`\n>> +GRAPH_COMMIT_DATA_OFFSET=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* $NUM_COMMITS`\n>> +GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n> All right, so the first is entry into record in Commit Data chunk, and\n> the latter points into tree entry in this record -- which entry is first\n> field:\n>\n> Commit Data (ID: {'C', 'G', 'E', 'T' }) (N * (H + 16) bytes)\n> * The first H bytes are for the OID of the root tree.\n>\n>> \n>> # usage: corrupt_graph_and_verify \n>> # Manipulates the commit-graph file at the position\n>> @@ -341,4 +343,9 @@ test_expect_success 'detect OID not in object database' '\n>> \t\t\"from object database\"\n>> '\n>> \n>> +test_expect_success 'detect incorrect tree OID' '\n>> +\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_TREE \"\\01\" \\\n>> +\t\t\"root tree OID for commit\"\n>> +'\n> All right.\n>\n> I wonder if we can create a test for first check added (that Commit\n> Graph data parses correctly), that is the one with the following error\n> message:\n>\n> \"failed to parse from commit-graph file\".\n>\n>> +\n>> test_done\n> Regards,\n\n"},{"id":"349039","messageId":"86lgbyum8l.fsf@gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-13-dstolee@microsoft.com","subject":"Re: [PATCH v3 12/20] commit-graph: verify parent list","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-06-01T23:21:14Z","receivedAt":"2018-06-01T23:21:24Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> The commit-graph file stores parents in a two-column portion of the\n> commit data chunk. If there is only one parent, then the second column\n> stores 0xFFFFFFFF to indicate no second parent.\n\nAll right, it is certainly nice to have this information, but isn't it\nsomething that one caan find in Documentation/technical/commit-graph-format.txt?\n\n>\n> The 'verify' subcommand checks the parent list for the commit loaded\n> from the commit-graph and the one parsed from the object database. Test\n> these checks for corrupt parents, too many parents, and wrong parents.\n>\n> The octopus merge will be tested in a later commit.\n\nDoes this mean that after this commit but before the next one the\n'verify' subcommand would have false negatives for octopus merges\n(falsely indicating that commit-graph is invalid)?\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 25 +++++++++++++++++++++++++\n> t/t5318-commit-graph.sh | 18 ++++++++++++++++++\n> 2 files changed, 43 insertions(+)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 19ea369fc6..fff22dc0c3 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -921,6 +921,7 @@ int verify_commit_graph(struct commit_graph *g)\n> \n> \tfor (i = 0; i < g->num_commits; i++) {\n> \t\tstruct commit *graph_commit, *odb_commit;\n> +\t\tstruct commit_list *graph_parents, *odb_parents;\n> \n> \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n> \n> @@ -938,6 +939,30 @@ int verify_commit_graph(struct commit_graph *g)\n> \t\t\t\t oid_to_hex(&cur_oid),\n> \t\t\t\t oid_to_hex(get_commit_tree_oid(graph_commit)),\n> \t\t\t\t oid_to_hex(get_commit_tree_oid(odb_commit)));\n> +\n> +\t\tgraph_parents = graph_commit->parents;\n> +\t\todb_parents = odb_commit->parents;\n> +\n> +\t\twhile (graph_parents) {\n> +\t\t\tif (odb_parents == NULL) {\n> +\t\t\t\tgraph_report(\"commit-graph parent list for commit %s is too long\",\n> +\t\t\t\t\t oid_to_hex(&cur_oid));\n> +\t\t\t\tbreak;\n> +\t\t\t}\n\nAll right, so this would catch the situation where there are more\nparents for a commit in commit-graph than they are in the object\ndatabase version.\n\n> +\n> +\t\t\tif (oidcmp(&graph_parents->item->object.oid, &odb_parents->item->object.oid))\n> +\t\t\t\tgraph_report(\"commit-graph parent for %s is %s != %s\",\n> +\t\t\t\t\t oid_to_hex(&cur_oid),\n> +\t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid),\n> +\t\t\t\t\t oid_to_hex(&odb_parents->item->object.oid));\n\nAll right, so this would catch the situation where parents do not match\nbetween commit-graph and the object database.\n\n> +\n> +\t\t\tgraph_parents = graph_parents->next;\n> +\t\t\todb_parents = odb_parents->next;\n> +\t\t}\n> +\n> +\t\tif (odb_parents != NULL)\n> +\t\t\tgraph_report(\"commit-graph parent list for commit %s terminates early\",\n> +\t\t\t\t oid_to_hex(&cur_oid));\n\nSo this would catch the situation where there are more parents for a\ncommit in the object database than they are in the commit-graph. Does\nthis handle octopus merges automatically, or is it left for the future\nwork/commit?\n\n> \t}\n> \n> \treturn verify_commit_graph_error;\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 21cc8e82f3..12f0d7f54d 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -269,6 +269,9 @@ GRAPH_BYTE_OID_LOOKUP_ORDER=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 8`\n> GRAPH_BYTE_OID_LOOKUP_MISSING=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* 4 + 10`\n> GRAPH_COMMIT_DATA_OFFSET=`expr $GRAPH_OID_LOOKUP_OFFSET + $HASH_LEN \\* $NUM_COMMITS`\n> GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n> +GRAPH_BYTE_COMMIT_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN`\n> +GRAPH_BYTE_COMMIT_EXTRA_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4`\n> +GRAPH_BYTE_COMMIT_WRONG_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3`\n> \n> # usage: corrupt_graph_and_verify \n> # Manipulates the commit-graph file at the position\n> @@ -348,4 +351,19 @@ test_expect_success 'detect incorrect tree OID' '\n> \t\t\"root tree OID for commit\"\n> '\n> \n> +test_expect_success 'detect incorrect parent int-id' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_PARENT \"\\01\" \\\n> +\t\t\"invalid parent\"\n> +'\n\nSo this would actually be caught by code introduced earlier, and not by\nthe one introduced in this commit -- but logically this test belongs\nhere, ian't it?\n\n> +\n> +test_expect_success 'detect extra parent int-id' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_EXTRA_PARENT \"\\00\" \\\n> +\t\t\"is too long\"\n> +'\n\nO.K., so the commit has one parent and we have corrupted it to read as\nif it had more than one (and commit-graph would then have more parents\nthan reality, that is the object database).\n\nSidenote: I think we can use regexp for checking if the error message\nmatches, isn't it?\n\n> +\n> +test_expect_success 'detect incorrect tree OID' '\n\nErrr... what? The name of this test seems very wrong...\n\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_WRONG_PARENT \"\\01\" \\\n> +\t\t\"commit-graph parent for\"\n> +'\n\nSo here you modify the prent list in commit graph so that the commit\nnumber points fits within commit-graph; it would of course make the\ncommit-graph and the object database version of parents do not match.\nGood.\n\n> +\n> test_done\n"},{"id":"349065","messageId":"CACsJy8A8NofWCo0MbMxCe=xKTrnAqmM5D+FHSJXC-wO-DPnXwg@mail.gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-10-dstolee@microsoft.com","subject":"Re: [PATCH v3 09/20] commit-graph: verify corrupt OID fanout and lookup","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-02T04:38:38Z","receivedAt":"2018-06-02T04:39:12Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, May 24, 2018 at 6:25 PM, Derrick Stolee wrote:\n> + if (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n> + graph_report(\"commit-graph has incorrect OID order: %s then %s\",\n> + oid_to_hex(&prev_oid),\n> + oid_to_hex(&cur_oid));\n\nShould these strings be marked for translation with _()?\n-- \nDuy\n"},{"id":"349087","messageId":"868t7xv0km.fsf@gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-14-dstolee@microsoft.com","subject":"Re: [PATCH v3 13/20] commit-graph: verify generation number","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-06-02T12:23:53Z","receivedAt":"2018-06-02T12:24:03Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> While iterating through the commit parents, perform the generation\n> number calculation and compare against the value stored in the\n> commit-graph.\n\nAll right, that's good.\n\nWhat about commit-graph files that have GENERATION_NUMBER_ZERO for all\nits commits (because we verify single commit-graph file only, there\nwouldn't be GENERATION_NUMBER_ZERO mixed with non-zero generation\nnumbers)?\n\nUnless we can assume that no commit-graph file in the wild would have\nGENERATION_NUMBER_ZERO.\n\n>\n> The tests demonstrate that having a different set of parents affects\n> the generation number calculation, and this value propagates to\n> descendants. Hence, we drop the single-line condition on the output.\n\nI don't understand what part of changes this paragraph of the commit\nmessage refers to.\n\nAnyway, changing parents may not lead to changed generation numbers;\ntake for example commit with single parent, which we change to other\ncommit with the same generation number.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 18 ++++++++++++++++++\n> t/t5318-commit-graph.sh | 6 ++++++\n\nSidenote: I have just realized that it may be better to put\nvalidation-related tests into different test file.\n\n> 2 files changed, 24 insertions(+)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index fff22dc0c3..ead92460c1 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -922,6 +922,7 @@ int verify_commit_graph(struct commit_graph *g)\n> \tfor (i = 0; i < g->num_commits; i++) {\n> \t\tstruct commit *graph_commit, *odb_commit;\n> \t\tstruct commit_list *graph_parents, *odb_parents;\n> +\t\tuint32_t max_generation = 0;\n> \n> \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n> \n> @@ -956,6 +957,9 @@ int verify_commit_graph(struct commit_graph *g)\n> \t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid),\n> \t\t\t\t\t oid_to_hex(&odb_parents->item->object.oid));\n> \n> +\t\t\tif (graph_parents->item->generation > max_generation)\n> +\t\t\t\tmax_generation = graph_parents->item->generation;\n> +\n\nAll right, that calculates the maximum of generation number of commit\nparents.\n\n> \t\t\tgraph_parents = graph_parents->next;\n> \t\t\todb_parents = odb_parents->next;\n> \t\t}\n> @@ -963,6 +967,20 @@ int verify_commit_graph(struct commit_graph *g)\n> \t\tif (odb_parents != NULL)\n> \t\t\tgraph_report(\"commit-graph parent list for commit %s terminates early\",\n> \t\t\t\t oid_to_hex(&cur_oid));\n> +\n> +\t\t/*\n> +\t\t * If one of our parents has generation GENERATION_NUMBER_MAX, then\n> +\t\t * our generation is also GENERATION_NUMBER_MAX. Decrement to avoid\n> +\t\t * extra logic in the following condition.\n> +\t\t */\n\nNice trick.\n\n> +\t\tif (max_generation == GENERATION_NUMBER_MAX)\n> +\t\t\tmax_generation--;\n\nWhat about GENERATION_NUMBER_ZERO?\n\n> +\n> +\t\tif (graph_commit->generation != max_generation + 1)\n> +\t\t\tgraph_report(\"commit-graph generation for commit %s is %u != %u\",\n> +\t\t\t\t oid_to_hex(&cur_oid),\n> +\t\t\t\t graph_commit->generation,\n> +\t\t\t\t max_generation + 1);\n\nI think we should also check that generation numbers do not exceed\nGENERATION_NUMBER_MAX... unless it is already taken care of?\n\n> \t}\n> \n> \treturn verify_commit_graph_error;\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 12f0d7f54d..673b0d37d5 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -272,6 +272,7 @@ GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n> GRAPH_BYTE_COMMIT_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN`\n> GRAPH_BYTE_COMMIT_EXTRA_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4`\n> GRAPH_BYTE_COMMIT_WRONG_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3`\n> +GRAPH_BYTE_COMMIT_GENERATION=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 8`\n> \n> # usage: corrupt_graph_and_verify \n> # Manipulates the commit-graph file at the position\n> @@ -366,4 +367,9 @@ test_expect_success 'detect incorrect tree OID' '\n> \t\t\"commit-graph parent for\"\n> '\n> \n> +test_expect_success 'detect incorrect generation number' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_GENERATION \"\\01\" \\\n\nI assume that you have checked that it actually corrupts generation\nnumber (without affecting commit date).\n\n> +\t\t\"generation\"\n\nA very minor nitpick: Not \"generation for commit\"?\n\n> +'\n> +\n> test_done\n"},{"id":"349089","messageId":"8636y5v0bn.fsf@gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-15-dstolee@microsoft.com","subject":"Re: [PATCH v3 14/20] commit-graph: verify commit date","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-06-02T12:29:16Z","receivedAt":"2018-06-02T12:29:24Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\nNice and simple. The only possible question may be the ordering of\npatches in the series, namely whether this change should be before or\nafter test checking generation numbers.\n\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 6 ++++++\n> t/t5318-commit-graph.sh | 6 ++++++\n> 2 files changed, 12 insertions(+)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index ead92460c1..d2b291aca2 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -981,6 +981,12 @@ int verify_commit_graph(struct commit_graph *g)\n> \t\t\t\t oid_to_hex(&cur_oid),\n> \t\t\t\t graph_commit->generation,\n> \t\t\t\t max_generation + 1);\n> +\n> +\t\tif (graph_commit->date != odb_commit->date)\n> +\t\t\tgraph_report(\"commit date for commit %s in commit-graph is %\"PRItime\" != %\"PRItime,\n> +\t\t\t\t oid_to_hex(&cur_oid),\n> +\t\t\t\t graph_commit->date,\n> +\t\t\t\t odb_commit->date);\n\nAll right. Very straightforward, good.\n\n> \t}\n> \n> \treturn verify_commit_graph_error;\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 673b0d37d5..58adb8246d 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -273,6 +273,7 @@ GRAPH_BYTE_COMMIT_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN`\n> GRAPH_BYTE_COMMIT_EXTRA_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4`\n> GRAPH_BYTE_COMMIT_WRONG_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3`\n> GRAPH_BYTE_COMMIT_GENERATION=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 8`\n> +GRAPH_BYTE_COMMIT_DATE=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 12`\n> \n> # usage: corrupt_graph_and_verify \n> # Manipulates the commit-graph file at the position\n> @@ -372,4 +373,9 @@ test_expect_success 'detect incorrect generation number' '\n> \t\t\"generation\"\n> '\n> \n> +test_expect_success 'detect incorrect commit date' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_DATE \"\\01\" \\\n> +\t\t\"commit date\"\n> +'\n\nAll right.\n\n> +\n> test_done\n"},{"id":"349090","messageId":"86tvqltla9.fsf@gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-16-dstolee@microsoft.com","subject":"Re: [PATCH v3 15/20] commit-graph: test for corrupted octopus edge","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-06-02T12:39:26Z","receivedAt":"2018-06-02T12:39:34Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> The commit-graph file has an extra chunk to store the parent int-ids for\n> parents beyond the first parent for octopus merges. Our test repo has a\n> single octopus merge that we can manipulate to demonstrate the 'verify'\n> subcommand detects incorrect values in that chunk.\n\nIf I understand it correctly the above means that our _reading_ code\nchecks for validity (which then 'git commit-graph verify' uses), just\nthere were not any tests for that.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> t/t5318-commit-graph.sh | 10 ++++++++++\n> 1 file changed, 10 insertions(+)\n>\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 58adb8246d..240aef6add 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -248,6 +248,7 @@ test_expect_success 'git commit-graph verify' '\n> '\n> \n> NUM_COMMITS=9\n> +NUM_OCTOPUS_EDGES=2\n> HASH_LEN=20\n> GRAPH_BYTE_VERSION=4\n> GRAPH_BYTE_HASH=5\n> @@ -274,6 +275,10 @@ GRAPH_BYTE_COMMIT_EXTRA_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4`\n> GRAPH_BYTE_COMMIT_WRONG_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3`\n> GRAPH_BYTE_COMMIT_GENERATION=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 8`\n> GRAPH_BYTE_COMMIT_DATE=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 12`\n> +GRAPH_COMMIT_DATA_WIDTH=`expr $HASH_LEN + 16`\n> +GRAPH_OCTOPUS_DATA_OFFSET=`expr $GRAPH_COMMIT_DATA_OFFSET + \\\n> +\t\t\t\t$GRAPH_COMMIT_DATA_WIDTH \\* $NUM_COMMITS`\n> +GRAPH_BYTE_OCTOPUS=`expr $GRAPH_OCTOPUS_DATA_OFFSET + 4`\n> \n> # usage: corrupt_graph_and_verify \n> # Manipulates the commit-graph file at the position\n> @@ -378,4 +383,9 @@ test_expect_success 'detect incorrect commit date' '\n> \t\t\"commit date\"\n> '\n> \n> +test_expect_success 'detect incorrect parent for octopus merge' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_OCTOPUS \"\\01\" \\\n> +\t\t\"invalid parent\"\n> +'\n\nSo we change the int-id to non-existing commit, and check that\ncommit-graph code checks for that.\n\nWhat about the case when there are octopus merges, but no EDGE chunk\n(which I think we can emulate by changing / corrupting number of\nchunks)?\n\nWhat about the case where int-id of edge in EDGE chunk is correct, that\nis points to a valid commit, but does not agree with what is in the\nobject database (what parents octopus merge has in reality)?\n\nDo we detect the situation where the second parent value in the commit\ndata stores an array position within a Large Edge chunk, but we do not\nreach a value with the most-significant bit on when reaching the end of\nLarge Edge chunk?\n\n> +\n> test_done\n"},{"id":"349091","messageId":"86muwdtcc7.fsf@gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-17-dstolee@microsoft.com","subject":"Re: [PATCH v3 16/20] commit-graph: verify contents match checksum","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-06-02T15:52:40Z","receivedAt":"2018-06-02T15:52:49Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> The commit-graph file ends with a SHA1 hash of the previous contents. If\n> a commit-graph file has errors but the checksum hash is correct, then we\n> know that the problem is a bug in Git and not simply file corruption\n> after-the-fact.\n>\n> Compute the checksum right away so it is the first error that appears,\n> and make the message translatable since this error can be \"corrected\" by\n> a user by simply deleting the file and recomputing. The rest of the\n> errors are useful only to developers.\n\nShould we then provide --quiet / --verbose options, so that ordinary\nuser is not flooded with error messages meant for power users and Git\ndevelopers, then?\n\n>\n> Be sure to continue checking the rest of the file data if the checksum\n> is wrong. This is important for our tests, as we break the checksum as\n> we modify bytes of the commit-graph file.\n\nWell, we could have used sha1sum program, or test-sha1 helper to fix the\nchecksum after corrupting the commit-graph file...\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 16 ++++++++++++++--\n> t/t5318-commit-graph.sh | 6 ++++++\n> 2 files changed, 20 insertions(+), 2 deletions(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index d2b291aca2..a33600c584 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -841,6 +841,7 @@ void write_commit_graph(const char *obj_dir,\n> \toids.nr = 0;\n> }\n> \n> +#define VERIFY_COMMIT_GRAPH_ERROR_HASH 2\n> static int verify_commit_graph_error;\n> \n> static void graph_report(const char *fmt, ...)\n> @@ -860,7 +861,9 @@ static void graph_report(const char *fmt, ...)\n> int verify_commit_graph(struct commit_graph *g)\n> {\n> \tuint32_t i, cur_fanout_pos = 0;\n> -\tstruct object_id prev_oid, cur_oid;\n> +\tstruct object_id prev_oid, cur_oid, checksum;\n> +\tstruct hashfile *f;\n> +\tint devnull;\n> \n> \tif (!g) {\n> \t\tgraph_report(\"no commit-graph file loaded\");\n> @@ -879,6 +882,15 @@ int verify_commit_graph(struct commit_graph *g)\n> \tif (verify_commit_graph_error)\n> \t\treturn verify_commit_graph_error;\n> \n> +\tdevnull = open(\"/dev/null\", O_WRONLY);\n> +\tf = hashfd(devnull, NULL);\n> +\thashwrite(f, g->data, g->data_len - g->hash_len);\n> +\tfinalize_hashfile(f, checksum.hash, CSUM_CLOSE);\n> +\tif (hashcmp(checksum.hash, g->data + g->data_len - g->hash_len)) {\n> +\t\tgraph_report(_(\"the commit-graph file has incorrect checksum and is likely corrupt\"));\n> +\t\tverify_commit_graph_error = VERIFY_COMMIT_GRAPH_ERROR_HASH;\n> +\t}\n\nIs it the best way of calculating the SHA-1 checksum that out internal\nAPIs provide? Is it how SHA-1 checksum is calculated and checked for\npackfiles?\n\n> +\n> \tfor (i = 0; i < g->num_commits; i++) {\n> \t\tstruct commit *graph_commit;\n> \n> @@ -916,7 +928,7 @@ int verify_commit_graph(struct commit_graph *g)\n> \t\tcur_fanout_pos++;\n> \t}\n> \n> -\tif (verify_commit_graph_error)\n> +\tif (verify_commit_graph_error & ~VERIFY_COMMIT_GRAPH_ERROR_HASH)\n> \t\treturn verify_commit_graph_error;\n\nSo if we detected that checksum do not match, but we have not found an\nerror, we say that it is all right?\n\n> \n> \tfor (i = 0; i < g->num_commits; i++) {\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 240aef6add..2680a2ebff 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -279,6 +279,7 @@ GRAPH_COMMIT_DATA_WIDTH=`expr $HASH_LEN + 16`\n> GRAPH_OCTOPUS_DATA_OFFSET=`expr $GRAPH_COMMIT_DATA_OFFSET + \\\n> \t\t\t\t$GRAPH_COMMIT_DATA_WIDTH \\* $NUM_COMMITS`\n> GRAPH_BYTE_OCTOPUS=`expr $GRAPH_OCTOPUS_DATA_OFFSET + 4`\n> +GRAPH_BYTE_FOOTER=`expr $GRAPH_OCTOPUS_DATA_OFFSET + 4 \\* $NUM_OCTOPUS_EDGES`\n> \n> # usage: corrupt_graph_and_verify \n> # Manipulates the commit-graph file at the position\n> @@ -388,4 +389,9 @@ test_expect_success 'detect incorrect parent for octopus merge' '\n> \t\t\"invalid parent\"\n> '\n> \n> +test_expect_success 'detect invalid checksum hash' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_FOOTER \"\\00\" \\\n> +\t\t\"incorrect checksum\"\n\nThis would not work under GETTEXT_POISON, as the message is marked as\ntranslatable, but corrupt_graph_and_verify uses 'grep' and not\n'test_i18grep' from t/test-lib-functions.sh\n\n> +'\n\nIf it is pure checksum corruption, wouldn't this fail because it is not\na failure (exit code is 0)?\n\n> +\n> test_done\n"},{"id":"349092","messageId":"867enhtb6n.fsf@gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-18-dstolee@microsoft.com","subject":"Re: [PATCH v3 17/20] fsck: verify commit-graph","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-06-02T16:17:36Z","receivedAt":"2018-06-02T16:17:45Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> If core.commitGraph is true, verify the contents of the commit-graph\n> during 'git fsck' using the 'git commit-graph verify' subcommand. Run\n> this check on all alternates, as well.\n\nAll right, so we have one config variable to control the use of\nserialized commit-graph feaature. Nice.\n\n>\n> We use a new process for two reasons:\n>\n> 1. The subcommand decouples the details of loading and verifying a\n> commit-graph file from the other fsck details.\n\nAll right, I can agree with that.\n\nOn the other hand using subcommand makes debugging harder, though not in\nthis case (well separated functionality that can be easily called with a\nstandalone command to be debugged).\n\n>\n> 2. The commit-graph verification requires the commits to be loaded\n> in a specific order to guarantee we parse from the commit-graph\n> file for some objects and from the object database for others.\n\nI don't quite understand this. Could you explain it in more detail?\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-fsck.txt | 3 +++\n> builtin/fsck.c | 21 +++++++++++++++++++++\n> t/t5318-commit-graph.sh | 8 ++++++++\n> 3 files changed, 32 insertions(+)\n>\n> diff --git a/Documentation/git-fsck.txt b/Documentation/git-fsck.txt\n> index b9f060e3b2..ab9a93fb9b 100644\n> --- a/Documentation/git-fsck.txt\n> +++ b/Documentation/git-fsck.txt\n> @@ -110,6 +110,9 @@ Any corrupt objects you will have to find in backups or other archives\n> (i.e., you can just remove them and do an 'rsync' with some other site in\n> the hopes that somebody else has the object you have corrupted).\n> \n> +If core.commitGraph is true, the commit-graph file will also be inspected\n\nShouldn't we use `core.commitGraph` here?\n\n> +using 'git commit-graph verify'. See linkgit:git-commit-graph[1].\n> +\n> Extracted Diagnostics\n> ---------------------\n> \n> diff --git a/builtin/fsck.c b/builtin/fsck.c\n> index ef78c6c00c..a6d5045b77 100644\n> --- a/builtin/fsck.c\n> +++ b/builtin/fsck.c\n> @@ -16,6 +16,7 @@\n> #include \"streaming.h\"\n> #include \"decorate.h\"\n> #include \"packfile.h\"\n> +#include \"run-command.h\"\n> \n> #define REACHABLE 0x0001\n> #define SEEN 0x0002\n> @@ -45,6 +46,7 @@ static int name_objects;\n> #define ERROR_REACHABLE 02\n> #define ERROR_PACK 04\n> #define ERROR_REFS 010\n> +#define ERROR_COMMIT_GRAPH 020\n\nMinor nitpick and a sidenote: I wonder if it wouldn't be better to\neither use hexadecimal constants, or use (1 << n) for all ERROR_*\npreprocesor constants.\n\n> \n> static const char *describe_object(struct object *obj)\n> {\n> @@ -815,5 +817,24 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n> \t}\n> \n> \tcheck_connectivity();\n> +\n> +\tif (core_commit_graph) {\n> +\t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n> +\t\tconst char *verify_argv[] = { \"commit-graph\", \"verify\", NULL, NULL, NULL, NULL };\n\nI see that NULL at index 2 and 3 (at 3rd and 4th place) are here for\n\"--object-dir\" and , the last one is\nterminator for that case, but what is next to last NULL (at 5th place)\nfor?\n\n> +\t\tcommit_graph_verify.argv = verify_argv;\n> +\t\tcommit_graph_verify.git_cmd = 1;\n> +\n> +\t\tif (run_command(&commit_graph_verify))\n> +\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n> +\n> +\t\tprepare_alt_odb();\n> +\t\tfor (alt = alt_odb_list; alt; alt = alt->next) {\n> +\t\t\tverify_argv[2] = \"--object-dir\";\n> +\t\t\tverify_argv[3] = alt->path;\n> +\t\t\tif (run_command(&commit_graph_verify))\n> +\t\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n> +\t\t}\n> +\t}\n\nFor performance reasons it may be better to start those 'git\ncommit-graph verify' commands asynchronously earlier, so that they can\nrun in parallel / concurrently wth other checks, and wait for them and\nget their error code at the end of git-fsck run.\n\nBut that is probably better left for a separate commit.\n\n> +\n> \treturn errors_found;\n> }\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 2680a2ebff..4941937163 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -394,4 +394,12 @@ test_expect_success 'detect invalid checksum hash' '\n> \t\t\"incorrect checksum\"\n> '\n> \n> +test_expect_success 'git fsck (checks commit-graph)' '\n> +\tcd \"$TRASH_DIRECTORY/full\" &&\n> +\tgit fsck &&\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_FOOTER \"\\00\" \\\n> +\t\t\"incorrect checksum\" &&\n> +\ttest_must_fail git fsck\n> +'\n\nAll right; though the same caveats apply as with previous commit in\nseries. Perhaps it would be better to truncate commit-graph file, or\ncorrupt it in some 'random' place.\n\n> +\n> test_done\n\nBest,\n-- \nJakub Narębski\n"},{"id":"349093","messageId":"86k1rhrt1u.fsf@gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-19-dstolee@microsoft.com","subject":"Re: [PATCH v3 18/20] commit-graph: add '--reachable' option","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-06-02T17:34:37Z","receivedAt":"2018-06-02T17:34:46Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> When writing commit-graph files, it can be convenient to ask for all\n> reachable commits (starting at the ref set) in the resulting file. This\n> is particularly helpful when writing to stdin is complicated, such as a\n> future integration with 'git gc' which will call\n> write_commit_graph_reachable() after performing cleanup of the object\n> database.\n\nNice.\n\nThe last sentence of the commit message is a bit long, though, in my\nopinion.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-commit-graph.txt | 8 ++++++--\n> builtin/commit-graph.c | 16 ++++++++++++----\n> commit-graph.c | 32 ++++++++++++++++++++++++++++++++\n> commit-graph.h | 1 +\n> t/t5318-commit-graph.sh | 10 ++++++++++\n> 5 files changed, 61 insertions(+), 6 deletions(-)\n>\n> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> index a222cfab08..dececb79d7 100644\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -38,12 +38,16 @@ Write a commit graph file based on the commits found in packfiles.\n> +\n> With the `--stdin-packs` option, generate the new commit graph by\n> walking objects only in the specified pack-indexes. (Cannot be combined\n> -with --stdin-commits.)\n> +with `--stdin-commits` or `--reachable`.)\n> +\n> With the `--stdin-commits` option, generate the new commit graph by\n> walking commits starting at the commits specified in stdin as a list\n> of OIDs in hex, one OID per line. (Cannot be combined with\n> ---stdin-packs.)\n> +`--stdin-packs` or `--reachable`.)\n> ++\n> +With the `--reachable` option, generate the new commit graph by walking\n> +commits starting at all refs. (Cannot be combined with `--stdin-commits`\n> +or `--stdin-packs`.)\n\nAll right (though I wonder a bit about the restriction).\n\nI think it might be a good idea to describe all of this in the usage\nstring for the 'git commit-graph write', instead of using ''\nplaceholder, that is instead of current:\n\n 'git commit-graph write' [--object-dir ]\n\nuse\n\n 'git commit-graph write' [--stdin-commits | --stdin-packs | --reachable]\n [--append] [--object-dir ]\n\nor something like that.\n\n> +\n> With the `--append` option, include all commits that are present in the\n> existing commit-graph file.\n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index 0433dd6e20..20ce6437ae 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -9,7 +9,7 @@ static char const * const builtin_commit_graph_usage[] = {\n> \tN_(\"git commit-graph [--object-dir ]\"),\n> \tN_(\"git commit-graph read [--object-dir ]\"),\n> \tN_(\"git commit-graph verify [--object-dir ]\"),\n> -\tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n> +\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n\nAll right, very straightforward. I guess they are put in [almost]\nalphabetical order, or is there some other reasoning behind the ordering\nused (which is different from the one in the manpage)?\n\n> \tNULL\n> };\n> \n> @@ -24,12 +24,13 @@ static const char * const builtin_commit_graph_read_usage[] = {\n> };\n> \n> static const char * const builtin_commit_graph_write_usage[] = {\n> -\tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n> +\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n\nThe same.\n\n> \tNULL\n> };\n> \n> static struct opts_commit_graph {\n> \tconst char *obj_dir;\n> +\tint reachable;\n> \tint stdin_packs;\n> \tint stdin_commits;\n> \tint append;\n> @@ -130,6 +131,8 @@ static int graph_write(int argc, const char **argv)\n> \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n> \t\t\tN_(\"dir\"),\n> \t\t\tN_(\"The object directory to store the graph\")),\n> +\t\tOPT_BOOL(0, \"reachable\", &opts.reachable,\n> +\t\t\tN_(\"start walk at all refs\")),\n\nErrr... does '--no-reachable' makes sense? Because if I am right\ncurrently it is supported, isn't it.\n\n> \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n> \t\t\tN_(\"scan pack-indexes listed by stdin for commits\")),\n> \t\tOPT_BOOL(0, \"stdin-commits\", &opts.stdin_commits,\n> @@ -143,11 +146,16 @@ static int graph_write(int argc, const char **argv)\n> \t\t\t builtin_commit_graph_write_options,\n> \t\t\t builtin_commit_graph_write_usage, 0);\n> \n> -\tif (opts.stdin_packs && opts.stdin_commits)\n> -\t\tdie(_(\"cannot use both --stdin-commits and --stdin-packs\"));\n> +\tif (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)\n\nNice trick.\n\n> +\t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\n\nIt is a pity that parseopt API does not have direct support for mutually\nexclusive groups of boolean options, like ArgumentParser.add_mutually_exclusive_group()\nin Python's argparse.\n\nStill, you need to use what it is there.\n\n> \tif (!opts.obj_dir)\n> \t\topts.obj_dir = get_object_directory();\n> \n> +\tif (opts.reachable) {\n> +\t\twrite_commit_graph_reachable(opts.obj_dir, opts.append);\n> +\t\treturn 0;\n> +\t}\n\nJust using the option.\n\n> +\n> \tif (opts.stdin_packs || opts.stdin_commits) {\n> \t\tstruct strbuf buf = STRBUF_INIT;\n> \t\tlines_nr = 0;\n> diff --git a/commit-graph.c b/commit-graph.c\n> index a33600c584..057d734926 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -6,6 +6,7 @@\n> #include \"packfile.h\"\n> #include \"commit.h\"\n> #include \"object.h\"\n> +#include \"refs.h\"\n> #include \"revision.h\"\n> #include \"sha1-lookup.h\"\n> #include \"commit-graph.h\"\n> @@ -651,6 +652,37 @@ static void compute_generation_numbers(struct packed_commit_list* commits)\n> \t}\n> }\n> \n> +struct hex_list {\n> +\tchar **hex_strs;\n> +\tint hex_nr;\n> +\tint hex_alloc;\n> +};\n\nIs this what git-for-each-ref / git-branch / git-tag uses?\n\nWould it be possible to use for example string-list API (documented in\nstring-list.h) instead? Anyway, it looks like the use of allocation\ngrowing API is simple enough... though perhaps it could be made simpler\nby noticing that all strings have the same width.\n\n> +\n> +static int add_ref_to_list(const char *refname,\n> +\t\t\t const struct object_id *oid,\n> +\t\t\t int flags, void *cb_data)\n> +{\n> +\tstruct hex_list *list = (struct hex_list*)cb_data;\n> +\n> +\tALLOC_GROW(list->hex_strs, list->hex_nr + 1, list->hex_alloc);\n> +\tlist->hex_strs[list->hex_nr] = xcalloc(GIT_MAX_HEXSZ + 1, 1);\n> +\tstrcpy(list->hex_strs[list->hex_nr], oid_to_hex(oid));\n\nWouldn't it be better to use strdup or xstrdup instead of\nxcalloc+strcpy?\n\n> +\tlist->hex_nr++;\n> +\treturn 0;\n> +}\n> +\n> +void write_commit_graph_reachable(const char *obj_dir, int append)\n> +{\n> +\tstruct hex_list list;\n> +\tlist.hex_nr = 0;\n> +\tlist.hex_alloc = 128;\n> +\tALLOC_ARRAY(list.hex_strs, list.hex_alloc);\n> +\n> +\tfor_each_ref(add_ref_to_list, &list);\n> +\n> +\twrite_commit_graph(obj_dir, NULL, 0, (const char **)list.hex_strs, list.hex_nr, append);\n\nWhere do we free the allocated data and allocated strings? If they are\ncleaned by process exit, perhaps they need to be UNLEAK-ed?\n\n> +}\n> +\n> void write_commit_graph(const char *obj_dir,\n> \t\t\tconst char **pack_indexes,\n> \t\t\tint nr_packs,\n> diff --git a/commit-graph.h b/commit-graph.h\n> index 71a39c5a57..9a06a5f188 100644\n> --- a/commit-graph.h\n> +++ b/commit-graph.h\n> @@ -46,6 +46,7 @@ struct commit_graph {\n> \n> struct commit_graph *load_commit_graph_one(const char *graph_file);\n> \n> +void write_commit_graph_reachable(const char *obj_dir, int append);\n> void write_commit_graph(const char *obj_dir,\n> \t\t\tconst char **pack_indexes,\n> \t\t\tint nr_packs,\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 4941937163..a659620332 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -205,6 +205,16 @@ test_expect_success 'build graph from commits with append' '\n> graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n> graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n> \n> +test_expect_success 'build graph using --reachable' '\n> +\tcd \"$TRASH_DIRECTORY/full\" &&\n> +\tgit commit-graph write --reachable &&\n> +\ttest_path_is_file $objdir/info/commit-graph &&\n> +\tgraph_read_expect \"11\" \"large_edges\"\n> +'\n\nAll right, here we check that commit-graph has expected features (11\ncommits, and large_edges optional chunk).\n\nPerhaps we could also check that different equivalent ways of creating\nserialized commit graph file produce byte-for-byte identical file?\n\n> +\n> +graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n> +graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n\nAll right, this supposedly tests that behavior does not change whether\nwe are using or we are not using the commit-graph feature... but I have\njust noticed that raph_git_two_modes() uses `git -c core.graph=true`;\nshouldn't it be `git -c core.commitGraph=true`?\n\n> +\n> test_expect_success 'setup bare repo' '\n> \tcd \"$TRASH_DIRECTORY\" &&\n> \tgit clone --bare --no-local full bare &&\n"},{"id":"349094","messageId":"866031rrq1.fsf@gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-20-dstolee@microsoft.com","subject":"Re: [PATCH v3 19/20] gc: automatically write commit-graph files","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-06-02T18:03:18Z","receivedAt":"2018-06-02T18:03:26Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> The commit-graph file is a very helpful feature for speeding up git\n> operations. In order to make it more useful, write the commit-graph file\n> by default during standard garbage collection operations.\n\nI think you meant here \"make it possible to write the commit-graph file\nduring standard garbage collection operations.\" (i.e. add \"make it\npossible\" because it hides behind new config option, and remove \"by\ndefault\" because currently it is not turned on by default).\n\n>\n> Add a 'gc.commitGraph' config setting that triggers writing a\n> commit-graph file after any non-trivial 'git gc' command. Defaults to\n> false while the commit-graph feature matures. We specifically do not\n> want to turn this on by default until the commit-graph feature is fully\n\ns/turn this on/have this on/ I think.\n\n> integrated with history-modifying features like shallow clones.\n\nTwo things.\n\nFirst, shallow clones, replacement mechanims (git-replace) and grafts\nare not \"history-modifying\" features; this name is in my opinion\nreserved for history-rewriting features such as interactive rebase, the\n`git filter-branch` feature or out-of-tree BFG Repo Cleaner or\nreposurgeon tools. They alter the _view_ of history; they should be\nIMVHO named \"history-view-altering\" features -- though I agree this is\nmouthful.\n\nSecond, shouldn't we, as Martin Ågren said, warn about the issue in the\nmanpage for git-commit-graph?\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/config.txt | 6 ++++++\n> Documentation/git-gc.txt | 4 ++++\n> builtin/gc.c | 6 ++++++\n> t/t5318-commit-graph.sh | 14 ++++++++++++++\n> 4 files changed, 30 insertions(+)\n>\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index 11f027194e..9a3abd87e7 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -1553,6 +1553,12 @@ gc.autoDetach::\n> \tMake `git gc --auto` return immediately and run in background\n> \tif the system supports it. Default is true.\n> \n> +gc.commitGraph::\n> +\tIf true, then gc will rewrite the commit-graph file after any\n> +\tchange to the object database. If '--auto' is used, then the\n> +\tcommit-graph will not be updated unless the threshold is met.\n\nWhat threshold? Ah, thresholds defined for `git gc --auto` (gc.auto,\ngc.autoPackLimit, gc.logExpiry,...).\n\n> +\tSee linkgit:git-commit-graph[1] for details.\n\nYou missed declaring the default value for this config option.\n\n> +\n> gc.logExpiry::\n> \tIf the file gc.log exists, then `git gc --auto` won't run\n> \tunless that file is more than 'gc.logExpiry' old. Default is\n> diff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\n> index 571b5a7e3c..17dd654a59 100644\n> --- a/Documentation/git-gc.txt\n> +++ b/Documentation/git-gc.txt\n> @@ -119,6 +119,10 @@ The optional configuration variable `gc.packRefs` determines if\n> it within all non-bare repos or it can be set to a boolean value.\n> This defaults to true.\n> \n> +The optional configuration variable 'gc.commitGraph' determines if\n> +'git gc' runs 'git commit-graph write'. This can be set to a boolean\n\nShould it be \"runs\" or \"should run\"?\n\n> +value. This defaults to false.\n\nShould it be '...' or `...`? Below we have `gc.aggresiveWindow`, above\nwe have 'gc.commitGraph', for example.\n\n> +\n> The optional configuration variable `gc.aggressiveWindow` controls how\n> much time is spent optimizing the delta compression of the objects in\n> the repository when the --aggressive option is specified. The larger\n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index 77fa720bd0..efd214a59f 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -20,6 +20,7 @@\n> #include \"argv-array.h\"\n> #include \"commit.h\"\n> #include \"packfile.h\"\n> +#include \"commit-graph.h\"\n> \n> #define FAILED_RUN \"failed to run %s\"\n> \n> @@ -34,6 +35,7 @@ static int aggressive_depth = 50;\n> static int aggressive_window = 250;\n> static int gc_auto_threshold = 6700;\n> static int gc_auto_pack_limit = 50;\n> +static int gc_commit_graph = 0;\n> static int detach_auto = 1;\n> static timestamp_t gc_log_expire_time;\n> static const char *gc_log_expire = \"1.day.ago\";\n> @@ -121,6 +123,7 @@ static void gc_config(void)\n> \tgit_config_get_int(\"gc.aggressivedepth\", &aggressive_depth);\n> \tgit_config_get_int(\"gc.auto\", &gc_auto_threshold);\n> \tgit_config_get_int(\"gc.autopacklimit\", &gc_auto_pack_limit);\n> +\tgit_config_get_bool(\"gc.commitgraph\", &gc_commit_graph);\n> \tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n> \tgit_config_get_expiry(\"gc.pruneexpire\", &prune_expire);\n> \tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n> @@ -480,6 +483,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n> \tif (pack_garbage.nr > 0)\n> \t\tclean_pack_garbage();\n> \n> +\tif (gc_commit_graph)\n> +\t\twrite_commit_graph_reachable(get_object_directory(), 0);\n> +\n\nNice.\n\nThough now I wonder when appending should be used...\n\n> \tif (auto_gc && too_many_loose_objects())\n> \t\twarning(_(\"There are too many unreachable loose objects; \"\n> \t\t\t\"run 'git prune' to remove them.\"));\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index a659620332..d20b17586f 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -245,6 +245,20 @@ test_expect_success 'perform fast-forward merge in full repo' '\n> \ttest_cmp expect output\n> '\n> \n> +test_expect_success 'check that gc clears commit-graph' '\n\nI wouldn't use the word \"clears\" here...\n\n> +\tcd \"$TRASH_DIRECTORY/full\" &&\n> +\tgit commit --allow-empty -m \"blank\" &&\n> +\tgit commit-graph write --reachable &&\n> +\tcp $objdir/info/commit-graph commit-graph-before-gc &&\n> +\tgit reset --hard HEAD~1 &&\n> +\tgit config gc.commitGraph true &&\n> +\tgit gc &&\n> +\tcp $objdir/info/commit-graph commit-graph-after-gc &&\n> +\t! test_cmp commit-graph-before-gc commit-graph-after-gc &&\n> +\tgit commit-graph write --reachable &&\n> +\ttest_cmp commit-graph-after-gc $objdir/info/commit-graph\n> +'\n\n...but otherwise, nice test: it checks that git-gc after rewriting\nhistory changes commit-graph file, and that the changed file is what we\nexpect it to be (note: here we compare commit-graph files directly, and\nnot just check the features via 'git commit-graph read').\n\n> +\n> # the verify tests below expect the commit-graph to contain\n> # exactly the commits reachable from the commits/8 branch.\n> # If the file changes the set of commits in the list, then the\n"},{"id":"349095","messageId":"86wovhqc0r.fsf@gmail.com","threadId":"48311","inReplyTo":"20180524162504.158394-21-dstolee@microsoft.com","subject":"Re: [PATCH v3 20/20] commit-graph: update design document","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-06-02T18:27:48Z","receivedAt":"2018-06-02T18:27:56Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> The commit-graph feature is now integrated with 'fsck' and 'gc',\n> so remove those items from the \"Future Work\" section of the\n> commit-graph design document.\n\nIt is always nice to have such commit as a summary what was done in the\nseries, and to have up to date roadmap.\n\n>\n> Also remove the section on lazy-loading trees, as that was completed\n> in an earlier patch series.\n\nAdmittedly, this part could have been sent in a separate patch at the\nstart of the series, but it doesn't matter at all; no need for extra\nwork.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/technical/commit-graph.txt | 22 ----------------------\n> 1 file changed, 22 deletions(-)\n>\n> diff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\n> index e1a883eb46..c664acbd76 100644\n> --- a/Documentation/technical/commit-graph.txt\n> +++ b/Documentation/technical/commit-graph.txt\n> @@ -118,9 +118,6 @@ Future Work\n> - The commit graph feature currently does not honor commit grafts. This can\n> be remedied by duplicating or refactoring the current graft logic.\n> \n> -- The 'commit-graph' subcommand does not have a \"verify\" mode that is\n> - necessary for integration with fsck.\n> -\n\nAll right.\n\n> - After computing and storing generation numbers, we must make graph\n> walks aware of generation numbers to gain the performance benefits they\n> enable. This will mostly be accomplished by swapping a commit-date-ordered\n> @@ -130,25 +127,6 @@ Future Work\n> - 'log --topo-order'\n> - 'tag --merged'\n> \n> -- Currently, parse_commit_gently() requires filling in the root tree\n> - object for a commit. This passes through lookup_tree() and consequently\n> - lookup_object(). Also, it calls lookup_commit() when loading the parents.\n> - These method calls check the ODB for object existence, even if the\n> - consumer does not need the content. For example, we do not need the\n> - tree contents when computing merge bases. Now that commit parsing is\n> - removed from the computation time, these lookup operations are the\n> - slowest operations keeping graph walks from being fast. Consider\n> - loading these objects without verifying their existence in the ODB and\n> - only loading them fully when consumers need them. Consider a method\n> - such as \"ensure_tree_loaded(commit)\" that fully loads a tree before\n> - using commit->tree.\n\nAll right, this is about the change done in previous series.\n\n> -\n> -- The current design uses the 'commit-graph' subcommand to generate the graph.\n> - When this feature stabilizes enough to recommend to most users, we should\n> - add automatic graph writes to common operations that create many commits.\n> - For example, one could compute a graph on 'clone', 'fetch', or 'repack'\n> - commands.\n\nAll right; actually it was done by augmenting 'gc' instead.\n\n> -\n> - A server could provide a commit graph file as part of the network protocol\n> to avoid extra calculations by clients. This feature is only of benefit if\n> the user is willing to trust the file, because verifying the file is correct\n\nGood work,\n-- \nJakub Narębski\n"},{"id":"349098","messageId":"86lgbxq8it.fsf@gmail.com","threadId":"48311","inReplyTo":"20180526203036.GL652292@genre.crustytoothpaste.net","subject":"Re: [PATCH v3 02/20] commit-graph: fix GRAPH_MIN_SIZE","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-06-02T19:43:22Z","receivedAt":"2018-06-02T19:43:31Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"brian m. carlson\" writes:\n\n> On Sat, May 26, 2018 at 08:46:09PM +0200, Jakub Narebski wrote:\n>> One issue: in the future when Git moves to NewHash, it could encounter\n>> then both commit-graph files using SHA-1 and using NewHash. What about\n>> GRPH_OID_LEN then: for one of those it would be incorrect. Unless it is\n>> about minimal length of checksum, that is we assume that NewHash would\n>> be longer than SHA-1, but ten why name it GRAPH_OID_LEN?\n>\n> My proposal is that whatever we're using in the .git directory is\n> consistent. If we're using SHA-1 for objects, then everything is SHA-1.\n> If we're using NewHash for objects, then all data is stored in NewHash\n> (except translation tables and such). Any conversions between SHA-1 and\n> NewHash require a lookup through the standard techniques.\n>\n> I agree that here it would be more helpful if it were a reference to\n> the_hash_algo, and I've applied a patch to my object-id-part14 series to\n> make that conversion.\n\nAll right, I can agree that it would make most sense to always use SHA-1\nfor OID, or always use NewHash for objects. This would make\ncommit-graph file with SHA-1 hash invalid for NewHash-using Git version.\n\nIt would be nice, however, to avoid having to redo all the hard work,\nlike calculating generation numbers (from old commit-graph file, or from\nserver that does not support NewHash yet -- the latter is not\nimplemented, but IIUC planned feature). But we can do it with explicit\nconversion step, e.g. 'git commit-graph convert' or 'upgrade'.\n\nBut all that is in the future.\n-- \nJakub Narębski\n"},{"id":"349100","messageId":"86d0x8rimu.fsf@gmail.com","threadId":"48311","inReplyTo":"4343f593-5650-3b24-54ec-a0d97d5251e9@gmail.com","subject":"Re: [PATCH v3 06/20] commit-graph: add 'verify' subcommand","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-06-02T21:19:37Z","receivedAt":"2018-06-02T21:39:37Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n> On 5/27/2018 6:55 PM, Jakub Narebski wrote:\n>> Derrick Stolee writes:\n[...]\n>>> +static int verify_commit_graph_error;\n>>> +\n>>> +static void graph_report(const char *fmt, ...)\n>>> +{\n>>> +\tva_list ap;\n>>> +\tstruct strbuf sb = STRBUF_INIT;\n>>> +\tverify_commit_graph_error = 1;\n>>> +\n>>> +\tva_start(ap, fmt);\n>>> +\tstrbuf_vaddf(&sb, fmt, ap);\n>>> +\n>>> +\tfprintf(stderr, \"%s\\n\", sb.buf);\n>>> +\tstrbuf_release(&sb);\n>>> +\tva_end(ap);\n>>\n>> Why do you use strbuf_vaddf + fprintf instead of straighforward\n>> vfprintf (or function instead of variable-level macro)?\n>>\n>> Is it because of [string] safety?\n>\n> It's because I've never used this variable-parameter thing before and\n> found a different example.\n>\n> I'll use vfprintf() in v4, as it is simpler.\n\nAll right, if it is not dangerous, then simpler is better.\n\nSidenote: such error messaging is often handled by variadic macros,\ne.g.:\n\n #define eprintf(...) fprintf(stderr, __VA_ARGS__)\n\n[...]\n>>> diff --git a/commit-graph.h b/commit-graph.h\n>>> index 96cccb10f3..71a39c5a57 100644\n>>> --- a/commit-graph.h\n>>> +++ b/commit-graph.h\n>>> @@ -53,4 +53,6 @@ void write_commit_graph(const char *obj_dir,\n>>> \t\t\tint nr_commits,\n>>> \t\t\tint append);\n>>> +int verify_commit_graph(struct commit_graph *g);\n>>> +\n>> Why does this need to be exported? I think it is not used outside of\n>> commit-graph.c, isn't it?\n>\n> Used by builtin/commit-graph.c\n\nAh, true.\n\n[...]\n>>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n>>> index 77d85aefe7..6ca451dfd2 100755\n>>> --- a/t/t5318-commit-graph.sh\n>>> +++ b/t/t5318-commit-graph.sh\n>>> @@ -11,6 +11,11 @@ test_expect_success 'setup full repo' '\n>>> \tobjdir=\".git/objects\"\n>>> '\n>>> +test_expect_success 'verify graph with no graph file' '\n>>> +\tcd \"$TRASH_DIRECTORY/full\" &&\n>>> +\tgit commit-graph verify\n>>> +'\n>>> +\n>>> test_expect_success 'write graph with no packs' '\n>>> \tcd \"$TRASH_DIRECTORY/full\" &&\n>>> \tgit commit-graph write --object-dir . &&\n>>> @@ -230,4 +235,9 @@ test_expect_success 'perform fast-forward merge in full repo' '\n>>> \ttest_cmp expect output\n>>> '\n>>> +test_expect_success 'git commit-graph verify' '\n>>> +\tcd \"$TRASH_DIRECTORY/full\" &&\n>>> +\tgit commit-graph verify >output\n>>> +'\n>> Those are tests with nearly the same code, but they are (by their\n>> descriptions) testing different things. This means that they rely on\n>> side effects of earlier tests.\n>>\n>> This is suboptimal, as it means that it would be impossible or very\n>> difficult to run individual tests (e.g. with GIT_SKIP_TESTS environment\n>> variable, or with an individual test suite --run option), unless you\n>> know which tests setup the repository state for later tests.\n>>\n>> It also means that running only failed tests with prove\n>> --state=failed,save or equivalently with\n>>\n>> $ make DEFAULT_TEST_TARGET=prove GIT_PROVE_OPTS='--state=failed,save' test\n>>\n>> wouldn't work correctly.\n>>\n>> As Johannes Schindelin (alias Dscho) said in latest Git Rev News\n>> interview: https://git.github.io/rev_news/2018/05/16/edition-39/\n>>\n>> JS> We have a test suite where debugging a regression may mean that you\n>> JS> have to run 98 test cases before the failing one every single time in\n>> JS> the edit/compile/debug cycle, because the 99th test case may depend on\n>> JS> a side effect of at least one of the preceding test cases. Git’s test\n>> JS> suite is so not [21st century best practices][1].\n>> JS>\n>> JS> [1]: https://www.slideshare.net/BuckHodges/lessons-learned-doing-devops-at-scale-at-microsoft\n>>\n>>\n>> I think can be solved quite efficiently by creating and using shell\n>> function, or two shell functions, which would either:\n>>\n>> * rename commit-graph file to some other temporary name if it exists,\n>> and move it back after the test.\n>> * create commit-graph file if it does not exist.\n>>\n>> For example (untested):\n>>\n>> prepare_no_commit_graph() {\n>> \tmv .git/info/commit-graph .git/info/commit-graph.away &&\n>> \ttest_when_finished \"mv .git/info/commit-graph.away .git/info/commit-graph\"\n>> }\n>>\n>> prepare_commit_graph() {\n>> \tif ! test -f \".git/info/commit-graph\"\n>> \tthen\n>> \t\tgit commit-graph write\n>> \tfi\n>> }\n>>\n>> Or something like that.\n>\n> Do we have a way to run individual steps of the test suite? I am\n> unfamiliar with that process.\n\nThe t/README describes three such ways in \"Skipping Tests\" section:\n\n- GIT_SKIP_TESTS environment variable, which can either can match the\n \"t[0-9]{4}\" part to skip the whole test, or t[0-9]{4} followed by\n \".$number\" to say which particular test to skip\n\n- For an individual test suite --run could be used to specify that\n only some tests should be run or that some tests should be\n excluded from a run (the latter with '!' prefix).\n\n- 'prove' harness can also run individual tests; one of more useful\n options is --state, which for example would allow to run only failed\n tests with --state=failed,save ... if the tests were independent.\n\n>\n> Adding the complexity of storing a copy of the commit-graph file for\n> re-use in a later test is wasted energy right now, because we need to\n> run the steps of the test that create the repo shape with the commits\n> laid out as set earlier in the test. This shape changes as we test\n> different states of the commit-graph (exists and contains all commits,\n> exists and doesn't contain all commits, etc.)\n\nI think we can solve most of the problem by separating validation tests\n(which all or almost all use the same commit-graph file) and other test;\nputting them in different test scripts. This means that the more\ncomplicated case would be limited to the subset of tests.\n\nAnyway, if the setup stages are clearly separated and clearly marked as\nsuch, we would be able to at least manually skip tests, or manually run\nonly a subset of tests.\n\nTest independence is certainly something nice to have, but as the git\ntestsuite is not in best shape wrt this, it is not a requirement.\n\nBest,\n-- \nJakub Narębski\n"},{"id":"349104","messageId":"866030rfd0.fsf@gmail.com","threadId":"48311","inReplyTo":"6d9d346c-b178-26f2-fdeb-bb0ffb070774@gmail.com","subject":"Re: [PATCH v3 07/20] commit-graph: verify catches corrupt signature","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-06-02T22:30:19Z","receivedAt":"2018-06-02T22:30:28Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n> On 5/28/2018 10:05 AM, Jakub Narebski wrote:\n>> Derrick Stolee writes:\n\n[...]\n>>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n>>> index 6ca451dfd2..bd64481c7a 100755\n>>> --- a/t/t5318-commit-graph.sh\n>>> +++ b/t/t5318-commit-graph.sh\n>>> @@ -235,9 +235,52 @@ test_expect_success 'perform fast-forward merge in full repo' '\n>>> \ttest_cmp expect output\n>>> '\n>>> +# the verify tests below expect the commit-graph to contain\n>>> +# exactly the commits reachable from the commits/8 branch.\n>>> +# If the file changes the set of commits in the list, then the\n>>> +# offsets into the binary file will result in different edits\n>>> +# and the tests will likely break.\n>>> +\n>>> test_expect_success 'git commit-graph verify' '\n>>> \tcd \"$TRASH_DIRECTORY/full\" &&\n>>> +\tgit rev-parse commits/8 | git commit-graph write --stdin-commits &&\n>>> \tgit commit-graph verify >output\n>>> '\n>> I don't quite understand what the change is meant to do.\n>\n> This gives us a constant commit-graph file to work with in the later tests.\n>\n> To get the \"independent test\" structure you want for the tests that\n> are coming, we need to do one of the following:\n>\n> 1. Write a new commit-graph file for every test (slows things down).\n\nOr check if correct graph-file exists, and if it doesn't only then write\na new commit-graph file (like I have proposed elsewhere in this thread).\n\nBarring this, I think it would be better if the preparation step was\nseparated into a 'setup ' step, so that one can easier select\nwhich tests to run, at least by hand.\n\n> 2. Do all corruption/verify checks in a single test (reduces the\n> information from a failed test, as it only reports the first failure).\n>\n> I don't like either of these options, so I went with this \"prepare\" step.\n\nThese are not the only possible options.\n\n>> Also, as I said earlier, I'd prefer if tests were as indpendent of each\n>> other as possible, to make running individual tests (e.g. only\n>> previously falling tests) easier.\n>>\n>> I especially do not like mixing running actual test with setting up the\n>> repository for future tests, as here.\n\n"},{"id":"349105","messageId":"86zi0cpzvh.fsf@gmail.com","threadId":"48311","inReplyTo":"d59f5723-7fb6-56d6-af2b-b8263cfdf354@gmail.com","subject":"Re: [PATCH v3 11/20] commit-graph: verify root tree OIDs","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-06-02T22:50:10Z","receivedAt":"2018-06-02T22:53:13Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n> On 5/30/2018 6:24 PM, Jakub Narebski wrote:\n\n[...]\n>> NOTE: we will be checking Commit Data chunk; I think it would be good\n>> idea to verify that size of Commit Data chunk matches (N * (H + 16) bytes)\n>> that format gives us, so that we don't accidentally red outside of\n>> memory if something got screwed up (like number of commits being wrong,\n>> or file truncated).\n>\n> This is actually how we calculate 'num_commits' during\n> load_commit_graph_one():\n>\n>     if (last_chunk_id == GRAPH_CHUNKID_OIDLOOKUP)\n>     {\n>         graph->num_commits = (chunk_offset - last_chunk_offset)\n>                              / graph->hash_len;\n>     }\n>\n> So, if the chunk doesn't match N*(H+16), we detect this because\n> FANOUT[255] != N.\n>\n> (There is one caveat here: (chunk_offset - last_chunk_offset) may not\n> be a multiple of hash_len, and \"accidentally\" truncate to N in the\n> division. I'll add more careful checks for this.)\n\nI have thought for some reason that number of commits N was stored\nsomewhere directly in the commit-graph header.\n\nAnyway we have three places that we can calculate (or simply read in\ncase of OID Fanour chunk) the number of commits:\n - FANOUT[255] == N\n - OID Lookup size = (N * H bytes)\n - N = (OID Lookup size) / hash_len\n - (OID Lookup size) % hash_len == 0\n - Commit Data size = (N * (H + 16) bytes)\n - N = (Commir Data size) / (hash_len + 16)\n - (Commir Data size) % (hash_len + 16) == 0\n\n>\n> We also check out-of-bounds offsets in that method.\n\nGood.\n\nBest,\n-- \nJakub Narębski\n"},{"id":"349227","messageId":"8d0165f3-6e0c-4799-6042-b0570b529b6e@gmail.com","threadId":"48311","inReplyTo":"86d0x8rimu.fsf@gmail.com","subject":"Re: [PATCH v3 06/20] commit-graph: add 'verify' subcommand","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-04T11:30:07Z","receivedAt":"2018-06-04T11:30:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/2/2018 5:19 PM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>> Do we have a way to run individual steps of the test suite? I am\n>> unfamiliar with that process.\n> The t/README describes three such ways in \"Skipping Tests\" section:\n>\n> - GIT_SKIP_TESTS environment variable, which can either can match the\n> \"t[0-9]{4}\" part to skip the whole test, or t[0-9]{4} followed by\n> \".$number\" to say which particular test to skip\n>\n> - For an individual test suite --run could be used to specify that\n> only some tests should be run or that some tests should be\n> excluded from a run (the latter with '!' prefix).\n>\n> - 'prove' harness can also run individual tests; one of more useful\n> options is --state, which for example would allow to run only failed\n> tests with --state=failed,save ... if the tests were independent.\n>\n>> Adding the complexity of storing a copy of the commit-graph file for\n>> re-use in a later test is wasted energy right now, because we need to\n>> run the steps of the test that create the repo shape with the commits\n>> laid out as set earlier in the test. This shape changes as we test\n>> different states of the commit-graph (exists and contains all commits,\n>> exists and doesn't contain all commits, etc.)\n> I think we can solve most of the problem by separating validation tests\n> (which all or almost all use the same commit-graph file) and other test;\n> putting them in different test scripts. This means that the more\n> complicated case would be limited to the subset of tests.\n>\n> Anyway, if the setup stages are clearly separated and clearly marked as\n> such, we would be able to at least manually skip tests, or manually run\n> only a subset of tests.\n>\n> Test independence is certainly something nice to have, but as the git\n> testsuite is not in best shape wrt this, it is not a requirement.\n\nI'm all for making the test suite better. In this case, I will hold that \nfor a later series (that is entirely focused on that feature) as I \nexpect we will want to discuss the correct pattern in detail.\n\nThanks,\n-Stolee\n"},{"id":"349228","messageId":"86d82fe4-3dcc-48a1-6800-70d49e30af8c@gmail.com","threadId":"48311","inReplyTo":"CACsJy8A8NofWCo0MbMxCe=xKTrnAqmM5D+FHSJXC-wO-DPnXwg@mail.gmail.com","subject":"Re: [PATCH v3 09/20] commit-graph: verify corrupt OID fanout and lookup","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-04T11:32:51Z","receivedAt":"2018-06-04T11:32:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/2/2018 12:38 AM, Duy Nguyen wrote:\n> On Thu, May 24, 2018 at 6:25 PM, Derrick Stolee wrote:\n>> + if (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n>> + graph_report(\"commit-graph has incorrect OID order: %s then %s\",\n>> + oid_to_hex(&prev_oid),\n>> + oid_to_hex(&cur_oid));\n> Should these strings be marked for translation with _()?\nI've been asking myself \"Is this message helpful to anyone other than a \nGit developer?\" and for this series the only one that is helpful to an \nend-user is the message about the final hash. If the hash is correct, \nbut these other messages appear, then there is a bug in the code that \nwrote the file. Otherwise, file corruption is more likely and the \ncorrect course of action is to delete and rebuild.\n\nThanks for being diligent in checking.\n\nThanks,\n-Stolee\n"},{"id":"349229","messageId":"51347e11-418b-5714-c6f0-668a9f7a6abf@gmail.com","threadId":"48311","inReplyTo":"868t7xv0km.fsf@gmail.com","subject":"Re: [PATCH v3 13/20] commit-graph: verify generation number","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-04T11:47:12Z","receivedAt":"2018-06-04T11:47:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/2/2018 8:23 AM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> While iterating through the commit parents, perform the generation\n>> number calculation and compare against the value stored in the\n>> commit-graph.\n> All right, that's good.\n>\n> What about commit-graph files that have GENERATION_NUMBER_ZERO for all\n> its commits (because we verify single commit-graph file only, there\n> wouldn't be GENERATION_NUMBER_ZERO mixed with non-zero generation\n> numbers)?\n>\n> Unless we can assume that no commit-graph file in the wild would have\n> GENERATION_NUMBER_ZERO.\n\nI was expecting that we would not have files in the wild with \nGENERATION_NUMBER_ZERO, but it looks like 2.18 will create files like \nthat. I'll put in logic to verify \"all are GENERATION_NUMBER_ZERO or all \nhave 'correct' generation number\".\n\n>\n>> The tests demonstrate that having a different set of parents affects\n>> the generation number calculation, and this value propagates to\n>> descendants. Hence, we drop the single-line condition on the output.\n> I don't understand what part of changes this paragraph of the commit\n> message refers to.\n>\n> Anyway, changing parents may not lead to changed generation numbers;\n> take for example commit with single parent, which we change to other\n> commit with the same generation number.\n\nThe tests introduced in the previous commit change the parent list, \nwhich then changes the generation number in some cases (the stored \ngeneration number doesn't match the generation number computed based on \nthe loaded parents). Since we report as many errors as possible (instead \nof failing on first error) those tests would fail if we say \"the _only_ \nerror should be the parent list\". (This comes up again when we report \nthe hash is incorrect, which would appear in every test.)\n\nThe test introduced in this commit only changes the generation number, \nso that test will have the one error.\n\n>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> commit-graph.c | 18 ++++++++++++++++++\n>> t/t5318-commit-graph.sh | 6 ++++++\n> Sidenote: I have just realized that it may be better to put\n> validation-related tests into different test file.\n>\n>> 2 files changed, 24 insertions(+)\n>>\n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index fff22dc0c3..ead92460c1 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> @@ -922,6 +922,7 @@ int verify_commit_graph(struct commit_graph *g)\n>> \tfor (i = 0; i < g->num_commits; i++) {\n>> \t\tstruct commit *graph_commit, *odb_commit;\n>> \t\tstruct commit_list *graph_parents, *odb_parents;\n>> +\t\tuint32_t max_generation = 0;\n>> \n>> \t\thashcpy(cur_oid.hash, g->chunk_oid_lookup + g->hash_len * i);\n>> \n>> @@ -956,6 +957,9 @@ int verify_commit_graph(struct commit_graph *g)\n>> \t\t\t\t\t oid_to_hex(&graph_parents->item->object.oid),\n>> \t\t\t\t\t oid_to_hex(&odb_parents->item->object.oid));\n>> \n>> +\t\t\tif (graph_parents->item->generation > max_generation)\n>> +\t\t\t\tmax_generation = graph_parents->item->generation;\n>> +\n> All right, that calculates the maximum of generation number of commit\n> parents.\n>\n>> \t\t\tgraph_parents = graph_parents->next;\n>> \t\t\todb_parents = odb_parents->next;\n>> \t\t}\n>> @@ -963,6 +967,20 @@ int verify_commit_graph(struct commit_graph *g)\n>> \t\tif (odb_parents != NULL)\n>> \t\t\tgraph_report(\"commit-graph parent list for commit %s terminates early\",\n>> \t\t\t\t oid_to_hex(&cur_oid));\n>> +\n>> +\t\t/*\n>> +\t\t * If one of our parents has generation GENERATION_NUMBER_MAX, then\n>> +\t\t * our generation is also GENERATION_NUMBER_MAX. Decrement to avoid\n>> +\t\t * extra logic in the following condition.\n>> +\t\t */\n> Nice trick.\n>\n>> +\t\tif (max_generation == GENERATION_NUMBER_MAX)\n>> +\t\t\tmax_generation--;\n> What about GENERATION_NUMBER_ZERO?\n>\n>> +\n>> +\t\tif (graph_commit->generation != max_generation + 1)\n>> +\t\t\tgraph_report(\"commit-graph generation for commit %s is %u != %u\",\n>> +\t\t\t\t oid_to_hex(&cur_oid),\n>> +\t\t\t\t graph_commit->generation,\n>> +\t\t\t\t max_generation + 1);\n> I think we should also check that generation numbers do not exceed\n> GENERATION_NUMBER_MAX... unless it is already taken care of?\n\nWe get that for free. First, the condition above would fail for at least \none commit. Second, we literally cannot store a value larger than \nGENERATION_NUMBER_MAX in the commit-graph as there are only 30 bits \ndedicated to the generation number.\n\n>\n>> \t}\n>> \n>> \treturn verify_commit_graph_error;\n>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n>> index 12f0d7f54d..673b0d37d5 100755\n>> --- a/t/t5318-commit-graph.sh\n>> +++ b/t/t5318-commit-graph.sh\n>> @@ -272,6 +272,7 @@ GRAPH_BYTE_COMMIT_TREE=$GRAPH_COMMIT_DATA_OFFSET\n>> GRAPH_BYTE_COMMIT_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN`\n>> GRAPH_BYTE_COMMIT_EXTRA_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4`\n>> GRAPH_BYTE_COMMIT_WRONG_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3`\n>> +GRAPH_BYTE_COMMIT_GENERATION=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 8`\n>> \n>> # usage: corrupt_graph_and_verify \n>> # Manipulates the commit-graph file at the position\n>> @@ -366,4 +367,9 @@ test_expect_success 'detect incorrect tree OID' '\n>> \t\t\"commit-graph parent for\"\n>> '\n>> \n>> +test_expect_success 'detect incorrect generation number' '\n>> +\tcorrupt_graph_and_verify $GRAPH_BYTE_COMMIT_GENERATION \"\\01\" \\\n> I assume that you have checked that it actually corrupts generation\n> number (without affecting commit date).\n>\n>> +\t\t\"generation\"\n> A very minor nitpick: Not \"generation for commit\"?\n>\n>> +'\n>> +\n>> test_done\n\n"},{"id":"349230","messageId":"8b7695b6-049a-de92-b4ee-70a73e5cf583@gmail.com","threadId":"48311","inReplyTo":"86muwdtcc7.fsf@gmail.com","subject":"Re: [PATCH v3 16/20] commit-graph: verify contents match checksum","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-04T11:55:36Z","receivedAt":"2018-06-04T11:55:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/2/2018 11:52 AM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> The commit-graph file ends with a SHA1 hash of the previous contents. If\n>> a commit-graph file has errors but the checksum hash is correct, then we\n>> know that the problem is a bug in Git and not simply file corruption\n>> after-the-fact.\n>>\n>> Compute the checksum right away so it is the first error that appears,\n>> and make the message translatable since this error can be \"corrected\" by\n>> a user by simply deleting the file and recomputing. The rest of the\n>> errors are useful only to developers.\n> Should we then provide --quiet / --verbose options, so that ordinary\n> user is not flooded with error messages meant for power users and Git\n> developers, then?\n>\n>> Be sure to continue checking the rest of the file data if the checksum\n>> is wrong. This is important for our tests, as we break the checksum as\n>> we modify bytes of the commit-graph file.\n> Well, we could have used sha1sum program, or test-sha1 helper to fix the\n> checksum after corrupting the commit-graph file...\n>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> commit-graph.c | 16 ++++++++++++++--\n>> t/t5318-commit-graph.sh | 6 ++++++\n>> 2 files changed, 20 insertions(+), 2 deletions(-)\n>>\n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index d2b291aca2..a33600c584 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> @@ -841,6 +841,7 @@ void write_commit_graph(const char *obj_dir,\n>> \toids.nr = 0;\n>> }\n>> \n>> +#define VERIFY_COMMIT_GRAPH_ERROR_HASH 2\n>> static int verify_commit_graph_error;\n>> \n>> static void graph_report(const char *fmt, ...)\n>> @@ -860,7 +861,9 @@ static void graph_report(const char *fmt, ...)\n>> int verify_commit_graph(struct commit_graph *g)\n>> {\n>> \tuint32_t i, cur_fanout_pos = 0;\n>> -\tstruct object_id prev_oid, cur_oid;\n>> +\tstruct object_id prev_oid, cur_oid, checksum;\n>> +\tstruct hashfile *f;\n>> +\tint devnull;\n>> \n>> \tif (!g) {\n>> \t\tgraph_report(\"no commit-graph file loaded\");\n>> @@ -879,6 +882,15 @@ int verify_commit_graph(struct commit_graph *g)\n>> \tif (verify_commit_graph_error)\n>> \t\treturn verify_commit_graph_error;\n>> \n>> +\tdevnull = open(\"/dev/null\", O_WRONLY);\n>> +\tf = hashfd(devnull, NULL);\n>> +\thashwrite(f, g->data, g->data_len - g->hash_len);\n>> +\tfinalize_hashfile(f, checksum.hash, CSUM_CLOSE);\n>> +\tif (hashcmp(checksum.hash, g->data + g->data_len - g->hash_len)) {\n>> +\t\tgraph_report(_(\"the commit-graph file has incorrect checksum and is likely corrupt\"));\n>> +\t\tverify_commit_graph_error = VERIFY_COMMIT_GRAPH_ERROR_HASH;\n>> +\t}\n> Is it the best way of calculating the SHA-1 checksum that out internal\n> APIs provide? Is it how SHA-1 checksum is calculated and checked for\n> packfiles?\nThis pattern is similar to hashfd_check() in csum-file.c, except we are \nhashing the file data directly instead of re-creating it from scratch \n(as is done for 'git index-pack --verify').\n\n>\n>> +\n>> \tfor (i = 0; i < g->num_commits; i++) {\n>> \t\tstruct commit *graph_commit;\n>> \n>> @@ -916,7 +928,7 @@ int verify_commit_graph(struct commit_graph *g)\n>> \t\tcur_fanout_pos++;\n>> \t}\n>> \n>> -\tif (verify_commit_graph_error)\n>> +\tif (verify_commit_graph_error & ~VERIFY_COMMIT_GRAPH_ERROR_HASH)\n>> \t\treturn verify_commit_graph_error;\n> So if we detected that checksum do not match, but we have not found an\n> error, we say that it is all right?\n\nThis only prevents us from stopping early. We will still report an error.\n\n>\n>> \n>> \tfor (i = 0; i < g->num_commits; i++) {\n>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n>> index 240aef6add..2680a2ebff 100755\n>> --- a/t/t5318-commit-graph.sh\n>> +++ b/t/t5318-commit-graph.sh\n>> @@ -279,6 +279,7 @@ GRAPH_COMMIT_DATA_WIDTH=`expr $HASH_LEN + 16`\n>> GRAPH_OCTOPUS_DATA_OFFSET=`expr $GRAPH_COMMIT_DATA_OFFSET + \\\n>> \t\t\t\t$GRAPH_COMMIT_DATA_WIDTH \\* $NUM_COMMITS`\n>> GRAPH_BYTE_OCTOPUS=`expr $GRAPH_OCTOPUS_DATA_OFFSET + 4`\n>> +GRAPH_BYTE_FOOTER=`expr $GRAPH_OCTOPUS_DATA_OFFSET + 4 \\* $NUM_OCTOPUS_EDGES`\n>> \n>> # usage: corrupt_graph_and_verify \n>> # Manipulates the commit-graph file at the position\n>> @@ -388,4 +389,9 @@ test_expect_success 'detect incorrect parent for octopus merge' '\n>> \t\t\"invalid parent\"\n>> '\n>> \n>> +test_expect_success 'detect invalid checksum hash' '\n>> +\tcorrupt_graph_and_verify $GRAPH_BYTE_FOOTER \"\\00\" \\\n>> +\t\t\"incorrect checksum\"\n> This would not work under GETTEXT_POISON, as the message is marked as\n> translatable, but corrupt_graph_and_verify uses 'grep' and not\n> 'test_i18grep' from t/test-lib-functions.sh\n\nI fixed this locally based on Szeder's comment.\n\n>\n>> +'\n> If it is pure checksum corruption, wouldn't this fail because it is not\n> a failure (exit code is 0)?\n\nIt is not zero, so the test passes.\n\n"},{"id":"349231","messageId":"c1fa14f3-f1cc-a91f-6192-e97e6437d0c4@gmail.com","threadId":"48311","inReplyTo":"867enhtb6n.fsf@gmail.com","subject":"Re: [PATCH v3 17/20] fsck: verify commit-graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-04T11:59:15Z","receivedAt":"2018-06-04T11:59:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/2/2018 12:17 PM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> If core.commitGraph is true, verify the contents of the commit-graph\n>> during 'git fsck' using the 'git commit-graph verify' subcommand. Run\n>> this check on all alternates, as well.\n> All right, so we have one config variable to control the use of\n> serialized commit-graph feaature. Nice.\n>\n>> We use a new process for two reasons:\n>>\n>> 1. The subcommand decouples the details of loading and verifying a\n>> commit-graph file from the other fsck details.\n> All right, I can agree with that.\n>\n> On the other hand using subcommand makes debugging harder, though not in\n> this case (well separated functionality that can be easily called with a\n> standalone command to be debugged).\n>\n>> 2. The commit-graph verification requires the commits to be loaded\n>> in a specific order to guarantee we parse from the commit-graph\n>> file for some objects and from the object database for others.\n> I don't quite understand this. Could you explain it in more detail?\n\nWe use `lookup_commit()` when verifying the commit-graph. If these \ncommits were loaded earlier in the process and parsed directly from the \nobject database, then we aren't comparing the commit-graph file contents \nagainst the ODB.\n\n>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> Documentation/git-fsck.txt | 3 +++\n>> builtin/fsck.c | 21 +++++++++++++++++++++\n>> t/t5318-commit-graph.sh | 8 ++++++++\n>> 3 files changed, 32 insertions(+)\n>>\n>> diff --git a/Documentation/git-fsck.txt b/Documentation/git-fsck.txt\n>> index b9f060e3b2..ab9a93fb9b 100644\n>> --- a/Documentation/git-fsck.txt\n>> +++ b/Documentation/git-fsck.txt\n>> @@ -110,6 +110,9 @@ Any corrupt objects you will have to find in backups or other archives\n>> (i.e., you can just remove them and do an 'rsync' with some other site in\n>> the hopes that somebody else has the object you have corrupted).\n>> \n>> +If core.commitGraph is true, the commit-graph file will also be inspected\n> Shouldn't we use `core.commitGraph` here?\n>\n>> +using 'git commit-graph verify'. See linkgit:git-commit-graph[1].\n>> +\n>> Extracted Diagnostics\n>> ---------------------\n>> \n>> diff --git a/builtin/fsck.c b/builtin/fsck.c\n>> index ef78c6c00c..a6d5045b77 100644\n>> --- a/builtin/fsck.c\n>> +++ b/builtin/fsck.c\n>> @@ -16,6 +16,7 @@\n>> #include \"streaming.h\"\n>> #include \"decorate.h\"\n>> #include \"packfile.h\"\n>> +#include \"run-command.h\"\n>> \n>> #define REACHABLE 0x0001\n>> #define SEEN 0x0002\n>> @@ -45,6 +46,7 @@ static int name_objects;\n>> #define ERROR_REACHABLE 02\n>> #define ERROR_PACK 04\n>> #define ERROR_REFS 010\n>> +#define ERROR_COMMIT_GRAPH 020\n> Minor nitpick and a sidenote: I wonder if it wouldn't be better to\n> either use hexadecimal constants, or use (1 << n) for all ERROR_*\n> preprocesor constants.\n>\n>> \n>> static const char *describe_object(struct object *obj)\n>> {\n>> @@ -815,5 +817,24 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n>> \t}\n>> \n>> \tcheck_connectivity();\n>> +\n>> +\tif (core_commit_graph) {\n>> +\t\tstruct child_process commit_graph_verify = CHILD_PROCESS_INIT;\n>> +\t\tconst char *verify_argv[] = { \"commit-graph\", \"verify\", NULL, NULL, NULL, NULL };\n> I see that NULL at index 2 and 3 (at 3rd and 4th place) are here for\n> \"--object-dir\" and , the last one is\n> terminator for that case, but what is next to last NULL (at 5th place)\n> for?\n>\n>> +\t\tcommit_graph_verify.argv = verify_argv;\n>> +\t\tcommit_graph_verify.git_cmd = 1;\n>> +\n>> +\t\tif (run_command(&commit_graph_verify))\n>> +\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n>> +\n>> +\t\tprepare_alt_odb();\n>> +\t\tfor (alt = alt_odb_list; alt; alt = alt->next) {\n>> +\t\t\tverify_argv[2] = \"--object-dir\";\n>> +\t\t\tverify_argv[3] = alt->path;\n>> +\t\t\tif (run_command(&commit_graph_verify))\n>> +\t\t\t\terrors_found |= ERROR_COMMIT_GRAPH;\n>> +\t\t}\n>> +\t}\n> For performance reasons it may be better to start those 'git\n> commit-graph verify' commands asynchronously earlier, so that they can\n> run in parallel / concurrently wth other checks, and wait for them and\n> get their error code at the end of git-fsck run.\n>\n> But that is probably better left for a separate commit.\n>\n>> +\n>> \treturn errors_found;\n>> }\n>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n>> index 2680a2ebff..4941937163 100755\n>> --- a/t/t5318-commit-graph.sh\n>> +++ b/t/t5318-commit-graph.sh\n>> @@ -394,4 +394,12 @@ test_expect_success 'detect invalid checksum hash' '\n>> \t\t\"incorrect checksum\"\n>> '\n>> \n>> +test_expect_success 'git fsck (checks commit-graph)' '\n>> +\tcd \"$TRASH_DIRECTORY/full\" &&\n>> +\tgit fsck &&\n>> +\tcorrupt_graph_and_verify $GRAPH_BYTE_FOOTER \"\\00\" \\\n>> +\t\t\"incorrect checksum\" &&\n>> +\ttest_must_fail git fsck\n>> +'\n> All right; though the same caveats apply as with previous commit in\n> series. Perhaps it would be better to truncate commit-graph file, or\n> corrupt it in some 'random' place.\n>\n>> +\n>> test_done\n> Best,\n\n"},{"id":"349236","messageId":"4dccf288-ccf1-6819-4112-00432db1ddc6@gmail.com","threadId":"48311","inReplyTo":"86k1rhrt1u.fsf@gmail.com","subject":"Re: [PATCH v3 18/20] commit-graph: add '--reachable' option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-04T12:44:19Z","receivedAt":"2018-06-04T12:44:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/2/2018 1:34 PM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> When writing commit-graph files, it can be convenient to ask for all\n>> reachable commits (starting at the ref set) in the resulting file. This\n>> is particularly helpful when writing to stdin is complicated, such as a\n>> future integration with 'git gc' which will call\n>> write_commit_graph_reachable() after performing cleanup of the object\n>> database.\n> Nice.\n>\n> The last sentence of the commit message is a bit long, though, in my\n> opinion.\n>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> Documentation/git-commit-graph.txt | 8 ++++++--\n>> builtin/commit-graph.c | 16 ++++++++++++----\n>> commit-graph.c | 32 ++++++++++++++++++++++++++++++++\n>> commit-graph.h | 1 +\n>> t/t5318-commit-graph.sh | 10 ++++++++++\n>> 5 files changed, 61 insertions(+), 6 deletions(-)\n>>\n>> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n>> index a222cfab08..dececb79d7 100644\n>> --- a/Documentation/git-commit-graph.txt\n>> +++ b/Documentation/git-commit-graph.txt\n>> @@ -38,12 +38,16 @@ Write a commit graph file based on the commits found in packfiles.\n>> +\n>> With the `--stdin-packs` option, generate the new commit graph by\n>> walking objects only in the specified pack-indexes. (Cannot be combined\n>> -with --stdin-commits.)\n>> +with `--stdin-commits` or `--reachable`.)\n>> +\n>> With the `--stdin-commits` option, generate the new commit graph by\n>> walking commits starting at the commits specified in stdin as a list\n>> of OIDs in hex, one OID per line. (Cannot be combined with\n>> ---stdin-packs.)\n>> +`--stdin-packs` or `--reachable`.)\n>> ++\n>> +With the `--reachable` option, generate the new commit graph by walking\n>> +commits starting at all refs. (Cannot be combined with `--stdin-commits`\n>> +or `--stdin-packs`.)\n> All right (though I wonder a bit about the restriction).\n>\n> I think it might be a good idea to describe all of this in the usage\n> string for the 'git commit-graph write', instead of using ''\n> placeholder, that is instead of current:\n>\n> 'git commit-graph write' [--object-dir ]\n>\n> use\n>\n> 'git commit-graph write' [--stdin-commits | --stdin-packs | --reachable]\n> [--append] [--object-dir ]\n>\n> or something like that.\n>\n>> +\n>> With the `--append` option, include all commits that are present in the\n>> existing commit-graph file.\n>> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n>> index 0433dd6e20..20ce6437ae 100644\n>> --- a/builtin/commit-graph.c\n>> +++ b/builtin/commit-graph.c\n>> @@ -9,7 +9,7 @@ static char const * const builtin_commit_graph_usage[] = {\n>> \tN_(\"git commit-graph [--object-dir ]\"),\n>> \tN_(\"git commit-graph read [--object-dir ]\"),\n>> \tN_(\"git commit-graph verify [--object-dir ]\"),\n>> -\tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n>> +\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n> All right, very straightforward. I guess they are put in [almost]\n> alphabetical order, or is there some other reasoning behind the ordering\n> used (which is different from the one in the manpage)?\n>\n>> \tNULL\n>> };\n>> \n>> @@ -24,12 +24,13 @@ static const char * const builtin_commit_graph_read_usage[] = {\n>> };\n>> \n>> static const char * const builtin_commit_graph_write_usage[] = {\n>> -\tN_(\"git commit-graph write [--object-dir ] [--append] [--stdin-packs|--stdin-commits]\"),\n>> +\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n> The same.\n>\n>> \tNULL\n>> };\n>> \n>> static struct opts_commit_graph {\n>> \tconst char *obj_dir;\n>> +\tint reachable;\n>> \tint stdin_packs;\n>> \tint stdin_commits;\n>> \tint append;\n>> @@ -130,6 +131,8 @@ static int graph_write(int argc, const char **argv)\n>> \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n>> \t\t\tN_(\"dir\"),\n>> \t\t\tN_(\"The object directory to store the graph\")),\n>> +\t\tOPT_BOOL(0, \"reachable\", &opts.reachable,\n>> +\t\t\tN_(\"start walk at all refs\")),\n> Errr... does '--no-reachable' makes sense? Because if I am right\n> currently it is supported, isn't it.\n\nTrue, but the same holds for many arguments in the codebase. Just \nlooking at 'git apply' as a sample, it uses OPT_BOOL for \"numstat\", \n\"summary\", \"check\", \"index\", \"cached\", \"apply\", and \"no-add\" (so \n\"--no-no-add\" works?). I think accepting \"--no-reachable\" is fine, since \nit will set opts.reachable = 0 and we will not use the reachable option.\n\n>\n>> \t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n>> \t\t\tN_(\"scan pack-indexes listed by stdin for commits\")),\n>> \t\tOPT_BOOL(0, \"stdin-commits\", &opts.stdin_commits,\n>> @@ -143,11 +146,16 @@ static int graph_write(int argc, const char **argv)\n>> \t\t\t builtin_commit_graph_write_options,\n>> \t\t\t builtin_commit_graph_write_usage, 0);\n>> \n>> -\tif (opts.stdin_packs && opts.stdin_commits)\n>> -\t\tdie(_(\"cannot use both --stdin-commits and --stdin-packs\"));\n>> +\tif (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)\n> Nice trick.\n>\n>> +\t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\n> It is a pity that parseopt API does not have direct support for mutually\n> exclusive groups of boolean options, like ArgumentParser.add_mutually_exclusive_group()\n> in Python's argparse.\n>\n> Still, you need to use what it is there.\n>\n>> \tif (!opts.obj_dir)\n>> \t\topts.obj_dir = get_object_directory();\n>> \n>> +\tif (opts.reachable) {\n>> +\t\twrite_commit_graph_reachable(opts.obj_dir, opts.append);\n>> +\t\treturn 0;\n>> +\t}\n> Just using the option.\n>\n>> +\n>> \tif (opts.stdin_packs || opts.stdin_commits) {\n>> \t\tstruct strbuf buf = STRBUF_INIT;\n>> \t\tlines_nr = 0;\n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index a33600c584..057d734926 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> @@ -6,6 +6,7 @@\n>> #include \"packfile.h\"\n>> #include \"commit.h\"\n>> #include \"object.h\"\n>> +#include \"refs.h\"\n>> #include \"revision.h\"\n>> #include \"sha1-lookup.h\"\n>> #include \"commit-graph.h\"\n>> @@ -651,6 +652,37 @@ static void compute_generation_numbers(struct packed_commit_list* commits)\n>> \t}\n>> }\n>> \n>> +struct hex_list {\n>> +\tchar **hex_strs;\n>> +\tint hex_nr;\n>> +\tint hex_alloc;\n>> +};\n> Is this what git-for-each-ref / git-branch / git-tag uses?\n>\n> Would it be possible to use for example string-list API (documented in\n> string-list.h) instead? Anyway, it looks like the use of allocation\n> growing API is simple enough... though perhaps it could be made simpler\n> by noticing that all strings have the same width.\n\nI should use the string-list API for more than just this list of \nstrings. I'll add a new commit that replaces the `const char **` \nparameters from write_commit_graph(), then this commit becomes much simpler.\n\n>\n>> +\n>> +static int add_ref_to_list(const char *refname,\n>> +\t\t\t const struct object_id *oid,\n>> +\t\t\t int flags, void *cb_data)\n>> +{\n>> +\tstruct hex_list *list = (struct hex_list*)cb_data;\n>> +\n>> +\tALLOC_GROW(list->hex_strs, list->hex_nr + 1, list->hex_alloc);\n>> +\tlist->hex_strs[list->hex_nr] = xcalloc(GIT_MAX_HEXSZ + 1, 1);\n>> +\tstrcpy(list->hex_strs[list->hex_nr], oid_to_hex(oid));\n> Wouldn't it be better to use strdup or xstrdup instead of\n> xcalloc+strcpy?\n>\n>> +\tlist->hex_nr++;\n>> +\treturn 0;\n>> +}\n>> +\n>> +void write_commit_graph_reachable(const char *obj_dir, int append)\n>> +{\n>> +\tstruct hex_list list;\n>> +\tlist.hex_nr = 0;\n>> +\tlist.hex_alloc = 128;\n>> +\tALLOC_ARRAY(list.hex_strs, list.hex_alloc);\n>> +\n>> +\tfor_each_ref(add_ref_to_list, &list);\n>> +\n>> +\twrite_commit_graph(obj_dir, NULL, 0, (const char **)list.hex_strs, list.hex_nr, append);\n> Where do we free the allocated data and allocated strings? If they are\n> cleaned by process exit, perhaps they need to be UNLEAK-ed?\n>\n>> +}\n>> +\n>> void write_commit_graph(const char *obj_dir,\n>> \t\t\tconst char **pack_indexes,\n>> \t\t\tint nr_packs,\n>> diff --git a/commit-graph.h b/commit-graph.h\n>> index 71a39c5a57..9a06a5f188 100644\n>> --- a/commit-graph.h\n>> +++ b/commit-graph.h\n>> @@ -46,6 +46,7 @@ struct commit_graph {\n>> \n>> struct commit_graph *load_commit_graph_one(const char *graph_file);\n>> \n>> +void write_commit_graph_reachable(const char *obj_dir, int append);\n>> void write_commit_graph(const char *obj_dir,\n>> \t\t\tconst char **pack_indexes,\n>> \t\t\tint nr_packs,\n>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n>> index 4941937163..a659620332 100755\n>> --- a/t/t5318-commit-graph.sh\n>> +++ b/t/t5318-commit-graph.sh\n>> @@ -205,6 +205,16 @@ test_expect_success 'build graph from commits with append' '\n>> graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n>> graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n>> \n>> +test_expect_success 'build graph using --reachable' '\n>> +\tcd \"$TRASH_DIRECTORY/full\" &&\n>> +\tgit commit-graph write --reachable &&\n>> +\ttest_path_is_file $objdir/info/commit-graph &&\n>> +\tgraph_read_expect \"11\" \"large_edges\"\n>> +'\n> All right, here we check that commit-graph has expected features (11\n> commits, and large_edges optional chunk).\n>\n> Perhaps we could also check that different equivalent ways of creating\n> serialized commit graph file produce byte-for-byte identical file?\n>\n>> +\n>> +graph_git_behavior 'append graph, commit 8 vs merge 1' full commits/8 merge/1\n>> +graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n> All right, this supposedly tests that behavior does not change whether\n> we are using or we are not using the commit-graph feature... but I have\n> just noticed that raph_git_two_modes() uses `git -c core.graph=true`;\n> shouldn't it be `git -c core.commitGraph=true`?\n\nWOW good catch. I've sent a patch to fix that immediately, as these \ntests are not doing what they should (verify the core.commitGraph \nsetting does not change the output of the walk).\n\n>\n>> +\n>> test_expect_success 'setup bare repo' '\n>> \tcd \"$TRASH_DIRECTORY\" &&\n>> \tgit clone --bare --no-local full bare &&\n\n"},{"id":"349237","messageId":"fc21c512-e6af-58c6-97e9-17e147b946bc@gmail.com","threadId":"48311","inReplyTo":"866031rrq1.fsf@gmail.com","subject":"Re: [PATCH v3 19/20] gc: automatically write commit-graph files","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-04T12:51:58Z","receivedAt":"2018-06-04T12:52:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/2/2018 2:03 PM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> The commit-graph file is a very helpful feature for speeding up git\n>> operations. In order to make it more useful, write the commit-graph file\n>> by default during standard garbage collection operations.\n> I think you meant here \"make it possible to write the commit-graph file\n> during standard garbage collection operations.\" (i.e. add \"make it\n> possible\" because it hides behind new config option, and remove \"by\n> default\" because currently it is not turned on by default).\n>\n>> Add a 'gc.commitGraph' config setting that triggers writing a\n>> commit-graph file after any non-trivial 'git gc' command. Defaults to\n>> false while the commit-graph feature matures. We specifically do not\n>> want to turn this on by default until the commit-graph feature is fully\n> s/turn this on/have this on/ I think.\n>\n>> integrated with history-modifying features like shallow clones.\n> Two things.\n>\n> First, shallow clones, replacement mechanims (git-replace) and grafts\n> are not \"history-modifying\" features; this name is in my opinion\n> reserved for history-rewriting features such as interactive rebase, the\n> `git filter-branch` feature or out-of-tree BFG Repo Cleaner or\n> reposurgeon tools. They alter the _view_ of history; they should be\n> IMVHO named \"history-view-altering\" features -- though I agree this is\n> mouthful.\n>\n> Second, shouldn't we, as Martin Ågren said, warn about the issue in the\n> manpage for git-commit-graph?\n>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> Documentation/config.txt | 6 ++++++\n>> Documentation/git-gc.txt | 4 ++++\n>> builtin/gc.c | 6 ++++++\n>> t/t5318-commit-graph.sh | 14 ++++++++++++++\n>> 4 files changed, 30 insertions(+)\n>>\n>> diff --git a/Documentation/config.txt b/Documentation/config.txt\n>> index 11f027194e..9a3abd87e7 100644\n>> --- a/Documentation/config.txt\n>> +++ b/Documentation/config.txt\n>> @@ -1553,6 +1553,12 @@ gc.autoDetach::\n>> \tMake `git gc --auto` return immediately and run in background\n>> \tif the system supports it. Default is true.\n>> \n>> +gc.commitGraph::\n>> +\tIf true, then gc will rewrite the commit-graph file after any\n>> +\tchange to the object database. If '--auto' is used, then the\n>> +\tcommit-graph will not be updated unless the threshold is met.\n> What threshold? Ah, thresholds defined for `git gc --auto` (gc.auto,\n> gc.autoPackLimit, gc.logExpiry,...).\n>\n>> +\tSee linkgit:git-commit-graph[1] for details.\n> You missed declaring the default value for this config option.\n>\n>> +\n>> gc.logExpiry::\n>> \tIf the file gc.log exists, then `git gc --auto` won't run\n>> \tunless that file is more than 'gc.logExpiry' old. Default is\n>> diff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\n>> index 571b5a7e3c..17dd654a59 100644\n>> --- a/Documentation/git-gc.txt\n>> +++ b/Documentation/git-gc.txt\n>> @@ -119,6 +119,10 @@ The optional configuration variable `gc.packRefs` determines if\n>> it within all non-bare repos or it can be set to a boolean value.\n>> This defaults to true.\n>> \n>> +The optional configuration variable 'gc.commitGraph' determines if\n>> +'git gc' runs 'git commit-graph write'. This can be set to a boolean\n> Should it be \"runs\" or \"should run\"?\n>\n>> +value. This defaults to false.\n> Should it be '...' or `...`? Below we have `gc.aggresiveWindow`, above\n> we have 'gc.commitGraph', for example.\n>\n>> +\n>> The optional configuration variable `gc.aggressiveWindow` controls how\n>> much time is spent optimizing the delta compression of the objects in\n>> the repository when the --aggressive option is specified. The larger\n>> diff --git a/builtin/gc.c b/builtin/gc.c\n>> index 77fa720bd0..efd214a59f 100644\n>> --- a/builtin/gc.c\n>> +++ b/builtin/gc.c\n>> @@ -20,6 +20,7 @@\n>> #include \"argv-array.h\"\n>> #include \"commit.h\"\n>> #include \"packfile.h\"\n>> +#include \"commit-graph.h\"\n>> \n>> #define FAILED_RUN \"failed to run %s\"\n>> \n>> @@ -34,6 +35,7 @@ static int aggressive_depth = 50;\n>> static int aggressive_window = 250;\n>> static int gc_auto_threshold = 6700;\n>> static int gc_auto_pack_limit = 50;\n>> +static int gc_commit_graph = 0;\n>> static int detach_auto = 1;\n>> static timestamp_t gc_log_expire_time;\n>> static const char *gc_log_expire = \"1.day.ago\";\n>> @@ -121,6 +123,7 @@ static void gc_config(void)\n>> \tgit_config_get_int(\"gc.aggressivedepth\", &aggressive_depth);\n>> \tgit_config_get_int(\"gc.auto\", &gc_auto_threshold);\n>> \tgit_config_get_int(\"gc.autopacklimit\", &gc_auto_pack_limit);\n>> +\tgit_config_get_bool(\"gc.commitgraph\", &gc_commit_graph);\n>> \tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n>> \tgit_config_get_expiry(\"gc.pruneexpire\", &prune_expire);\n>> \tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n>> @@ -480,6 +483,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n>> \tif (pack_garbage.nr > 0)\n>> \t\tclean_pack_garbage();\n>> \n>> +\tif (gc_commit_graph)\n>> +\t\twrite_commit_graph_reachable(get_object_directory(), 0);\n>> +\n> Nice.\n>\n> Though now I wonder when appending should be used...\n\nAppending is probably useless in the 'reachable' case, but is valuable \nin the '--stdin-packs' case (which is what we use in GVFS to maintain \nthe commit-graph).\n\n>\n>> \tif (auto_gc && too_many_loose_objects())\n>> \t\twarning(_(\"There are too many unreachable loose objects; \"\n>> \t\t\t\"run 'git prune' to remove them.\"));\n>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n>> index a659620332..d20b17586f 100755\n>> --- a/t/t5318-commit-graph.sh\n>> +++ b/t/t5318-commit-graph.sh\n>> @@ -245,6 +245,20 @@ test_expect_success 'perform fast-forward merge in full repo' '\n>> \ttest_cmp expect output\n>> '\n>> \n>> +test_expect_success 'check that gc clears commit-graph' '\n> I wouldn't use the word \"clears\" here...\n>\n>> +\tcd \"$TRASH_DIRECTORY/full\" &&\n>> +\tgit commit --allow-empty -m \"blank\" &&\n>> +\tgit commit-graph write --reachable &&\n>> +\tcp $objdir/info/commit-graph commit-graph-before-gc &&\n>> +\tgit reset --hard HEAD~1 &&\n>> +\tgit config gc.commitGraph true &&\n>> +\tgit gc &&\n>> +\tcp $objdir/info/commit-graph commit-graph-after-gc &&\n>> +\t! test_cmp commit-graph-before-gc commit-graph-after-gc &&\n>> +\tgit commit-graph write --reachable &&\n>> +\ttest_cmp commit-graph-after-gc $objdir/info/commit-graph\n>> +'\n> ...but otherwise, nice test: it checks that git-gc after rewriting\n> history changes commit-graph file, and that the changed file is what we\n> expect it to be (note: here we compare commit-graph files directly, and\n> not just check the features via 'git commit-graph read').\n>\n>> +\n>> # the verify tests below expect the commit-graph to contain\n>> # exactly the commits reachable from the commits/8 branch.\n>> # If the file changes the set of commits in the list, then the\n\n"},{"id":"349238","messageId":"7665886b-1343-ca9d-aa41-db7a5b7d2735@gmail.com","threadId":"48311","inReplyTo":"86tvqltla9.fsf@gmail.com","subject":"Re: [PATCH v3 15/20] commit-graph: test for corrupted octopus edge","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-06-04T13:08:59Z","receivedAt":"2018-06-04T13:09:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/2/2018 8:39 AM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> The commit-graph file has an extra chunk to store the parent int-ids for\n>> parents beyond the first parent for octopus merges. Our test repo has a\n>> single octopus merge that we can manipulate to demonstrate the 'verify'\n>> subcommand detects incorrect values in that chunk.\n> If I understand it correctly the above means that our _reading_ code\n> checks for validity (which then 'git commit-graph verify' uses), just\n> there were not any tests for that.\n>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> t/t5318-commit-graph.sh | 10 ++++++++++\n>> 1 file changed, 10 insertions(+)\n>>\n>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n>> index 58adb8246d..240aef6add 100755\n>> --- a/t/t5318-commit-graph.sh\n>> +++ b/t/t5318-commit-graph.sh\n>> @@ -248,6 +248,7 @@ test_expect_success 'git commit-graph verify' '\n>> '\n>> \n>> NUM_COMMITS=9\n>> +NUM_OCTOPUS_EDGES=2\n>> HASH_LEN=20\n>> GRAPH_BYTE_VERSION=4\n>> GRAPH_BYTE_HASH=5\n>> @@ -274,6 +275,10 @@ GRAPH_BYTE_COMMIT_EXTRA_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 4`\n>> GRAPH_BYTE_COMMIT_WRONG_PARENT=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 3`\n>> GRAPH_BYTE_COMMIT_GENERATION=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 8`\n>> GRAPH_BYTE_COMMIT_DATE=`expr $GRAPH_COMMIT_DATA_OFFSET + $HASH_LEN + 12`\n>> +GRAPH_COMMIT_DATA_WIDTH=`expr $HASH_LEN + 16`\n>> +GRAPH_OCTOPUS_DATA_OFFSET=`expr $GRAPH_COMMIT_DATA_OFFSET + \\\n>> +\t\t\t\t$GRAPH_COMMIT_DATA_WIDTH \\* $NUM_COMMITS`\n>> +GRAPH_BYTE_OCTOPUS=`expr $GRAPH_OCTOPUS_DATA_OFFSET + 4`\n>> \n>> # usage: corrupt_graph_and_verify \n>> # Manipulates the commit-graph file at the position\n>> @@ -378,4 +383,9 @@ test_expect_success 'detect incorrect commit date' '\n>> \t\t\"commit date\"\n>> '\n>> \n>> +test_expect_success 'detect incorrect parent for octopus merge' '\n>> +\tcorrupt_graph_and_verify $GRAPH_BYTE_OCTOPUS \"\\01\" \\\n>> +\t\t\"invalid parent\"\n>> +'\n> So we change the int-id to non-existing commit, and check that\n> commit-graph code checks for that.\n>\n> What about the case when there are octopus merges, but no EDGE chunk\n> (which I think we can emulate by changing / corrupting number of\n> chunks)?\n>\n> What about the case where int-id of edge in EDGE chunk is correct, that\n> is points to a valid commit, but does not agree with what is in the\n> object database (what parents octopus merge has in reality)?\n>\n> Do we detect the situation where the second parent value in the commit\n> data stores an array position within a Large Edge chunk, but we do not\n> reach a value with the most-significant bit on when reaching the end of\n> Large Edge chunk?\n\nThere are a few holes like this, but I think they are better suited to a \nfollow-up series, as this series is already quite large.\n\n-Stolee\n"},{"id":"349249","messageId":"CACsJy8CGqrAH0A8TvSju8MqJihn6vym8PrLvtuf5bbczj=kUwQ@mail.gmail.com","threadId":"48311","inReplyTo":"86d82fe4-3dcc-48a1-6800-70d49e30af8c@gmail.com","subject":"Re: [PATCH v3 09/20] commit-graph: verify corrupt OID fanout and lookup","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-06-04T14:42:09Z","receivedAt":"2018-06-04T14:42:43Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Jun 4, 2018 at 1:32 PM, Derrick Stolee wrote:\n> On 6/2/2018 12:38 AM, Duy Nguyen wrote:\n>>\n>> On Thu, May 24, 2018 at 6:25 PM, Derrick Stolee \n>> wrote:\n>>>\n>>> + if (i && oidcmp(&prev_oid, &cur_oid) >= 0)\n>>> + graph_report(\"commit-graph has incorrect OID\n>>> order: %s then %s\",\n>>> + oid_to_hex(&prev_oid),\n>>> + oid_to_hex(&cur_oid));\n>>\n>> Should these strings be marked for translation with _()?\n>\n> I've been asking myself \"Is this message helpful to anyone other than a Git\n> developer?\" and for this series the only one that is helpful to an end-user\n> is the message about the final hash. If the hash is correct, but these other\n> messages appear, then there is a bug in the code that wrote the file.\n> Otherwise, file corruption is more likely and the correct course of action\n> is to delete and rebuild.\n\nDev-only strings like this are typically prefixed with \"BUG:\" or\n\"internal error:\" (unless BUG() is a better choice). Git is\nunfortunately not fully i18n-ized and devs from time to time still\nforget to mark string for translations when appropriate, including me.\nBecause of this, we still have to slowly scan through the code base\nand mark more strings for translation. Something to say clearly \"not\ntranslatable on purpose\" would help a lot. If \"BUG:\" and friends are\ntoo much noise, a /* no translate */ comment or some other form could\nalso help.\n\nBut your explanation to me still sounds like corrupted file in some\nform, which should be translated unless it's too cryptic. commit-graph\nformat may be available in non-English languages and people can still\ntry to figure out the problem without relying entirely on git\ndevelopers.\n-- \nDuy\n"}]}