{"thread":{"id":"50302","subject":"[PATCH 0/6] Create commit-graph file format v2","startedAt":"2019-01-23T21:59:18Z","lastAt":"2019-07-01T12:19:03Z","messageCount":89,"participants":["Derrick Stolee via GitGitGadget","Jonathan Tan","Ævar Arnfjörð Bjarmason","Derrick Stolee","Junio C Hamano","SZEDER Gábor","Christian Couder","Josh Steadmon"],"isPatch":true,"patchVersion":1,"patchTotal":6},"messages":[{"id":"367484","messageId":"pull.112.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":null,"subject":"[PATCH 0/6] Create commit-graph file format v2","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T21:59:14Z","receivedAt":"2019-01-23T21:59:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file format has some shortcomings that were discussed\non-list:\n\n 1. It doesn't use the 4-byte format ID from the_hash_algo.\n \n \n 2. There is no way to change the reachability index from generation numbers\n to corrected commit date [1].\n \n \n 3. The unused byte in the format could be used to signal the file is\n incremental, but current clients ignore the value even if it is\n non-zero.\n \n \n\nThis series adds a new version (2) to the commit-graph file. The fifth byte\nalready specified the file format, so existing clients will gracefully\nrespond to files with a different version number. The only real change now\nis that the header takes 12 bytes instead of 8, due to using the 4-byte\nformat ID for the hash algorithm.\n\nThe new bytes reserved for the reachability index version and incremental\nfile formats are now expected to be equal to the defaults. When we update\nthese values to be flexible in the future, if a client understands\ncommit-graph v2 but not those new values, then it will fail gracefully.\n\nThis series is based on ab/commit-graph-write-progress and bc/sha-256.\n\nThanks, -Stolee\n\n[1] \nhttps://public-inbox.org/git/6367e30a-1b3a-4fe9-611b-d931f51effef@gmail.com/\n\nDerrick Stolee (6):\n commit-graph: return with errors during write\n commit-graph: collapse parameters into flags\n commit-graph: create new version flags\n commit-graph: add --version= option\n commit-graph: implement file format version 2\n commit-graph: test verifying a corrupt v2 header\n\n Documentation/git-commit-graph.txt | 3 +\n .../technical/commit-graph-format.txt | 26 ++-\n builtin/commit-graph.c | 43 +++--\n builtin/commit.c | 5 +-\n builtin/gc.c | 7 +-\n commit-graph.c | 158 +++++++++++++-----\n commit-graph.h | 16 +-\n t/t5318-commit-graph.sh | 42 ++++-\n 8 files changed, 233 insertions(+), 67 deletions(-)\n\n\nbase-commit: 91b3ce35eeb93be1f4406e25ccdc4ab983a8e5af\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-112%2Fderrickstolee%2Fgraph%2Fv2-head-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-112/derrickstolee/graph/v2-head-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/112\n-- \ngitgitgadget\n"},{"id":"367485","messageId":"e72498d0c5dbc32fd04ecef611fe196ef815a2d7.1548280753.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.git.gitgitgadget@gmail.com","subject":"[PATCH 1/6] commit-graph: return with errors during write","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T21:59:15Z","receivedAt":"2019-01-23T21:59:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method uses die() to report failure and\nexit when confronted with an unexpected condition. This use of\ndie() in a library function is incorrect and is now replaced by\nerror() statements and an int return type.\n\nNow that we use 'goto cleanup' to jump to the terminal condition\non an error, we have new paths that could lead to uninitialized\nvalues. New initializers are added to correct for this.\n\nThe builtins 'commit-graph', 'gc', and 'commit' call these methods,\nso update them to check the return value.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 19 +++++++------\n builtin/commit.c | 5 ++--\n builtin/gc.c | 7 ++---\n commit-graph.c | 60 +++++++++++++++++++++++++++++-------------\n commit-graph.h | 10 +++----\n 5 files changed, 62 insertions(+), 39 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 4ae502754c..b12d46fdc8 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -126,6 +126,7 @@ static int graph_write(int argc, const char **argv)\n \tstruct string_list *pack_indexes = NULL;\n \tstruct string_list *commit_hex = NULL;\n \tstruct string_list lines;\n+\tint result;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n@@ -153,10 +154,8 @@ static int graph_write(int argc, const char **argv)\n \n \tread_replace_refs = 0;\n \n-\tif (opts.reachable) {\n-\t\twrite_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n-\t\treturn 0;\n-\t}\n+\tif (opts.reachable)\n+\t\treturn write_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n \n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n@@ -173,14 +172,14 @@ static int graph_write(int argc, const char **argv)\n \t\tUNLEAK(buf);\n \t}\n \n-\twrite_commit_graph(opts.obj_dir,\n-\t\t\t pack_indexes,\n-\t\t\t commit_hex,\n-\t\t\t opts.append,\n-\t\t\t 1);\n+\tresult = write_commit_graph(opts.obj_dir,\n+\t\t\t\t pack_indexes,\n+\t\t\t\t commit_hex,\n+\t\t\t\t opts.append,\n+\t\t\t\t 1);\n \n \tUNLEAK(lines);\n-\treturn 0;\n+\treturn result;\n }\n \n int cmd_commit_graph(int argc, const char **argv, const char *prefix)\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 004b816635..04b0717b35 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1667,8 +1667,9 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\t \"new_index file. Check that disk is not full and quota is\\n\"\n \t\t \"not exceeded, and then \\\"git reset HEAD\\\" to recover.\"));\n \n-\tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0))\n-\t\twrite_commit_graph_reachable(get_object_directory(), 0, 0);\n+\tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n+\t write_commit_graph_reachable(get_object_directory(), 0, 0))\n+\t\treturn 1;\n \n \trepo_rerere(the_repository, 0);\n \trun_command_v_opt(argv_gc_auto, RUN_GIT_CMD);\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 7696017cd4..9c6c9c9007 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -662,9 +662,10 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tif (pack_garbage.nr > 0)\n \t\tclean_pack_garbage();\n \n-\tif (gc_write_commit_graph)\n-\t\twrite_commit_graph_reachable(get_object_directory(), 0,\n-\t\t\t\t\t !quiet && !daemonized);\n+\tif (gc_write_commit_graph &&\n+\t write_commit_graph_reachable(get_object_directory(), 0,\n+\t\t\t\t\t !quiet && !daemonized))\n+\t\treturn 1;\n \n \tif (auto_gc && too_many_loose_objects())\n \t\twarning(_(\"There are too many unreachable loose objects; \"\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0f8274d15d..162b9f2a85 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -755,27 +755,30 @@ static int add_ref_to_list(const char *refname,\n \treturn 0;\n }\n \n-void write_commit_graph_reachable(const char *obj_dir, int append,\n-\t\t\t\t int report_progress)\n+int write_commit_graph_reachable(const char *obj_dir, int append,\n+\t\t\t\t int report_progress)\n {\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n+\tint result;\n \n \tfor_each_ref(add_ref_to_list, &list);\n-\twrite_commit_graph(obj_dir, NULL, &list, append, report_progress);\n+\tresult = write_commit_graph(obj_dir, NULL, &list,\n+\t\t\t\t append, report_progress);\n \n \tstring_list_clear(&list, 0);\n+\treturn result;\n }\n \n-void write_commit_graph(const char *obj_dir,\n-\t\t\tstruct string_list *pack_indexes,\n-\t\t\tstruct string_list *commit_hex,\n-\t\t\tint append, int report_progress)\n+int write_commit_graph(const char *obj_dir,\n+\t\t struct string_list *pack_indexes,\n+\t\t struct string_list *commit_hex,\n+\t\t int append, int report_progress)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n \tstruct hashfile *f;\n \tuint32_t i, count_distinct = 0;\n-\tchar *graph_name;\n+\tchar *graph_name = NULL;\n \tstruct lock_file lk = LOCK_INIT;\n \tuint32_t chunk_ids[5];\n \tuint64_t chunk_offsets[5];\n@@ -787,15 +790,17 @@ void write_commit_graph(const char *obj_dir,\n \tuint64_t progress_cnt = 0;\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tunsigned long approx_nr_objects;\n+\tint res = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n-\t\treturn;\n+\t\treturn 0;\n \n \toids.nr = 0;\n \tapprox_nr_objects = approximate_object_count();\n \toids.alloc = approx_nr_objects / 32;\n \toids.progress = NULL;\n \toids.progress_done = 0;\n+\tcommits.list = NULL;\n \n \tif (append) {\n \t\tprepare_commit_graph_one(the_repository, obj_dir);\n@@ -836,10 +841,16 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tstrbuf_setlen(&packname, dirlen);\n \t\t\tstrbuf_addstr(&packname, pack_indexes->items[i].string);\n \t\t\tp = add_packed_git(packname.buf, packname.len, 1);\n-\t\t\tif (!p)\n-\t\t\t\tdie(_(\"error adding pack %s\"), packname.buf);\n-\t\t\tif (open_pack_index(p))\n-\t\t\t\tdie(_(\"error opening index for %s\"), packname.buf);\n+\t\t\tif (!p) {\n+\t\t\t\terror(_(\"error adding pack %s\"), packname.buf);\n+\t\t\t\tres = 1;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\t\t\tif (open_pack_index(p)) {\n+\t\t\t\terror(_(\"error opening index for %s\"), packname.buf);\n+\t\t\t\tres = 1;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \t\t\tfor_each_object_in_pack(p, add_packed_commits, &oids,\n \t\t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n \t\t\tclose_pack(p);\n@@ -910,8 +921,11 @@ void write_commit_graph(const char *obj_dir,\n \t}\n \tstop_progress(&progress);\n \n-\tif (count_distinct >= GRAPH_PARENT_MISSING)\n-\t\tdie(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n+\tif (count_distinct >= GRAPH_PARENT_MISSING) {\n+\t\terror(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n+\t\tres = 1;\n+\t\tgoto cleanup;\n+\t}\n \n \tcommits.nr = 0;\n \tcommits.alloc = count_distinct;\n@@ -943,16 +957,21 @@ void write_commit_graph(const char *obj_dir,\n \tnum_chunks = num_extra_edges ? 4 : 3;\n \tstop_progress(&progress);\n \n-\tif (commits.nr >= GRAPH_PARENT_MISSING)\n-\t\tdie(_(\"too many commits to write graph\"));\n+\tif (commits.nr >= GRAPH_PARENT_MISSING) {\n+\t\terror(_(\"too many commits to write graph\"));\n+\t\tres = 1;\n+\t\tgoto cleanup;\n+\t}\n \n \tcompute_generation_numbers(&commits, report_progress);\n \n \tgraph_name = get_commit_graph_filename(obj_dir);\n \tif (safe_create_leading_directories(graph_name)) {\n \t\tUNLEAK(graph_name);\n-\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n-\t\t\t graph_name);\n+\t\terror(_(\"unable to create leading directories of %s\"),\n+\t\t\tgraph_name);\n+\t\tres = errno;\n+\t\tgoto cleanup;\n \t}\n \n \thold_lock_file_for_update(&lk, graph_name, LOCK_DIE_ON_ERROR);\n@@ -1011,9 +1030,12 @@ void write_commit_graph(const char *obj_dir,\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n \tcommit_lock_file(&lk);\n \n+cleanup:\n \tfree(graph_name);\n \tfree(commits.list);\n \tfree(oids.list);\n+\n+\treturn res;\n }\n \n #define VERIFY_COMMIT_GRAPH_ERROR_HASH 2\ndiff --git a/commit-graph.h b/commit-graph.h\nindex e6aff2c2e1..cd333a0cd0 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -60,12 +60,12 @@ struct commit_graph *load_commit_graph_one(const char *graph_file);\n */\n int generation_numbers_enabled(struct repository *r);\n \n-void write_commit_graph_reachable(const char *obj_dir, int append,\n+int write_commit_graph_reachable(const char *obj_dir, int append,\n \t\t\t\t int report_progress);\n-void write_commit_graph(const char *obj_dir,\n-\t\t\tstruct string_list *pack_indexes,\n-\t\t\tstruct string_list *commit_hex,\n-\t\t\tint append, int report_progress);\n+int write_commit_graph(const char *obj_dir,\n+\t\t struct string_list *pack_indexes,\n+\t\t struct string_list *commit_hex,\n+\t\t int append, int report_progress);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \n-- \ngitgitgadget\n\n"},{"id":"367486","messageId":"43a40d0c43a1402a047b8c329ae8448643bedcce.1548280753.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.git.gitgitgadget@gmail.com","subject":"[PATCH 2/6] commit-graph: collapse parameters into flags","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T21:59:16Z","receivedAt":"2019-01-23T21:59:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() and write_commit_graph_reachable() methods\ncurrently take two boolean parameters: 'append' and 'report_progress'.\nWe will soon expand the possible options to send to these methods, so\ninstead of complicating the parameter list, first simplify it.\n\nCollapse these parameters into a 'flags' parameter, and adjust the\ncallers to provide flags as necessary.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 8 +++++---\n builtin/commit.c | 2 +-\n builtin/gc.c | 4 ++--\n commit-graph.c | 9 +++++----\n commit-graph.h | 8 +++++---\n 5 files changed, 18 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex b12d46fdc8..0c92421f75 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -127,6 +127,7 @@ static int graph_write(int argc, const char **argv)\n \tstruct string_list *commit_hex = NULL;\n \tstruct string_list lines;\n \tint result;\n+\tint flags = COMMIT_GRAPH_PROGRESS;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n@@ -151,11 +152,13 @@ static int graph_write(int argc, const char **argv)\n \t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n+\tif (opts.append)\n+\t\tflags |= COMMIT_GRAPH_APPEND;\n \n \tread_replace_refs = 0;\n \n \tif (opts.reachable)\n-\t\treturn write_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n+\t\treturn write_commit_graph_reachable(opts.obj_dir, flags);\n \n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n@@ -175,8 +178,7 @@ static int graph_write(int argc, const char **argv)\n \tresult = write_commit_graph(opts.obj_dir,\n \t\t\t\t pack_indexes,\n \t\t\t\t commit_hex,\n-\t\t\t\t opts.append,\n-\t\t\t\t 1);\n+\t\t\t\t flags);\n \n \tUNLEAK(lines);\n \treturn result;\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 04b0717b35..3228de4e3c 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1668,7 +1668,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\t \"not exceeded, and then \\\"git reset HEAD\\\" to recover.\"));\n \n \tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n-\t write_commit_graph_reachable(get_object_directory(), 0, 0))\n+\t write_commit_graph_reachable(get_object_directory(), 0))\n \t\treturn 1;\n \n \trepo_rerere(the_repository, 0);\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 9c6c9c9007..198872206b 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -663,8 +663,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tclean_pack_garbage();\n \n \tif (gc_write_commit_graph &&\n-\t write_commit_graph_reachable(get_object_directory(), 0,\n-\t\t\t\t\t !quiet && !daemonized))\n+\t write_commit_graph_reachable(get_object_directory(),\n+\t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_PROGRESS : 0))\n \t\treturn 1;\n \n \tif (auto_gc && too_many_loose_objects())\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 162b9f2a85..28fe2378be 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -755,15 +755,14 @@ static int add_ref_to_list(const char *refname,\n \treturn 0;\n }\n \n-int write_commit_graph_reachable(const char *obj_dir, int append,\n-\t\t\t\t int report_progress)\n+int write_commit_graph_reachable(const char *obj_dir, int flags)\n {\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n \tint result;\n \n \tfor_each_ref(add_ref_to_list, &list);\n \tresult = write_commit_graph(obj_dir, NULL, &list,\n-\t\t\t\t append, report_progress);\n+\t\t\t\t flags);\n \n \tstring_list_clear(&list, 0);\n \treturn result;\n@@ -772,7 +771,7 @@ int write_commit_graph_reachable(const char *obj_dir, int append,\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t int append, int report_progress)\n+\t\t int flags)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n@@ -791,6 +790,8 @@ int write_commit_graph(const char *obj_dir,\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tunsigned long approx_nr_objects;\n \tint res = 0;\n+\tint append = flags & COMMIT_GRAPH_APPEND;\n+\tint report_progress = flags & COMMIT_GRAPH_PROGRESS;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn 0;\ndiff --git a/commit-graph.h b/commit-graph.h\nindex cd333a0cd0..83fa548138 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -60,12 +60,14 @@ struct commit_graph *load_commit_graph_one(const char *graph_file);\n */\n int generation_numbers_enabled(struct repository *r);\n \n-int write_commit_graph_reachable(const char *obj_dir, int append,\n-\t\t\t\t int report_progress);\n+#define COMMIT_GRAPH_APPEND (1 << 0)\n+#define COMMIT_GRAPH_PROGRESS (1 << 1)\n+\n+int write_commit_graph_reachable(const char *obj_dir, int flags);\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t int append, int report_progress);\n+\t\t int flags);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \n-- \ngitgitgadget\n\n"},{"id":"367488","messageId":"c55e0a738c79c30affcf2cb68564c01c36c66dff.1548280753.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.git.gitgitgadget@gmail.com","subject":"[PATCH 5/6] commit-graph: implement file format version 2","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T21:59:19Z","receivedAt":"2019-01-23T21:59:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe commit-graph file format had some shortcomings which we now\ncorrect:\n\n 1. The hash algorithm was determined by a single byte, instead\n of the 4-byte format identifier.\n\n 2. There was no way to update the reachability index we used.\n We currently only support generation numbers, but that will\n change in the future.\n\n 3. Git did not fail with error if the unused eighth byte was\n non-zero, so we could not use that to indicate an incremental\n file format without breaking compatibility across versions.\n\nThe new format modifies the header of the commit-graph to solve\nthese problems. We use the 4-byte hash format id, freeing up a byte\nin our 32-bit alignment to introduce a reachability index version.\nWe can also fail to read the commit-graph if the eighth byte is\nnon-zero.\n\nThe 'git commit-graph read' subcommand needs updating to show the\nnew data.\n\nSet the default file format version to 2, and adjust the tests to\nexpect the new 'git commit-graph read' output.\n\nSigned-off-by: Derrick Stolee \n---\n .../technical/commit-graph-format.txt | 26 +++++++++-\n builtin/commit-graph.c | 9 ++++\n commit-graph.c | 47 ++++++++++++++++---\n commit-graph.h | 1 +\n t/t5318-commit-graph.sh | 9 +++-\n 5 files changed, 83 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex 16452a0504..e367aa94b1 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -31,13 +31,22 @@ and hash type.\n \n All 4-byte numbers are in network order.\n \n+There are two versions available, 1 and 2. These currently differ only in\n+the header.\n+\n HEADER:\n \n+All commit-graph files use the first five bytes for the same purpose.\n+\n 4-byte signature:\n The signature is: {'C', 'G', 'P', 'H'}\n \n 1-byte version number:\n- Currently, the only valid version is 1.\n+ Currently, the valid version numbers are 1 and 2.\n+\n+The remainder of the header changes depending on the version.\n+\n+Version 1:\n \n 1-byte Hash Version (1 = SHA-1)\n We infer the hash length (H) from this value.\n@@ -47,6 +56,21 @@ HEADER:\n 1-byte (reserved for later use)\n Current clients should ignore this value.\n \n+Version 2:\n+\n+ 1-byte number (C) of \"chunks\"\n+\n+ 1-byte reachability index version number:\n+ Currently, the only valid number is 1.\n+\n+ 1-byte (reserved for later use)\n+ Current clients expect this value to be zero, and will not\n+ try to read the commit-graph file if it is non-zero.\n+\n+ 4-byte format identifier for the hash algorithm:\n+ If this identifier does not agree with the repository's current\n+ hash algorithm, then the client will not read the commit graph.\n+\n CHUNK LOOKUP:\n \n (C + 1) * 12 bytes listing the table of contents for the chunks:\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex b1bed84260..28787d0c9c 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -102,6 +102,11 @@ static int graph_read(int argc, const char **argv)\n \t\t*(unsigned char*)(graph->data + 5),\n \t\t*(unsigned char*)(graph->data + 6),\n \t\t*(unsigned char*)(graph->data + 7));\n+\n+\tif (*(unsigned char *)(graph->data + 4) == 2)\n+\t\tprintf(\"hash algorithm: %X\\n\",\n+\t\t get_be32(graph->data + 8));\n+\n \tprintf(\"num_commits: %u\\n\", graph->num_commits);\n \tprintf(\"chunks:\");\n \n@@ -162,6 +167,10 @@ static int graph_write(int argc, const char **argv)\n \tcase 1:\n \t\tflags |= COMMIT_GRAPH_VERSION_1;\n \t\tbreak;\n+\n+\tcase 2:\n+\t\tflags |= COMMIT_GRAPH_VERSION_2;\n+\t\tbreak;\n \t}\n \n \tread_replace_refs = 0;\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f7f45893fd..aeb6cae656 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -90,7 +90,8 @@ struct commit_graph *load_commit_graph_one(const char *graph_file)\n \tuint64_t last_chunk_offset;\n \tuint32_t last_chunk_id;\n \tuint32_t graph_signature;\n-\tunsigned char graph_version, hash_version;\n+\tunsigned char graph_version, hash_version, reach_index_version;\n+\tuint32_t hash_id;\n \n \tif (fd < 0)\n \t\treturn NULL;\n@@ -115,9 +116,9 @@ struct commit_graph *load_commit_graph_one(const char *graph_file)\n \t}\n \n \tgraph_version = *(unsigned char*)(data + 4);\n-\tif (graph_version != 1) {\n-\t\terror(_(\"graph version %X does not match version %X\"),\n-\t\t graph_version, 1);\n+\tif (!graph_version || graph_version > 2) {\n+\t\terror(_(\"unsupported graph version %X\"),\n+\t\t graph_version);\n \t\tgoto cleanup_fail;\n \t}\n \n@@ -135,6 +136,30 @@ struct commit_graph *load_commit_graph_one(const char *graph_file)\n \t\tgraph->num_chunks = *(unsigned char*)(data + 6);\n \t\tchunk_lookup = data + 8;\n \t\tbreak;\n+\n+\tcase 2:\n+\t\tgraph->num_chunks = *(unsigned char *)(data + 5);\n+\n+\t\treach_index_version = *(unsigned char *)(data + 6);\n+\t\tif (reach_index_version != 1) {\n+\t\t\terror(_(\"unsupported reachability index version %d\"),\n+\t\t\t reach_index_version);\n+\t\t\tgoto cleanup_fail;\n+\t\t}\n+\n+\t\tif (*(unsigned char*)(data + 7)) {\n+\t\t\terror(_(\"unsupported value in commit-graph header\"));\n+\t\t\tgoto cleanup_fail;\n+\t\t}\n+\n+\t\thash_id = get_be32(data + 8);\n+\t\tif (hash_id != the_hash_algo->format_id) {\n+\t\t\terror(_(\"commit-graph hash algorithm does not match current algorithm\"));\n+\t\t\tgoto cleanup_fail;\n+\t\t}\n+\n+\t\tchunk_lookup = data + 12;\n+\t\tbreak;\n \t}\n \n \tgraph->hash_len = the_hash_algo->rawsz;\n@@ -802,9 +827,11 @@ int write_commit_graph(const char *obj_dir,\n \n \tif (flags & COMMIT_GRAPH_VERSION_1)\n \t\tversion = 1;\n+\tif (flags & COMMIT_GRAPH_VERSION_2)\n+\t\tversion = 2;\n \tif (!version)\n-\t\tversion = 1;\n-\tif (version != 1) {\n+\t\tversion = 2;\n+\tif (version <= 0 || version > 2) {\n \t\terror(_(\"unsupported commit-graph version %d\"),\n \t\t version);\n \t\treturn 1;\n@@ -1003,6 +1030,14 @@ int write_commit_graph(const char *obj_dir,\n \t\thashwrite_u8(f, 0); /* unused padding byte */\n \t\theader_size = 8;\n \t\tbreak;\n+\n+\tcase 2:\n+\t\thashwrite_u8(f, num_chunks);\n+\t\thashwrite_u8(f, 1); /* reachability index version */\n+\t\thashwrite_u8(f, 0); /* unused padding byte */\n+\t\thashwrite_be32(f, the_hash_algo->format_id);\n+\t\theader_size = 12;\n+\t\tbreak;\n \t}\n \n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\ndiff --git a/commit-graph.h b/commit-graph.h\nindex e03df54e33..050137063b 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -63,6 +63,7 @@ int generation_numbers_enabled(struct repository *r);\n #define COMMIT_GRAPH_APPEND (1 << 0)\n #define COMMIT_GRAPH_PROGRESS (1 << 1)\n #define COMMIT_GRAPH_VERSION_1 (1 << 2)\n+#define COMMIT_GRAPH_VERSION_2 (1 << 3)\n \n int write_commit_graph_reachable(const char *obj_dir, int flags);\n int write_commit_graph(const char *obj_dir,\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex b79d6263e9..3ff5e3b48d 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -65,7 +65,8 @@ graph_read_expect() {\n \t\tNUM_CHUNKS=$((3 + $(echo \"$2\" | wc -w)))\n \tfi\n \tcat >expect <<- EOF\n-\theader: 43475048 1 1 $NUM_CHUNKS 0\n+\theader: 43475048 2 $NUM_CHUNKS 1 0\n+\thash algorithm: 73686131\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata$OPTIONAL\n \tEOF\n@@ -390,10 +391,14 @@ test_expect_success 'detect bad signature' '\n '\n \n test_expect_success 'detect bad version' '\n-\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\02\" \\\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\03\" \\\n \t\t\"graph version\"\n '\n \n+test_expect_success 'detect version 2 with version 1 data' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\02\" \\\n+\t\t\"reachability index version\"\n+'\n test_expect_success 'detect bad hash version' '\n \tcorrupt_graph_and_verify $GRAPH_BYTE_HASH \"\\02\" \\\n \t\t\"hash version\"\n-- \ngitgitgadget\n\n"},{"id":"367487","messageId":"e7ae3007f55efcccd8827ce7d8db5f9dc180cb39.1548280753.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.git.gitgitgadget@gmail.com","subject":"[PATCH 4/6] commit-graph: add --version= option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T21:59:18Z","receivedAt":"2019-01-23T21:59:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAllo the commit-graph builtin to specify the file format version\nusing the '--version=' option. Specify the version exactly in\nthe verification tests as using a different version would change\nthe offsets used in those tests.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 3 +++\n builtin/commit-graph.c | 13 +++++++++++--\n t/t5318-commit-graph.sh | 2 +-\n 3 files changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 624470e198..1d1cc70de4 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -51,6 +51,9 @@ or `--stdin-packs`.)\n +\n With the `--append` option, include all commits that are present in the\n existing commit-graph file.\n++\n+With the `--version=` option, specify the file format version. Used\n+only for testing.\n \n 'read'::\n \ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 0c92421f75..b1bed84260 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -10,7 +10,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph verify [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits] [--version=]\"),\n \tNULL\n };\n \n@@ -25,7 +25,7 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits] [--version=]\"),\n \tNULL\n };\n \n@@ -35,6 +35,7 @@ static struct opts_commit_graph {\n \tint stdin_packs;\n \tint stdin_commits;\n \tint append;\n+\tint version;\n } opts;\n \n \n@@ -141,6 +142,8 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"start walk at commits listed by stdin\")),\n \t\tOPT_BOOL(0, \"append\", &opts.append,\n \t\t\tN_(\"include all commits already in the commit-graph file\")),\n+\t\tOPT_INTEGER(0, \"version\", &opts.version,\n+\t\t\tN_(\"specify the file format version\")),\n \t\tOPT_END(),\n \t};\n \n@@ -155,6 +158,12 @@ static int graph_write(int argc, const char **argv)\n \tif (opts.append)\n \t\tflags |= COMMIT_GRAPH_APPEND;\n \n+\tswitch (opts.version) {\n+\tcase 1:\n+\t\tflags |= COMMIT_GRAPH_VERSION_1;\n+\t\tbreak;\n+\t}\n+\n \tread_replace_refs = 0;\n \n \tif (opts.reachable)\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex f4deb13b1d..b79d6263e9 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -328,7 +328,7 @@ test_expect_success 'replace-objects invalidates commit-graph' '\n \n test_expect_success 'git commit-graph verify' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n-\tgit rev-parse commits/8 | git commit-graph write --stdin-commits &&\n+\tgit rev-parse commits/8 | git commit-graph write --stdin-commits --version=1 &&\n \tgit commit-graph verify >output\n '\n \n-- \ngitgitgadget\n\n"},{"id":"367489","messageId":"693900b4c5af9adb98d98549b9340ad9f9d09562.1548280754.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.git.gitgitgadget@gmail.com","subject":"[PATCH 6/6] commit-graph: test verifying a corrupt v2 header","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T21:59:20Z","receivedAt":"2019-01-23T21:59:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe commit-graph file format v2 changes the v1 data only in the\nheader information. Add tests that check the 'verify' subcommand\ncatches corruption in the v2 header.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5318-commit-graph.sh | 31 +++++++++++++++++++++++++++++++\n 1 file changed, 31 insertions(+)\n\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 3ff5e3b48d..be7bbf911a 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -497,6 +497,37 @@ test_expect_success 'git fsck (checks commit-graph)' '\n \ttest_must_fail git fsck\n '\n \n+test_expect_success 'rewrite commmit-graph with version 2' '\n+\trm -f .git/objects/info/commit-graph &&\n+\tgit commit-graph write --reachable --version=2 &&\n+\tgit commit-graph verify\n+'\n+\n+GRAPH_BYTE_CHUNK_COUNT=5\n+GRAPH_BYTE_REACH_INDEX=6\n+GRAPH_BYTE_UNUSED=7\n+GRAPH_BYTE_HASH=8\n+\n+test_expect_success 'detect low chunk count (v2)' '\n+\tcorrupt_graph_and_verify $GRAPH_CHUNK_COUNT \"\\02\" \\\n+\t\t\"missing the .* chunk\"\n+'\n+\n+test_expect_success 'detect incorrect reachability index' '\n+\tcorrupt_graph_and_verify $GRAPH_REACH_INDEX \"\\03\" \\\n+\t\t\"reachability index version\"\n+'\n+\n+test_expect_success 'detect non-zero unused byte' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_UNUSED \"\\01\" \\\n+\t\t\"unsupported value\"\n+'\n+\n+test_expect_success 'detect bad hash version (v2)' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_HASH \"\\00\" \\\n+\t\t\"hash algorithm\"\n+'\n+\n test_expect_success 'setup non-the_repository tests' '\n \trm -rf repo &&\n \tgit init repo &&\n-- \ngitgitgadget\n"},{"id":"367490","messageId":"39319e36bc35dc89aff7640a11c0819c3bce8b2f.1548280753.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.git.gitgitgadget@gmail.com","subject":"[PATCH 3/6] commit-graph: create new version flags","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T21:59:17Z","receivedAt":"2019-01-23T21:59:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn anticipation of a new commit-graph file format version, create\na flag for the write_commit_graph() and write_commit_graph_reachable()\nmethods to take a version number.\n\nWhen there is no specified version, the implementation selects a\ndefault value. Currently, the only valid value is 1.\n\nThe file format will change the header information, so place the\nexisting header logic inside a switch statement with only one case.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 58 +++++++++++++++++++++++++++++++++-----------------\n commit-graph.h | 1 +\n 2 files changed, 40 insertions(+), 19 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 28fe2378be..f7f45893fd 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -25,9 +25,6 @@\n \n #define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n \n-#define GRAPH_VERSION_1 0x1\n-#define GRAPH_VERSION GRAPH_VERSION_1\n-\n #define GRAPH_EXTRA_EDGES_NEEDED 0x80000000\n #define GRAPH_PARENT_MISSING 0x7fffffff\n #define GRAPH_EDGE_LAST_MASK 0x7fffffff\n@@ -118,30 +115,35 @@ struct commit_graph *load_commit_graph_one(const char *graph_file)\n \t}\n \n \tgraph_version = *(unsigned char*)(data + 4);\n-\tif (graph_version != GRAPH_VERSION) {\n+\tif (graph_version != 1) {\n \t\terror(_(\"graph version %X does not match version %X\"),\n-\t\t graph_version, GRAPH_VERSION);\n-\t\tgoto cleanup_fail;\n-\t}\n-\n-\thash_version = *(unsigned char*)(data + 5);\n-\tif (hash_version != oid_version()) {\n-\t\terror(_(\"hash version %X does not match version %X\"),\n-\t\t hash_version, oid_version());\n+\t\t graph_version, 1);\n \t\tgoto cleanup_fail;\n \t}\n \n \tgraph = alloc_commit_graph();\n \n+\tswitch (graph_version) {\n+\tcase 1:\n+\t\thash_version = *(unsigned char*)(data + 5);\n+\t\tif (hash_version != oid_version()) {\n+\t\t\terror(_(\"hash version %X does not match version %X\"),\n+\t\t\t hash_version, oid_version());\n+\t\t\tgoto cleanup_fail;\n+\t\t}\n+\n+\t\tgraph->num_chunks = *(unsigned char*)(data + 6);\n+\t\tchunk_lookup = data + 8;\n+\t\tbreak;\n+\t}\n+\n \tgraph->hash_len = the_hash_algo->rawsz;\n-\tgraph->num_chunks = *(unsigned char*)(data + 6);\n \tgraph->graph_fd = fd;\n \tgraph->data = graph_map;\n \tgraph->data_len = graph_size;\n \n \tlast_chunk_id = 0;\n \tlast_chunk_offset = 8;\n-\tchunk_lookup = data + 8;\n \tfor (i = 0; i < graph->num_chunks; i++) {\n \t\tuint32_t chunk_id = get_be32(chunk_lookup + 0);\n \t\tuint64_t chunk_offset = get_be64(chunk_lookup + 4);\n@@ -792,10 +794,22 @@ int write_commit_graph(const char *obj_dir,\n \tint res = 0;\n \tint append = flags & COMMIT_GRAPH_APPEND;\n \tint report_progress = flags & COMMIT_GRAPH_PROGRESS;\n+\tint version = 0;\n+\tint header_size = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn 0;\n \n+\tif (flags & COMMIT_GRAPH_VERSION_1)\n+\t\tversion = 1;\n+\tif (!version)\n+\t\tversion = 1;\n+\tif (version != 1) {\n+\t\terror(_(\"unsupported commit-graph version %d\"),\n+\t\t version);\n+\t\treturn 1;\n+\t}\n+\n \toids.nr = 0;\n \tapprox_nr_objects = approximate_object_count();\n \toids.alloc = approx_nr_objects / 32;\n@@ -980,10 +994,16 @@ int write_commit_graph(const char *obj_dir,\n \n \thashwrite_be32(f, GRAPH_SIGNATURE);\n \n-\thashwrite_u8(f, GRAPH_VERSION);\n-\thashwrite_u8(f, oid_version());\n-\thashwrite_u8(f, num_chunks);\n-\thashwrite_u8(f, 0); /* unused padding byte */\n+\thashwrite_u8(f, version);\n+\n+\tswitch (version) {\n+\tcase 1:\n+\t\thashwrite_u8(f, oid_version());\n+\t\thashwrite_u8(f, num_chunks);\n+\t\thashwrite_u8(f, 0); /* unused padding byte */\n+\t\theader_size = 8;\n+\t\tbreak;\n+\t}\n \n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n@@ -994,7 +1014,7 @@ int write_commit_graph(const char *obj_dir,\n \t\tchunk_ids[3] = 0;\n \tchunk_ids[4] = 0;\n \n-\tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n+\tchunk_offsets[0] = header_size + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n \tchunk_offsets[2] = chunk_offsets[1] + hashsz * commits.nr;\n \tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * commits.nr;\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 83fa548138..e03df54e33 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -62,6 +62,7 @@ int generation_numbers_enabled(struct repository *r);\n \n #define COMMIT_GRAPH_APPEND (1 << 0)\n #define COMMIT_GRAPH_PROGRESS (1 << 1)\n+#define COMMIT_GRAPH_VERSION_1 (1 << 2)\n \n int write_commit_graph_reachable(const char *obj_dir, int flags);\n int write_commit_graph(const char *obj_dir,\n-- \ngitgitgadget\n\n"},{"id":"367508","messageId":"20190123235630.183779-1-jonathantanmy@google.com","threadId":"50302","inReplyTo":"c55e0a738c79c30affcf2cb68564c01c36c66dff.1548280753.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/6] commit-graph: implement file format version 2","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2019-01-23T23:56:30Z","receivedAt":"2019-01-23T23:56:36Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> +Version 2:\n> +\n> + 1-byte number (C) of \"chunks\"\n> +\n> + 1-byte reachability index version number:\n> + Currently, the only valid number is 1.\n> +\n> + 1-byte (reserved for later use)\n> + Current clients expect this value to be zero, and will not\n> + try to read the commit-graph file if it is non-zero.\n> +\n> + 4-byte format identifier for the hash algorithm:\n> + If this identifier does not agree with the repository's current\n> + hash algorithm, then the client will not read the commit graph.\n\n[snip]\n\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index b79d6263e9..3ff5e3b48d 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -65,7 +65,8 @@ graph_read_expect() {\n> \t\tNUM_CHUNKS=$((3 + $(echo \"$2\" | wc -w)))\n> \tfi\n> \tcat >expect <<- EOF\n> -\theader: 43475048 1 1 $NUM_CHUNKS 0\n> +\theader: 43475048 2 $NUM_CHUNKS 1 0\n> +\thash algorithm: 73686131\n> \tnum_commits: $1\n> \tchunks: oid_fanout oid_lookup commit_metadata$OPTIONAL\n> \tEOF\n> @@ -390,10 +391,14 @@ test_expect_success 'detect bad signature' '\n> '\n> \n> test_expect_success 'detect bad version' '\n> -\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\02\" \\\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\03\" \\\n> \t\t\"graph version\"\n> '\n> \n> +test_expect_success 'detect version 2 with version 1 data' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\02\" \\\n> +\t\t\"reachability index version\"\n> +'\n> test_expect_success 'detect bad hash version' '\n> \tcorrupt_graph_and_verify $GRAPH_BYTE_HASH \"\\02\" \\\n> \t\t\"hash version\"\n\nShould there also be a test that the \"reserved\" section be 0 and the\n4-byte identifier agrees with the repo's hash algorithm? I assume that\nthis can be done by \"corrupting\" the version to 2 and then truly\ncorrupting the subsequent bytes.\n\nOther than that, this and the previous patches look good.\n"},{"id":"367509","messageId":"20190123235933.185949-1-jonathantanmy@google.com","threadId":"50302","inReplyTo":"693900b4c5af9adb98d98549b9340ad9f9d09562.1548280754.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 6/6] commit-graph: test verifying a corrupt v2 header","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2019-01-23T23:59:33Z","receivedAt":"2019-01-23T23:59:40Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"> From: Derrick Stolee \n> \n> The commit-graph file format v2 changes the v1 data only in the\n> header information. Add tests that check the 'verify' subcommand\n> catches corruption in the v2 header.\n\nAh, I should have read this patch before I wrote [1]. I think the commit\nmessage of that patch should contain a note that verification of the v2\nfile format is done in a subsequent patch.\n\nWith or without that additional note, this series looks good to me.\n\n[1] https://public-inbox.org/git/20190123235630.183779-1-jonathantanmy@google.com/\n\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> t/t5318-commit-graph.sh | 31 +++++++++++++++++++++++++++++++\n> 1 file changed, 31 insertions(+)\n> \n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 3ff5e3b48d..be7bbf911a 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -497,6 +497,37 @@ test_expect_success 'git fsck (checks commit-graph)' '\n> \ttest_must_fail git fsck\n> '\n> \n> +test_expect_success 'rewrite commmit-graph with version 2' '\n> +\trm -f .git/objects/info/commit-graph &&\n> +\tgit commit-graph write --reachable --version=2 &&\n> +\tgit commit-graph verify\n> +'\n> +\n> +GRAPH_BYTE_CHUNK_COUNT=5\n> +GRAPH_BYTE_REACH_INDEX=6\n> +GRAPH_BYTE_UNUSED=7\n> +GRAPH_BYTE_HASH=8\n> +\n> +test_expect_success 'detect low chunk count (v2)' '\n> +\tcorrupt_graph_and_verify $GRAPH_CHUNK_COUNT \"\\02\" \\\n> +\t\t\"missing the .* chunk\"\n> +'\n> +\n> +test_expect_success 'detect incorrect reachability index' '\n> +\tcorrupt_graph_and_verify $GRAPH_REACH_INDEX \"\\03\" \\\n> +\t\t\"reachability index version\"\n> +'\n> +\n> +test_expect_success 'detect non-zero unused byte' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_UNUSED \"\\01\" \\\n> +\t\t\"unsupported value\"\n> +'\n> +\n> +test_expect_success 'detect bad hash version (v2)' '\n> +\tcorrupt_graph_and_verify $GRAPH_BYTE_HASH \"\\00\" \\\n> +\t\t\"hash algorithm\"\n> +'\n> +\n> test_expect_success 'setup non-the_repository tests' '\n> \trm -rf repo &&\n> \tgit init repo &&\n> -- \n> gitgitgadget\n"},{"id":"367531","messageId":"87a7jq4d1u.fsf@evledraar.gmail.com","threadId":"50302","inReplyTo":"e7ae3007f55efcccd8827ce7d8db5f9dc180cb39.1548280753.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/6] commit-graph: add --version= option","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-24T09:31:25Z","receivedAt":"2019-01-24T09:31:30Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jan 23 2019, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> Allo the commit-graph builtin to specify the file format version\n\n\"Allow\"\n"},{"id":"367532","messageId":"878sza4cna.fsf@evledraar.gmail.com","threadId":"50302","inReplyTo":"c55e0a738c79c30affcf2cb68564c01c36c66dff.1548280753.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/6] commit-graph: implement file format version 2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-01-24T09:40:09Z","receivedAt":"2019-01-24T09:40:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jan 23 2019, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> The commit-graph file format had some shortcomings which we now\n> correct:\n>\n> 1. The hash algorithm was determined by a single byte, instead\n> of the 4-byte format identifier.\n>\n> 2. There was no way to update the reachability index we used.\n> We currently only support generation numbers, but that will\n> change in the future.\n>\n> 3. Git did not fail with error if the unused eighth byte was\n> non-zero, so we could not use that to indicate an incremental\n> file format without breaking compatibility across versions.\n>\n> The new format modifies the header of the commit-graph to solve\n> these problems. We use the 4-byte hash format id, freeing up a byte\n> in our 32-bit alignment to introduce a reachability index version.\n> We can also fail to read the commit-graph if the eighth byte is\n> non-zero.\n\nI haven't tested, but it looks from the patch like we can transparently\nread existing v1 data and then will write v2 the next time. Would be\nhelpful for reviewers if this was noted explicitly in the commit\nmessage.\n\nShould there be a GIT_TEST_COMMIT_GRAPH_VERSION=[12] going forward to\ntest the non-default version, or do you feel confident the tests added\nhere test the upgrade path & old code well enough?\n\n> The 'git commit-graph read' subcommand needs updating to show the\n> new data.\n\nLet's say \"The ... subcommand has been updated to show the new\ndata\". This sounds like a later patch is going to do that, but in fact\nit's done here.\n\n> Set the default file format version to 2, and adjust the tests to\n> expect the new 'git commit-graph read' output.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> .../technical/commit-graph-format.txt | 26 +++++++++-\n> builtin/commit-graph.c | 9 ++++\n> commit-graph.c | 47 ++++++++++++++++---\n> commit-graph.h | 1 +\n> t/t5318-commit-graph.sh | 9 +++-\n> 5 files changed, 83 insertions(+), 9 deletions(-)\n>\n> diff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\n> index 16452a0504..e367aa94b1 100644\n> --- a/Documentation/technical/commit-graph-format.txt\n> +++ b/Documentation/technical/commit-graph-format.txt\n> @@ -31,13 +31,22 @@ and hash type.\n>\n> All 4-byte numbers are in network order.\n>\n> +There are two versions available, 1 and 2. These currently differ only in\n> +the header.\n\nShouldn't this be s/currently/ / ? Won't we add a version 3 if we make\nnew changes?\n"},{"id":"367554","messageId":"cd07c73f-e839-844c-9652-1bdf95724f5a@gmail.com","threadId":"50302","inReplyTo":"878sza4cna.fsf@evledraar.gmail.com","subject":"Re: [PATCH 5/6] commit-graph: implement file format version 2","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-01-24T14:34:40Z","receivedAt":"2019-01-24T14:34:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/24/2019 4:40 AM, Ævar Arnfjörð Bjarmason wrote:\n> On Wed, Jan 23 2019, Derrick Stolee via GitGitGadget wrote:\n>\n>> The new format modifies the header of the commit-graph to solve\n>> these problems. We use the 4-byte hash format id, freeing up a byte\n>> in our 32-bit alignment to introduce a reachability index version.\n>> We can also fail to read the commit-graph if the eighth byte is\n>> non-zero.\n> I haven't tested, but it looks from the patch like we can transparently\n> read existing v1 data and then will write v2 the next time. Would be\n> helpful for reviewers if this was noted explicitly in the commit\n> message.\n\nCan do.\n\n>\n> Should there be a GIT_TEST_COMMIT_GRAPH_VERSION=[12] going forward to\n> test the non-default version, or do you feel confident the tests added\n> here test the upgrade path & old code well enough?\n\nYou're right that we should have an explicit \"upgrade\" test:\n\n  1. Write a v1 commit-graph\n  2. Add a commit\n  3. Write a v2 commit-graph\n\nAs for a new GIT_TEST_ variable, we should only need to continue relying \non GIT_TEST_COMMIT_GRAPH to test v2. I can add a 'graph_git_behavior' \ncall on an explicitly v1 commit-graph file to get most of the coverage \nwe need.\n\n>> The 'git commit-graph read' subcommand needs updating to show the\n>> new data.\n> Let's say \"The ... subcommand has been updated to show the new\n> data\". This sounds like a later patch is going to do that, but in fact\n> it's done here.\n\nWill clean up.\n\n>> Set the default file format version to 2, and adjust the tests to\n>> expect the new 'git commit-graph read' output.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> .../technical/commit-graph-format.txt | 26 +++++++++-\n>> builtin/commit-graph.c | 9 ++++\n>> commit-graph.c | 47 ++++++++++++++++---\n>> commit-graph.h | 1 +\n>> t/t5318-commit-graph.sh | 9 +++-\n>> 5 files changed, 83 insertions(+), 9 deletions(-)\n>>\n>> diff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\n>> index 16452a0504..e367aa94b1 100644\n>> --- a/Documentation/technical/commit-graph-format.txt\n>> +++ b/Documentation/technical/commit-graph-format.txt\n>> @@ -31,13 +31,22 @@ and hash type.\n>>\n>> All 4-byte numbers are in network order.\n>>\n>> +There are two versions available, 1 and 2. These currently differ only in\n>> +the header.\n> Shouldn't this be s/currently/ / ? Won't we add a version 3 if we make\n> new changes?\n\nWhen we add a new reachability index version, then the content of the \ndata chunk will change. Since we have a separate byte for versioning \nthat data, we do not need a v3 for the file format as a whole. A similar \nstatement applies to the unused byte reserved for the incremental file \nformat: we won't need to increase the file format version as we will \nmake that number non-zero and add a chunk with extra data.\n\nThanks,\n-Stolee\n"},{"id":"367627","messageId":"xmqq4l9xhd1h.fsf@gitster-ct.c.googlers.com","threadId":"50302","inReplyTo":"pull.112.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/6] Create commit-graph file format v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-24T23:05:30Z","receivedAt":"2019-01-24T23:05:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> The commit-graph file format has some shortcomings that were discussed\n> on-list:\n> ...\n> This series adds a new version (2) to the commit-graph file.\n\nSigh. It is unfortunate that we have to bump the format version\nthis early before we can say \"it is no longer experimental\" with\nconfidence X-<. Perhaps we have been moving too fast for our own\ngood and should slow down in introducing new low-level machineries?\n\nWill queue. Thanks.\n"},{"id":"367628","messageId":"xmqqzhrpfwvk.fsf@gitster-ct.c.googlers.com","threadId":"50302","inReplyTo":"pull.112.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/6] Create commit-graph file format v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-24T23:39:59Z","receivedAt":"2019-01-24T23:40:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> This series is based on ab/commit-graph-write-progress and bc/sha-256.\n\nThanks.\n\nIt seems that the base (i.e. merge between these two topics) you\nused may have used a version of either topic (most likely the\nlatter) slightly older than what I have, as patches 1 and 2 seem to\nlack the local variable \"hashsz\" in the context, near the beginning\nof commit_graph_write(). I wiggled the patches in, but it has too\nheavy conflict merging to 'pu', so it may have to wait until the\nother topics stabilize a bit further.\n\n"},{"id":"367659","messageId":"82be29f0-9b68-4bd6-4d5f-c6cfa1a0ccc6@gmail.com","threadId":"50302","inReplyTo":"xmqqzhrpfwvk.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 0/6] Create commit-graph file format v2","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-01-25T13:54:27Z","receivedAt":"2019-01-25T13:54:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/24/2019 6:39 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> This series is based on ab/commit-graph-write-progress and bc/sha-256.\n> \n> Thanks.\n> \n> It seems that the base (i.e. merge between these two topics) you\n> used may have used a version of either topic (most likely the\n> latter) slightly older than what I have, as patches 1 and 2 seem to\n> lack the local variable \"hashsz\" in the context, near the beginning\n> of commit_graph_write(). I wiggled the patches in, but it has too\n> heavy conflict merging to 'pu', so it may have to wait until the\n> other topics stabilize a bit further.\n\nSorry that the merge was painful. I would have waited longer for\nthings to stabilize, but I'm expecting to go on paternity leave\nsoon. Didn't want to get the idea out there before I disappear\nfor a while.\n\nWhen things stabilize, I may have time to do a rebase and work\nout the details myself. Otherwise, everyone has my blessing to\ntake work I've started and move it forward themselves.\n\nThanks,\n-Stolee\n"},{"id":"372077","messageId":"87tvfwd1l8.fsf@evledraar.gmail.com","threadId":"50302","inReplyTo":"c55e0a738c79c30affcf2cb68564c01c36c66dff.1548280753.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/6] commit-graph: implement file format version 2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-03-21T09:21:55Z","receivedAt":"2019-03-21T09:22:00Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jan 23 2019, Derrick Stolee via GitGitGadget wrote:\n\n> \tgraph_version = *(unsigned char*)(data + 4);\n> -\tif (graph_version != 1) {\n> -\t\terror(_(\"graph version %X does not match version %X\"),\n> -\t\t graph_version, 1);\n> +\tif (!graph_version || graph_version > 2) {\n> +\t\terror(_(\"unsupported graph version %X\"),\n> +\t\t graph_version);\n> \t\tgoto cleanup_fail;\n> \t}\n\nJust noticed this while writing\nhttps://public-inbox.org/git/87va0cd1zp.fsf@evledraar.gmail.com/ i.e. to\nresolve the conflict with my commit-graph segfault fixing series.\n\nThis really should be something like:\n\n\t/* earlier */\n\t#define GRAPH_MAX_VERSION 2\n\n\tif (!graph_version || graph_version > GRAPH_MAX_VERSION) {\n\t\terror(_(\"commit-graph unsupported graph version %X, we support up to %X\"),\n\t\t\tgraph_version, GRAPH_MAX_VERSION);\n\nAlso, I'm confused as to what these patches are based on. Your is doing\n\"!= 1\", but on \"master\" and ever since 2a2e32bdc5 (\"commit-graph:\nimplement git commit-graph read\", 2018-04-10) this has been the macro\n\"GRAPH_VERSION\" instead of \"1\".\n"},{"id":"374418","messageId":"pull.112.v2.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.git.gitgitgadget@gmail.com","subject":"[PATCH v2 0/5] Create commit-graph file format v2","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-04-24T19:58:02Z","receivedAt":"2019-04-24T19:58:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file format has some shortcomings that were discussed\non-list:\n\n 1. It doesn't use the 4-byte format ID from the_hash_algo.\n \n \n 2. There is no way to change the reachability index from generation numbers\n to corrected commit date [1].\n \n \n 3. The unused byte in the format could be used to signal the file is\n incremental, but current clients ignore the value even if it is\n non-zero.\n \n \n\nThis series adds a new version (2) to the commit-graph file. The fifth byte\nalready specified the file format, so existing clients will gracefully\nrespond to files with a different version number. The only real change now\nis that the header takes 12 bytes instead of 8, due to using the 4-byte\nformat ID for the hash algorithm.\n\nThe new bytes reserved for the reachability index version and incremental\nfile formats are now expected to be equal to the defaults. When we update\nthese values to be flexible in the future, if a client understands\ncommit-graph v2 but not those new values, then it will fail gracefully.\n\nNOTE: this series was rebased onto ab/commit-graph-fixes, as the conflicts\nwere significant and subtle.\n\nThanks, -Stolee\n\n[1] \nhttps://public-inbox.org/git/6367e30a-1b3a-4fe9-611b-d931f51effef@gmail.com/\n\nDerrick Stolee (5):\n commit-graph: return with errors during write\n commit-graph: collapse parameters into flags\n commit-graph: create new version flags\n commit-graph: add --version= option\n commit-graph: implement file format version 2\n\n Documentation/git-commit-graph.txt | 3 +\n .../technical/commit-graph-format.txt | 26 ++-\n builtin/commit-graph.c | 43 +++--\n builtin/commit.c | 5 +-\n builtin/gc.c | 7 +-\n commit-graph.c | 156 +++++++++++++-----\n commit-graph.h | 16 +-\n t/t5318-commit-graph.sh | 68 +++++++-\n 8 files changed, 254 insertions(+), 70 deletions(-)\n\n\nbase-commit: 93b4405ffe4ad9308740e7c1c71383bfc369baaa\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-112%2Fderrickstolee%2Fgraph%2Fv2-head-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-112/derrickstolee/graph/v2-head-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/112\n\nRange-diff vs v1:\n\n 1: e72498d0c5 ! 1: 91f300ec0a commit-graph: return with errors during write\n @@ -82,8 +82,8 @@\n --- a/builtin/gc.c\n +++ b/builtin/gc.c\n @@\n - \tif (pack_garbage.nr > 0)\n \t\tclean_pack_garbage();\n + \t}\n \n -\tif (gc_write_commit_graph)\n -\t\twrite_commit_graph_reachable(get_object_directory(), 0,\n @@ -182,9 +182,9 @@\n \t}\n \tstop_progress(&progress);\n \n --\tif (count_distinct >= GRAPH_PARENT_MISSING)\n +-\tif (count_distinct >= GRAPH_EDGE_LAST_MASK)\n -\t\tdie(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n -+\tif (count_distinct >= GRAPH_PARENT_MISSING) {\n ++\tif (count_distinct >= GRAPH_EDGE_LAST_MASK) {\n +\t\terror(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n +\t\tres = 1;\n +\t\tgoto cleanup;\n @@ -196,9 +196,9 @@\n \tnum_chunks = num_extra_edges ? 4 : 3;\n \tstop_progress(&progress);\n \n --\tif (commits.nr >= GRAPH_PARENT_MISSING)\n +-\tif (commits.nr >= GRAPH_EDGE_LAST_MASK)\n -\t\tdie(_(\"too many commits to write graph\"));\n -+\tif (commits.nr >= GRAPH_PARENT_MISSING) {\n ++\tif (commits.nr >= GRAPH_EDGE_LAST_MASK) {\n +\t\terror(_(\"too many commits to write graph\"));\n +\t\tres = 1;\n +\t\tgoto cleanup;\n 2: 43a40d0c43 ! 2: 04f5df1135 commit-graph: collapse parameters into flags\n @@ -66,7 +66,7 @@\n --- a/builtin/gc.c\n +++ b/builtin/gc.c\n @@\n - \t\tclean_pack_garbage();\n + \t}\n \n \tif (gc_write_commit_graph &&\n -\t write_commit_graph_reachable(get_object_directory(), 0,\n 3: 39319e36bc ! 3: 4ddb829163 commit-graph: create new version flags\n @@ -25,25 +25,25 @@\n -#define GRAPH_VERSION GRAPH_VERSION_1\n -\n #define GRAPH_EXTRA_EDGES_NEEDED 0x80000000\n - #define GRAPH_PARENT_MISSING 0x7fffffff\n #define GRAPH_EDGE_LAST_MASK 0x7fffffff\n + #define GRAPH_PARENT_NONE 0x70000000\n @@\n \t}\n \n \tgraph_version = *(unsigned char*)(data + 4);\n -\tif (graph_version != GRAPH_VERSION) {\n +\tif (graph_version != 1) {\n - \t\terror(_(\"graph version %X does not match version %X\"),\n + \t\terror(_(\"commit-graph version %X does not match version %X\"),\n -\t\t graph_version, GRAPH_VERSION);\n --\t\tgoto cleanup_fail;\n +-\t\treturn NULL;\n -\t}\n -\n -\thash_version = *(unsigned char*)(data + 5);\n -\tif (hash_version != oid_version()) {\n --\t\terror(_(\"hash version %X does not match version %X\"),\n +-\t\terror(_(\"commit-graph hash version %X does not match version %X\"),\n -\t\t hash_version, oid_version());\n +\t\t graph_version, 1);\n - \t\tgoto cleanup_fail;\n + \t\treturn NULL;\n \t}\n \n \tgraph = alloc_commit_graph();\n @@ -52,9 +52,9 @@\n +\tcase 1:\n +\t\thash_version = *(unsigned char*)(data + 5);\n +\t\tif (hash_version != oid_version()) {\n -+\t\t\terror(_(\"hash version %X does not match version %X\"),\n ++\t\t\terror(_(\"commit-graph hash version %X does not match version %X\"),\n +\t\t\t hash_version, oid_version());\n -+\t\t\tgoto cleanup_fail;\n ++\t\t\treturn NULL;\n +\t\t}\n +\n +\t\tgraph->num_chunks = *(unsigned char*)(data + 6);\n @@ -72,8 +72,8 @@\n \tlast_chunk_offset = 8;\n -\tchunk_lookup = data + 8;\n \tfor (i = 0; i < graph->num_chunks; i++) {\n - \t\tuint32_t chunk_id = get_be32(chunk_lookup + 0);\n - \t\tuint64_t chunk_offset = get_be64(chunk_lookup + 4);\n + \t\tuint32_t chunk_id;\n + \t\tuint64_t chunk_offset;\n @@\n \tint res = 0;\n \tint append = flags & COMMIT_GRAPH_APPEND;\n 4: e7ae3007f5 ! 4: b1b0c76eb4 commit-graph: add --version= option\n @@ -2,7 +2,7 @@\n \n commit-graph: add --version= option\n \n - Allo the commit-graph builtin to specify the file format version\n + Allow the commit-graph builtin to specify the file format version\n using the '--version=' option. Specify the version exactly in\n the verification tests as using a different version would change\n the offsets used in those tests.\n 5: c55e0a738c ! 5: 09362bda1b commit-graph: implement file format version 2\n @@ -22,12 +22,20 @@\n We can also fail to read the commit-graph if the eighth byte is\n non-zero.\n \n - The 'git commit-graph read' subcommand needs updating to show the\n - new data.\n + Update the 'git commit-graph read' subcommand to display the new\n + data.\n \n Set the default file format version to 2, and adjust the tests to\n expect the new 'git commit-graph read' output.\n \n + Add explicit tests for the upgrade path from version 1 to 2. Users\n + with an existing commit-graph with version 1 will seamlessly\n + upgrade to version 2 on their next write.\n +\n + While we converted the existing 'verify' tests to use a version 1\n + file to avoid recalculating data offsets, add explicit 'verify'\n + tests on a version 2 file that corrupt the new header values.\n +\n Signed-off-by: Derrick Stolee \n \n diff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\n @@ -118,21 +126,17 @@\n +\tunsigned char graph_version, hash_version, reach_index_version;\n +\tuint32_t hash_id;\n \n - \tif (fd < 0)\n + \tif (!graph_map)\n \t\treturn NULL;\n @@\n \t}\n \n \tgraph_version = *(unsigned char*)(data + 4);\n -\tif (graph_version != 1) {\n --\t\terror(_(\"graph version %X does not match version %X\"),\n --\t\t graph_version, 1);\n +\tif (!graph_version || graph_version > 2) {\n -+\t\terror(_(\"unsupported graph version %X\"),\n -+\t\t graph_version);\n - \t\tgoto cleanup_fail;\n - \t}\n - \n + \t\terror(_(\"commit-graph version %X does not match version %X\"),\n + \t\t graph_version, 1);\n + \t\treturn NULL;\n @@\n \t\tgraph->num_chunks = *(unsigned char*)(data + 6);\n \t\tchunk_lookup = data + 8;\n @@ -145,18 +149,18 @@\n +\t\tif (reach_index_version != 1) {\n +\t\t\terror(_(\"unsupported reachability index version %d\"),\n +\t\t\t reach_index_version);\n -+\t\t\tgoto cleanup_fail;\n ++\t\t\treturn NULL;\n +\t\t}\n +\n +\t\tif (*(unsigned char*)(data + 7)) {\n +\t\t\terror(_(\"unsupported value in commit-graph header\"));\n -+\t\t\tgoto cleanup_fail;\n ++\t\t\treturn NULL;\n +\t\t}\n +\n +\t\thash_id = get_be32(data + 8);\n +\t\tif (hash_id != the_hash_algo->format_id) {\n +\t\t\terror(_(\"commit-graph hash algorithm does not match current algorithm\"));\n -+\t\t\tgoto cleanup_fail;\n ++\t\t\treturn NULL;\n +\t\t}\n +\n +\t\tchunk_lookup = data + 12;\n @@ -209,6 +213,31 @@\n diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n --- a/t/t5318-commit-graph.sh\n +++ b/t/t5318-commit-graph.sh\n +@@\n + \tgit repack\n + '\n + \n +-graph_git_two_modes() {\n ++graph_git_two_modes () {\n + \tgit -c core.commitGraph=true $1 >output\n + \tgit -c core.commitGraph=false $1 >expect\n + \ttest_cmp expect output\n + }\n + \n +-graph_git_behavior() {\n ++graph_git_behavior () {\n + \tMSG=$1\n + \tDIR=$2\n + \tBRANCH=$3\n +@@\n + \n + graph_git_behavior 'no graph' full commits/3 commits/1\n + \n +-graph_read_expect() {\n ++graph_read_expect () {\n + \tOPTIONAL=\"\"\n + \tNUM_CHUNKS=3\n + \tif test ! -z $2\n @@\n \t\tNUM_CHUNKS=$((3 + $(echo \"$2\" | wc -w)))\n \tfi\n @@ -219,6 +248,40 @@\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata$OPTIONAL\n \tEOF\n +@@\n + \t)\n + '\n + \n ++test_expect_success 'write v1 graph' '\n ++\tgit commit-graph write --reachable --version=1 &&\n ++\tgit commit-graph verify\n ++'\n ++\n ++graph_git_behavior 'version 1 graph, commit 8 vs merge 2' full commits/8 merge/2\n ++graph_git_behavior 'version 1 graph, commit 8 vs merge 2' full commits/8 merge/2\n ++\n ++test_expect_success 'upgrade from v1 to v2' '\n ++\tgit checkout -b new-commit-for-upgrade &&\n ++\ttest_commit force-upgrade &&\n ++\tgit commit-graph write --reachable --version=2 &&\n ++\tgit commit-graph verify\n ++'\n ++\n ++graph_git_behavior 'upgraded graph, commit 8 vs merge 2' full commits/8 merge/2\n ++graph_git_behavior 'upgraded graph, commit 8 vs merge 2' full commits/8 merge/2\n ++\n + # the verify tests below expect the commit-graph to contain\n + # exactly the commits reachable from the commits/8 branch.\n + # If the file changes the set of commits in the list, then the\n +@@\n + # starting at , then runs 'git commit-graph verify'\n + # and places the output in the file 'err'. Test 'err' for\n + # the given string.\n +-corrupt_graph_and_verify() {\n ++corrupt_graph_and_verify () {\n + \tpos=$1\n + \tdata=\"${2:-\\0}\"\n + \tgrepstr=$3\n @@\n '\n \n @@ -235,3 +298,41 @@\n test_expect_success 'detect bad hash version' '\n \tcorrupt_graph_and_verify $GRAPH_BYTE_HASH \"\\02\" \\\n \t\t\"hash version\"\n +@@\n + \ttest_must_fail git fsck\n + '\n + \n ++test_expect_success 'rewrite commmit-graph with version 2' '\n ++\trm -f .git/objects/info/commit-graph &&\n ++\tgit commit-graph write --reachable --version=2 &&\n ++\tgit commit-graph verify\n ++'\n ++\n ++GRAPH_BYTE_CHUNK_COUNT=5\n ++GRAPH_BYTE_REACH_INDEX=6\n ++GRAPH_BYTE_UNUSED=7\n ++GRAPH_BYTE_HASH=8\n ++\n ++test_expect_success 'detect low chunk count (v2)' '\n ++\tcorrupt_graph_and_verify $GRAPH_BYTE_CHUNK_COUNT \"\\02\" \\\n ++\t\t\"missing the .* chunk\"\n ++'\n ++\n ++test_expect_success 'detect incorrect reachability index' '\n ++\tcorrupt_graph_and_verify $GRAPH_BYTE_REACH_INDEX \"\\03\" \\\n ++\t\t\"reachability index version\"\n ++'\n ++\n ++test_expect_success 'detect non-zero unused byte' '\n ++\tcorrupt_graph_and_verify $GRAPH_BYTE_UNUSED \"\\01\" \\\n ++\t\t\"unsupported value\"\n ++'\n ++\n ++test_expect_success 'detect bad hash version (v2)' '\n ++\tcorrupt_graph_and_verify $GRAPH_BYTE_HASH \"\\00\" \\\n ++\t\t\"hash algorithm\"\n ++'\n ++\n + test_expect_success 'setup non-the_repository tests' '\n + \trm -rf repo &&\n + \tgit init repo &&\n 6: 693900b4c5 < -: ---------- commit-graph: test verifying a corrupt v2 header\n\n-- \ngitgitgadget\n"},{"id":"374419","messageId":"91f300ec0acd825494fa13d93cdc4770bbf2ac14.1556135881.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 1/5] commit-graph: return with errors during write","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-04-24T19:58:03Z","receivedAt":"2019-04-24T19:58:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method uses die() to report failure and\nexit when confronted with an unexpected condition. This use of\ndie() in a library function is incorrect and is now replaced by\nerror() statements and an int return type.\n\nNow that we use 'goto cleanup' to jump to the terminal condition\non an error, we have new paths that could lead to uninitialized\nvalues. New initializers are added to correct for this.\n\nThe builtins 'commit-graph', 'gc', and 'commit' call these methods,\nso update them to check the return value.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 19 +++++++------\n builtin/commit.c | 5 ++--\n builtin/gc.c | 7 ++---\n commit-graph.c | 60 +++++++++++++++++++++++++++++-------------\n commit-graph.h | 10 +++----\n 5 files changed, 62 insertions(+), 39 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 537fdfd0f0..2e86251f02 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -141,6 +141,7 @@ static int graph_write(int argc, const char **argv)\n \tstruct string_list *pack_indexes = NULL;\n \tstruct string_list *commit_hex = NULL;\n \tstruct string_list lines;\n+\tint result;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n@@ -168,10 +169,8 @@ static int graph_write(int argc, const char **argv)\n \n \tread_replace_refs = 0;\n \n-\tif (opts.reachable) {\n-\t\twrite_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n-\t\treturn 0;\n-\t}\n+\tif (opts.reachable)\n+\t\treturn write_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n \n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n@@ -188,14 +187,14 @@ static int graph_write(int argc, const char **argv)\n \t\tUNLEAK(buf);\n \t}\n \n-\twrite_commit_graph(opts.obj_dir,\n-\t\t\t pack_indexes,\n-\t\t\t commit_hex,\n-\t\t\t opts.append,\n-\t\t\t 1);\n+\tresult = write_commit_graph(opts.obj_dir,\n+\t\t\t\t pack_indexes,\n+\t\t\t\t commit_hex,\n+\t\t\t\t opts.append,\n+\t\t\t\t 1);\n \n \tUNLEAK(lines);\n-\treturn 0;\n+\treturn result;\n }\n \n int cmd_commit_graph(int argc, const char **argv, const char *prefix)\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 2986553d5f..b9ea7222fa 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1669,8 +1669,9 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\t \"new_index file. Check that disk is not full and quota is\\n\"\n \t\t \"not exceeded, and then \\\"git reset HEAD\\\" to recover.\"));\n \n-\tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0))\n-\t\twrite_commit_graph_reachable(get_object_directory(), 0, 0);\n+\tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n+\t write_commit_graph_reachable(get_object_directory(), 0, 0))\n+\t\treturn 1;\n \n \trepo_rerere(the_repository, 0);\n \trun_command_v_opt(argv_gc_auto, RUN_GIT_CMD);\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 020f725acc..3984addf73 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -664,9 +664,10 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tclean_pack_garbage();\n \t}\n \n-\tif (gc_write_commit_graph)\n-\t\twrite_commit_graph_reachable(get_object_directory(), 0,\n-\t\t\t\t\t !quiet && !daemonized);\n+\tif (gc_write_commit_graph &&\n+\t write_commit_graph_reachable(get_object_directory(), 0,\n+\t\t\t\t\t !quiet && !daemonized))\n+\t\treturn 1;\n \n \tif (auto_gc && too_many_loose_objects())\n \t\twarning(_(\"There are too many unreachable loose objects; \"\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 66865acbd7..ee487a364b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -851,27 +851,30 @@ static int add_ref_to_list(const char *refname,\n \treturn 0;\n }\n \n-void write_commit_graph_reachable(const char *obj_dir, int append,\n-\t\t\t\t int report_progress)\n+int write_commit_graph_reachable(const char *obj_dir, int append,\n+\t\t\t\t int report_progress)\n {\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n+\tint result;\n \n \tfor_each_ref(add_ref_to_list, &list);\n-\twrite_commit_graph(obj_dir, NULL, &list, append, report_progress);\n+\tresult = write_commit_graph(obj_dir, NULL, &list,\n+\t\t\t\t append, report_progress);\n \n \tstring_list_clear(&list, 0);\n+\treturn result;\n }\n \n-void write_commit_graph(const char *obj_dir,\n-\t\t\tstruct string_list *pack_indexes,\n-\t\t\tstruct string_list *commit_hex,\n-\t\t\tint append, int report_progress)\n+int write_commit_graph(const char *obj_dir,\n+\t\t struct string_list *pack_indexes,\n+\t\t struct string_list *commit_hex,\n+\t\t int append, int report_progress)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n \tstruct hashfile *f;\n \tuint32_t i, count_distinct = 0;\n-\tchar *graph_name;\n+\tchar *graph_name = NULL;\n \tstruct lock_file lk = LOCK_INIT;\n \tuint32_t chunk_ids[5];\n \tuint64_t chunk_offsets[5];\n@@ -883,15 +886,17 @@ void write_commit_graph(const char *obj_dir,\n \tuint64_t progress_cnt = 0;\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tunsigned long approx_nr_objects;\n+\tint res = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n-\t\treturn;\n+\t\treturn 0;\n \n \toids.nr = 0;\n \tapprox_nr_objects = approximate_object_count();\n \toids.alloc = approx_nr_objects / 32;\n \toids.progress = NULL;\n \toids.progress_done = 0;\n+\tcommits.list = NULL;\n \n \tif (append) {\n \t\tprepare_commit_graph_one(the_repository, obj_dir);\n@@ -932,10 +937,16 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tstrbuf_setlen(&packname, dirlen);\n \t\t\tstrbuf_addstr(&packname, pack_indexes->items[i].string);\n \t\t\tp = add_packed_git(packname.buf, packname.len, 1);\n-\t\t\tif (!p)\n-\t\t\t\tdie(_(\"error adding pack %s\"), packname.buf);\n-\t\t\tif (open_pack_index(p))\n-\t\t\t\tdie(_(\"error opening index for %s\"), packname.buf);\n+\t\t\tif (!p) {\n+\t\t\t\terror(_(\"error adding pack %s\"), packname.buf);\n+\t\t\t\tres = 1;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\t\t\tif (open_pack_index(p)) {\n+\t\t\t\terror(_(\"error opening index for %s\"), packname.buf);\n+\t\t\t\tres = 1;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \t\t\tfor_each_object_in_pack(p, add_packed_commits, &oids,\n \t\t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n \t\t\tclose_pack(p);\n@@ -1006,8 +1017,11 @@ void write_commit_graph(const char *obj_dir,\n \t}\n \tstop_progress(&progress);\n \n-\tif (count_distinct >= GRAPH_EDGE_LAST_MASK)\n-\t\tdie(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n+\tif (count_distinct >= GRAPH_EDGE_LAST_MASK) {\n+\t\terror(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n+\t\tres = 1;\n+\t\tgoto cleanup;\n+\t}\n \n \tcommits.nr = 0;\n \tcommits.alloc = count_distinct;\n@@ -1039,16 +1053,21 @@ void write_commit_graph(const char *obj_dir,\n \tnum_chunks = num_extra_edges ? 4 : 3;\n \tstop_progress(&progress);\n \n-\tif (commits.nr >= GRAPH_EDGE_LAST_MASK)\n-\t\tdie(_(\"too many commits to write graph\"));\n+\tif (commits.nr >= GRAPH_EDGE_LAST_MASK) {\n+\t\terror(_(\"too many commits to write graph\"));\n+\t\tres = 1;\n+\t\tgoto cleanup;\n+\t}\n \n \tcompute_generation_numbers(&commits, report_progress);\n \n \tgraph_name = get_commit_graph_filename(obj_dir);\n \tif (safe_create_leading_directories(graph_name)) {\n \t\tUNLEAK(graph_name);\n-\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n-\t\t\t graph_name);\n+\t\terror(_(\"unable to create leading directories of %s\"),\n+\t\t\tgraph_name);\n+\t\tres = errno;\n+\t\tgoto cleanup;\n \t}\n \n \thold_lock_file_for_update(&lk, graph_name, LOCK_DIE_ON_ERROR);\n@@ -1107,9 +1126,12 @@ void write_commit_graph(const char *obj_dir,\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n \tcommit_lock_file(&lk);\n \n+cleanup:\n \tfree(graph_name);\n \tfree(commits.list);\n \tfree(oids.list);\n+\n+\treturn res;\n }\n \n #define VERIFY_COMMIT_GRAPH_ERROR_HASH 2\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 7dfb8c896f..d15670bf46 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -65,12 +65,12 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n */\n int generation_numbers_enabled(struct repository *r);\n \n-void write_commit_graph_reachable(const char *obj_dir, int append,\n+int write_commit_graph_reachable(const char *obj_dir, int append,\n \t\t\t\t int report_progress);\n-void write_commit_graph(const char *obj_dir,\n-\t\t\tstruct string_list *pack_indexes,\n-\t\t\tstruct string_list *commit_hex,\n-\t\t\tint append, int report_progress);\n+int write_commit_graph(const char *obj_dir,\n+\t\t struct string_list *pack_indexes,\n+\t\t struct string_list *commit_hex,\n+\t\t int append, int report_progress);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \n-- \ngitgitgadget\n\n"},{"id":"374421","messageId":"04f5df11351ec117a3a019c9186df38c19151d16.1556135881.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 2/5] commit-graph: collapse parameters into flags","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-04-24T19:58:04Z","receivedAt":"2019-04-24T19:58:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() and write_commit_graph_reachable() methods\ncurrently take two boolean parameters: 'append' and 'report_progress'.\nWe will soon expand the possible options to send to these methods, so\ninstead of complicating the parameter list, first simplify it.\n\nCollapse these parameters into a 'flags' parameter, and adjust the\ncallers to provide flags as necessary.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 8 +++++---\n builtin/commit.c | 2 +-\n builtin/gc.c | 4 ++--\n commit-graph.c | 9 +++++----\n commit-graph.h | 8 +++++---\n 5 files changed, 18 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 2e86251f02..828b1a713f 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -142,6 +142,7 @@ static int graph_write(int argc, const char **argv)\n \tstruct string_list *commit_hex = NULL;\n \tstruct string_list lines;\n \tint result;\n+\tint flags = COMMIT_GRAPH_PROGRESS;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n@@ -166,11 +167,13 @@ static int graph_write(int argc, const char **argv)\n \t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n+\tif (opts.append)\n+\t\tflags |= COMMIT_GRAPH_APPEND;\n \n \tread_replace_refs = 0;\n \n \tif (opts.reachable)\n-\t\treturn write_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n+\t\treturn write_commit_graph_reachable(opts.obj_dir, flags);\n \n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n@@ -190,8 +193,7 @@ static int graph_write(int argc, const char **argv)\n \tresult = write_commit_graph(opts.obj_dir,\n \t\t\t\t pack_indexes,\n \t\t\t\t commit_hex,\n-\t\t\t\t opts.append,\n-\t\t\t\t 1);\n+\t\t\t\t flags);\n \n \tUNLEAK(lines);\n \treturn result;\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex b9ea7222fa..b001ef565d 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1670,7 +1670,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\t \"not exceeded, and then \\\"git reset HEAD\\\" to recover.\"));\n \n \tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n-\t write_commit_graph_reachable(get_object_directory(), 0, 0))\n+\t write_commit_graph_reachable(get_object_directory(), 0))\n \t\treturn 1;\n \n \trepo_rerere(the_repository, 0);\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 3984addf73..df2573f124 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -665,8 +665,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (gc_write_commit_graph &&\n-\t write_commit_graph_reachable(get_object_directory(), 0,\n-\t\t\t\t\t !quiet && !daemonized))\n+\t write_commit_graph_reachable(get_object_directory(),\n+\t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_PROGRESS : 0))\n \t\treturn 1;\n \n \tif (auto_gc && too_many_loose_objects())\ndiff --git a/commit-graph.c b/commit-graph.c\nindex ee487a364b..b16c71fd82 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -851,15 +851,14 @@ static int add_ref_to_list(const char *refname,\n \treturn 0;\n }\n \n-int write_commit_graph_reachable(const char *obj_dir, int append,\n-\t\t\t\t int report_progress)\n+int write_commit_graph_reachable(const char *obj_dir, int flags)\n {\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n \tint result;\n \n \tfor_each_ref(add_ref_to_list, &list);\n \tresult = write_commit_graph(obj_dir, NULL, &list,\n-\t\t\t\t append, report_progress);\n+\t\t\t\t flags);\n \n \tstring_list_clear(&list, 0);\n \treturn result;\n@@ -868,7 +867,7 @@ int write_commit_graph_reachable(const char *obj_dir, int append,\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t int append, int report_progress)\n+\t\t int flags)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n@@ -887,6 +886,8 @@ int write_commit_graph(const char *obj_dir,\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tunsigned long approx_nr_objects;\n \tint res = 0;\n+\tint append = flags & COMMIT_GRAPH_APPEND;\n+\tint report_progress = flags & COMMIT_GRAPH_PROGRESS;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn 0;\ndiff --git a/commit-graph.h b/commit-graph.h\nindex d15670bf46..390474047c 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -65,12 +65,14 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n */\n int generation_numbers_enabled(struct repository *r);\n \n-int write_commit_graph_reachable(const char *obj_dir, int append,\n-\t\t\t\t int report_progress);\n+#define COMMIT_GRAPH_APPEND (1 << 0)\n+#define COMMIT_GRAPH_PROGRESS (1 << 1)\n+\n+int write_commit_graph_reachable(const char *obj_dir, int flags);\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t int append, int report_progress);\n+\t\t int flags);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \n-- \ngitgitgadget\n\n"},{"id":"374420","messageId":"4ddb82916321f093d88282ef4bcab99993339fb5.1556135881.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 3/5] commit-graph: create new version flags","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-04-24T19:58:05Z","receivedAt":"2019-04-24T19:58:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn anticipation of a new commit-graph file format version, create\na flag for the write_commit_graph() and write_commit_graph_reachable()\nmethods to take a version number.\n\nWhen there is no specified version, the implementation selects a\ndefault value. Currently, the only valid value is 1.\n\nThe file format will change the header information, so place the\nexisting header logic inside a switch statement with only one case.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 58 +++++++++++++++++++++++++++++++++-----------------\n commit-graph.h | 1 +\n 2 files changed, 40 insertions(+), 19 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex b16c71fd82..e75e1655fb 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -25,9 +25,6 @@\n \n #define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n \n-#define GRAPH_VERSION_1 0x1\n-#define GRAPH_VERSION GRAPH_VERSION_1\n-\n #define GRAPH_EXTRA_EDGES_NEEDED 0x80000000\n #define GRAPH_EDGE_LAST_MASK 0x7fffffff\n #define GRAPH_PARENT_NONE 0x70000000\n@@ -173,30 +170,35 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t}\n \n \tgraph_version = *(unsigned char*)(data + 4);\n-\tif (graph_version != GRAPH_VERSION) {\n+\tif (graph_version != 1) {\n \t\terror(_(\"commit-graph version %X does not match version %X\"),\n-\t\t graph_version, GRAPH_VERSION);\n-\t\treturn NULL;\n-\t}\n-\n-\thash_version = *(unsigned char*)(data + 5);\n-\tif (hash_version != oid_version()) {\n-\t\terror(_(\"commit-graph hash version %X does not match version %X\"),\n-\t\t hash_version, oid_version());\n+\t\t graph_version, 1);\n \t\treturn NULL;\n \t}\n \n \tgraph = alloc_commit_graph();\n \n+\tswitch (graph_version) {\n+\tcase 1:\n+\t\thash_version = *(unsigned char*)(data + 5);\n+\t\tif (hash_version != oid_version()) {\n+\t\t\terror(_(\"commit-graph hash version %X does not match version %X\"),\n+\t\t\t hash_version, oid_version());\n+\t\t\treturn NULL;\n+\t\t}\n+\n+\t\tgraph->num_chunks = *(unsigned char*)(data + 6);\n+\t\tchunk_lookup = data + 8;\n+\t\tbreak;\n+\t}\n+\n \tgraph->hash_len = the_hash_algo->rawsz;\n-\tgraph->num_chunks = *(unsigned char*)(data + 6);\n \tgraph->graph_fd = fd;\n \tgraph->data = graph_map;\n \tgraph->data_len = graph_size;\n \n \tlast_chunk_id = 0;\n \tlast_chunk_offset = 8;\n-\tchunk_lookup = data + 8;\n \tfor (i = 0; i < graph->num_chunks; i++) {\n \t\tuint32_t chunk_id;\n \t\tuint64_t chunk_offset;\n@@ -888,10 +890,22 @@ int write_commit_graph(const char *obj_dir,\n \tint res = 0;\n \tint append = flags & COMMIT_GRAPH_APPEND;\n \tint report_progress = flags & COMMIT_GRAPH_PROGRESS;\n+\tint version = 0;\n+\tint header_size = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn 0;\n \n+\tif (flags & COMMIT_GRAPH_VERSION_1)\n+\t\tversion = 1;\n+\tif (!version)\n+\t\tversion = 1;\n+\tif (version != 1) {\n+\t\terror(_(\"unsupported commit-graph version %d\"),\n+\t\t version);\n+\t\treturn 1;\n+\t}\n+\n \toids.nr = 0;\n \tapprox_nr_objects = approximate_object_count();\n \toids.alloc = approx_nr_objects / 32;\n@@ -1076,10 +1090,16 @@ int write_commit_graph(const char *obj_dir,\n \n \thashwrite_be32(f, GRAPH_SIGNATURE);\n \n-\thashwrite_u8(f, GRAPH_VERSION);\n-\thashwrite_u8(f, oid_version());\n-\thashwrite_u8(f, num_chunks);\n-\thashwrite_u8(f, 0); /* unused padding byte */\n+\thashwrite_u8(f, version);\n+\n+\tswitch (version) {\n+\tcase 1:\n+\t\thashwrite_u8(f, oid_version());\n+\t\thashwrite_u8(f, num_chunks);\n+\t\thashwrite_u8(f, 0); /* unused padding byte */\n+\t\theader_size = 8;\n+\t\tbreak;\n+\t}\n \n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n@@ -1090,7 +1110,7 @@ int write_commit_graph(const char *obj_dir,\n \t\tchunk_ids[3] = 0;\n \tchunk_ids[4] = 0;\n \n-\tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n+\tchunk_offsets[0] = header_size + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n \tchunk_offsets[2] = chunk_offsets[1] + hashsz * commits.nr;\n \tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * commits.nr;\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 390474047c..d7cd13deb3 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -67,6 +67,7 @@ int generation_numbers_enabled(struct repository *r);\n \n #define COMMIT_GRAPH_APPEND (1 << 0)\n #define COMMIT_GRAPH_PROGRESS (1 << 1)\n+#define COMMIT_GRAPH_VERSION_1 (1 << 2)\n \n int write_commit_graph_reachable(const char *obj_dir, int flags);\n int write_commit_graph(const char *obj_dir,\n-- \ngitgitgadget\n\n"},{"id":"374422","messageId":"b1b0c76eb494307a289caf63399a3450926bab9a.1556135881.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 4/5] commit-graph: add --version= option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-04-24T19:58:06Z","receivedAt":"2019-04-24T19:58:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAllow the commit-graph builtin to specify the file format version\nusing the '--version=' option. Specify the version exactly in\nthe verification tests as using a different version would change\nthe offsets used in those tests.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 3 +++\n builtin/commit-graph.c | 13 +++++++++++--\n t/t5318-commit-graph.sh | 2 +-\n 3 files changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 624470e198..1d1cc70de4 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -51,6 +51,9 @@ or `--stdin-packs`.)\n +\n With the `--append` option, include all commits that are present in the\n existing commit-graph file.\n++\n+With the `--version=` option, specify the file format version. Used\n+only for testing.\n \n 'read'::\n \ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 828b1a713f..65ceb7a141 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -10,7 +10,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph verify [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits] [--version=]\"),\n \tNULL\n };\n \n@@ -25,7 +25,7 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits] [--version=]\"),\n \tNULL\n };\n \n@@ -35,6 +35,7 @@ static struct opts_commit_graph {\n \tint stdin_packs;\n \tint stdin_commits;\n \tint append;\n+\tint version;\n } opts;\n \n \n@@ -156,6 +157,8 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"start walk at commits listed by stdin\")),\n \t\tOPT_BOOL(0, \"append\", &opts.append,\n \t\t\tN_(\"include all commits already in the commit-graph file\")),\n+\t\tOPT_INTEGER(0, \"version\", &opts.version,\n+\t\t\tN_(\"specify the file format version\")),\n \t\tOPT_END(),\n \t};\n \n@@ -170,6 +173,12 @@ static int graph_write(int argc, const char **argv)\n \tif (opts.append)\n \t\tflags |= COMMIT_GRAPH_APPEND;\n \n+\tswitch (opts.version) {\n+\tcase 1:\n+\t\tflags |= COMMIT_GRAPH_VERSION_1;\n+\t\tbreak;\n+\t}\n+\n \tread_replace_refs = 0;\n \n \tif (opts.reachable)\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex e80c1cac02..4eb5a09ef3 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -328,7 +328,7 @@ test_expect_success 'replace-objects invalidates commit-graph' '\n \n test_expect_success 'git commit-graph verify' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n-\tgit rev-parse commits/8 | git commit-graph write --stdin-commits &&\n+\tgit rev-parse commits/8 | git commit-graph write --stdin-commits --version=1 &&\n \tgit commit-graph verify >output\n '\n \n-- \ngitgitgadget\n\n"},{"id":"374423","messageId":"09362bda1b6c2e7f3292b634b9ed380fe2a5ccd5.1556135881.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 5/5] commit-graph: implement file format version 2","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-04-24T19:58:07Z","receivedAt":"2019-04-24T19:58:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe commit-graph file format had some shortcomings which we now\ncorrect:\n\n 1. The hash algorithm was determined by a single byte, instead\n of the 4-byte format identifier.\n\n 2. There was no way to update the reachability index we used.\n We currently only support generation numbers, but that will\n change in the future.\n\n 3. Git did not fail with error if the unused eighth byte was\n non-zero, so we could not use that to indicate an incremental\n file format without breaking compatibility across versions.\n\nThe new format modifies the header of the commit-graph to solve\nthese problems. We use the 4-byte hash format id, freeing up a byte\nin our 32-bit alignment to introduce a reachability index version.\nWe can also fail to read the commit-graph if the eighth byte is\nnon-zero.\n\nUpdate the 'git commit-graph read' subcommand to display the new\ndata.\n\nSet the default file format version to 2, and adjust the tests to\nexpect the new 'git commit-graph read' output.\n\nAdd explicit tests for the upgrade path from version 1 to 2. Users\nwith an existing commit-graph with version 1 will seamlessly\nupgrade to version 2 on their next write.\n\nWhile we converted the existing 'verify' tests to use a version 1\nfile to avoid recalculating data offsets, add explicit 'verify'\ntests on a version 2 file that corrupt the new header values.\n\nSigned-off-by: Derrick Stolee \n---\n .../technical/commit-graph-format.txt | 26 +++++++-\n builtin/commit-graph.c | 9 +++\n commit-graph.c | 43 ++++++++++--\n commit-graph.h | 1 +\n t/t5318-commit-graph.sh | 66 +++++++++++++++++--\n 5 files changed, 134 insertions(+), 11 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex 16452a0504..e367aa94b1 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -31,13 +31,22 @@ and hash type.\n \n All 4-byte numbers are in network order.\n \n+There are two versions available, 1 and 2. These currently differ only in\n+the header.\n+\n HEADER:\n \n+All commit-graph files use the first five bytes for the same purpose.\n+\n 4-byte signature:\n The signature is: {'C', 'G', 'P', 'H'}\n \n 1-byte version number:\n- Currently, the only valid version is 1.\n+ Currently, the valid version numbers are 1 and 2.\n+\n+The remainder of the header changes depending on the version.\n+\n+Version 1:\n \n 1-byte Hash Version (1 = SHA-1)\n We infer the hash length (H) from this value.\n@@ -47,6 +56,21 @@ HEADER:\n 1-byte (reserved for later use)\n Current clients should ignore this value.\n \n+Version 2:\n+\n+ 1-byte number (C) of \"chunks\"\n+\n+ 1-byte reachability index version number:\n+ Currently, the only valid number is 1.\n+\n+ 1-byte (reserved for later use)\n+ Current clients expect this value to be zero, and will not\n+ try to read the commit-graph file if it is non-zero.\n+\n+ 4-byte format identifier for the hash algorithm:\n+ If this identifier does not agree with the repository's current\n+ hash algorithm, then the client will not read the commit graph.\n+\n CHUNK LOOKUP:\n \n (C + 1) * 12 bytes listing the table of contents for the chunks:\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 65ceb7a141..1485b4daaf 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -117,6 +117,11 @@ static int graph_read(int argc, const char **argv)\n \t\t*(unsigned char*)(graph->data + 5),\n \t\t*(unsigned char*)(graph->data + 6),\n \t\t*(unsigned char*)(graph->data + 7));\n+\n+\tif (*(unsigned char *)(graph->data + 4) == 2)\n+\t\tprintf(\"hash algorithm: %X\\n\",\n+\t\t get_be32(graph->data + 8));\n+\n \tprintf(\"num_commits: %u\\n\", graph->num_commits);\n \tprintf(\"chunks:\");\n \n@@ -177,6 +182,10 @@ static int graph_write(int argc, const char **argv)\n \tcase 1:\n \t\tflags |= COMMIT_GRAPH_VERSION_1;\n \t\tbreak;\n+\n+\tcase 2:\n+\t\tflags |= COMMIT_GRAPH_VERSION_2;\n+\t\tbreak;\n \t}\n \n \tread_replace_refs = 0;\ndiff --git a/commit-graph.c b/commit-graph.c\nindex e75e1655fb..14d6aebd99 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -152,7 +152,8 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \tuint64_t last_chunk_offset;\n \tuint32_t last_chunk_id;\n \tuint32_t graph_signature;\n-\tunsigned char graph_version, hash_version;\n+\tunsigned char graph_version, hash_version, reach_index_version;\n+\tuint32_t hash_id;\n \n \tif (!graph_map)\n \t\treturn NULL;\n@@ -170,7 +171,7 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t}\n \n \tgraph_version = *(unsigned char*)(data + 4);\n-\tif (graph_version != 1) {\n+\tif (!graph_version || graph_version > 2) {\n \t\terror(_(\"commit-graph version %X does not match version %X\"),\n \t\t graph_version, 1);\n \t\treturn NULL;\n@@ -190,6 +191,30 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t\tgraph->num_chunks = *(unsigned char*)(data + 6);\n \t\tchunk_lookup = data + 8;\n \t\tbreak;\n+\n+\tcase 2:\n+\t\tgraph->num_chunks = *(unsigned char *)(data + 5);\n+\n+\t\treach_index_version = *(unsigned char *)(data + 6);\n+\t\tif (reach_index_version != 1) {\n+\t\t\terror(_(\"unsupported reachability index version %d\"),\n+\t\t\t reach_index_version);\n+\t\t\treturn NULL;\n+\t\t}\n+\n+\t\tif (*(unsigned char*)(data + 7)) {\n+\t\t\terror(_(\"unsupported value in commit-graph header\"));\n+\t\t\treturn NULL;\n+\t\t}\n+\n+\t\thash_id = get_be32(data + 8);\n+\t\tif (hash_id != the_hash_algo->format_id) {\n+\t\t\terror(_(\"commit-graph hash algorithm does not match current algorithm\"));\n+\t\t\treturn NULL;\n+\t\t}\n+\n+\t\tchunk_lookup = data + 12;\n+\t\tbreak;\n \t}\n \n \tgraph->hash_len = the_hash_algo->rawsz;\n@@ -898,9 +923,11 @@ int write_commit_graph(const char *obj_dir,\n \n \tif (flags & COMMIT_GRAPH_VERSION_1)\n \t\tversion = 1;\n+\tif (flags & COMMIT_GRAPH_VERSION_2)\n+\t\tversion = 2;\n \tif (!version)\n-\t\tversion = 1;\n-\tif (version != 1) {\n+\t\tversion = 2;\n+\tif (version <= 0 || version > 2) {\n \t\terror(_(\"unsupported commit-graph version %d\"),\n \t\t version);\n \t\treturn 1;\n@@ -1099,6 +1126,14 @@ int write_commit_graph(const char *obj_dir,\n \t\thashwrite_u8(f, 0); /* unused padding byte */\n \t\theader_size = 8;\n \t\tbreak;\n+\n+\tcase 2:\n+\t\thashwrite_u8(f, num_chunks);\n+\t\thashwrite_u8(f, 1); /* reachability index version */\n+\t\thashwrite_u8(f, 0); /* unused padding byte */\n+\t\thashwrite_be32(f, the_hash_algo->format_id);\n+\t\theader_size = 12;\n+\t\tbreak;\n \t}\n \n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\ndiff --git a/commit-graph.h b/commit-graph.h\nindex d7cd13deb3..2c461770e8 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -68,6 +68,7 @@ int generation_numbers_enabled(struct repository *r);\n #define COMMIT_GRAPH_APPEND (1 << 0)\n #define COMMIT_GRAPH_PROGRESS (1 << 1)\n #define COMMIT_GRAPH_VERSION_1 (1 << 2)\n+#define COMMIT_GRAPH_VERSION_2 (1 << 3)\n \n int write_commit_graph_reachable(const char *obj_dir, int flags);\n int write_commit_graph(const char *obj_dir,\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 4eb5a09ef3..0c766e7cdb 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -33,13 +33,13 @@ test_expect_success 'create commits and repack' '\n \tgit repack\n '\n \n-graph_git_two_modes() {\n+graph_git_two_modes () {\n \tgit -c core.commitGraph=true $1 >output\n \tgit -c core.commitGraph=false $1 >expect\n \ttest_cmp expect output\n }\n \n-graph_git_behavior() {\n+graph_git_behavior () {\n \tMSG=$1\n \tDIR=$2\n \tBRANCH=$3\n@@ -56,7 +56,7 @@ graph_git_behavior() {\n \n graph_git_behavior 'no graph' full commits/3 commits/1\n \n-graph_read_expect() {\n+graph_read_expect () {\n \tOPTIONAL=\"\"\n \tNUM_CHUNKS=3\n \tif test ! -z $2\n@@ -65,7 +65,8 @@ graph_read_expect() {\n \t\tNUM_CHUNKS=$((3 + $(echo \"$2\" | wc -w)))\n \tfi\n \tcat >expect <<- EOF\n-\theader: 43475048 1 1 $NUM_CHUNKS 0\n+\theader: 43475048 2 $NUM_CHUNKS 1 0\n+\thash algorithm: 73686131\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata$OPTIONAL\n \tEOF\n@@ -320,6 +321,24 @@ test_expect_success 'replace-objects invalidates commit-graph' '\n \t)\n '\n \n+test_expect_success 'write v1 graph' '\n+\tgit commit-graph write --reachable --version=1 &&\n+\tgit commit-graph verify\n+'\n+\n+graph_git_behavior 'version 1 graph, commit 8 vs merge 2' full commits/8 merge/2\n+graph_git_behavior 'version 1 graph, commit 8 vs merge 2' full commits/8 merge/2\n+\n+test_expect_success 'upgrade from v1 to v2' '\n+\tgit checkout -b new-commit-for-upgrade &&\n+\ttest_commit force-upgrade &&\n+\tgit commit-graph write --reachable --version=2 &&\n+\tgit commit-graph verify\n+'\n+\n+graph_git_behavior 'upgraded graph, commit 8 vs merge 2' full commits/8 merge/2\n+graph_git_behavior 'upgraded graph, commit 8 vs merge 2' full commits/8 merge/2\n+\n # the verify tests below expect the commit-graph to contain\n # exactly the commits reachable from the commits/8 branch.\n # If the file changes the set of commits in the list, then the\n@@ -392,7 +411,7 @@ corrupt_graph_verify() {\n # starting at , then runs 'git commit-graph verify'\n # and places the output in the file 'err'. Test 'err' for\n # the given string.\n-corrupt_graph_and_verify() {\n+corrupt_graph_and_verify () {\n \tpos=$1\n \tdata=\"${2:-\\0}\"\n \tgrepstr=$3\n@@ -424,10 +443,14 @@ test_expect_success 'detect bad signature' '\n '\n \n test_expect_success 'detect bad version' '\n-\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\02\" \\\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\03\" \\\n \t\t\"graph version\"\n '\n \n+test_expect_success 'detect version 2 with version 1 data' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\02\" \\\n+\t\t\"reachability index version\"\n+'\n test_expect_success 'detect bad hash version' '\n \tcorrupt_graph_and_verify $GRAPH_BYTE_HASH \"\\02\" \\\n \t\t\"hash version\"\n@@ -532,6 +555,37 @@ test_expect_success 'git fsck (checks commit-graph)' '\n \ttest_must_fail git fsck\n '\n \n+test_expect_success 'rewrite commmit-graph with version 2' '\n+\trm -f .git/objects/info/commit-graph &&\n+\tgit commit-graph write --reachable --version=2 &&\n+\tgit commit-graph verify\n+'\n+\n+GRAPH_BYTE_CHUNK_COUNT=5\n+GRAPH_BYTE_REACH_INDEX=6\n+GRAPH_BYTE_UNUSED=7\n+GRAPH_BYTE_HASH=8\n+\n+test_expect_success 'detect low chunk count (v2)' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_CHUNK_COUNT \"\\02\" \\\n+\t\t\"missing the .* chunk\"\n+'\n+\n+test_expect_success 'detect incorrect reachability index' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_REACH_INDEX \"\\03\" \\\n+\t\t\"reachability index version\"\n+'\n+\n+test_expect_success 'detect non-zero unused byte' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_UNUSED \"\\01\" \\\n+\t\t\"unsupported value\"\n+'\n+\n+test_expect_success 'detect bad hash version (v2)' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_HASH \"\\00\" \\\n+\t\t\"hash algorithm\"\n+'\n+\n test_expect_success 'setup non-the_repository tests' '\n \trm -rf repo &&\n \tgit init repo &&\n-- \ngitgitgadget\n"},{"id":"374455","messageId":"xmqqk1fiy80k.fsf@gitster-ct.c.googlers.com","threadId":"50302","inReplyTo":"04f5df11351ec117a3a019c9186df38c19151d16.1556135881.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/5] commit-graph: collapse parameters into flags","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-04-25T05:21:47Z","receivedAt":"2019-04-25T05:21:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> The write_commit_graph() and write_commit_graph_reachable() methods\n> currently take two boolean parameters: 'append' and 'report_progress'.\n> We will soon expand the possible options to send to these methods, so\n> instead of complicating the parameter list, first simplify it.\n>\n> Collapse these parameters into a 'flags' parameter, and adjust the\n> callers to provide flags as necessary.\n\nNice. It would make more sense for a collection of independent bits\nto be in an unsigned, not signed integer variable, though. Unless\nyou assign some special meaning to the top-most bit, that is.\n\n"},{"id":"374456","messageId":"xmqqftq6y7nv.fsf@gitster-ct.c.googlers.com","threadId":"50302","inReplyTo":"4ddb82916321f093d88282ef4bcab99993339fb5.1556135881.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/5] commit-graph: create new version flags","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-04-25T05:29:24Z","receivedAt":"2019-04-25T05:29:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> +\tint version = 0;\n> ...\n> +\tif (flags & COMMIT_GRAPH_VERSION_1)\n> +\t\tversion = 1;\n> +\tif (!version)\n> +\t\tversion = 1;\n> +\tif (version != 1) {\n> +\t\terror(_(\"unsupported commit-graph version %d\"),\n> +\t\t version);\n> +\t\treturn 1;\n> +\t}\n\nThe above sequence had a certain \"Huh?\" factor before 5/5 introduced\nthe support for a later version that is in use by default.\n\nIs it sensible to define VERSION_$N as if they are independent bits\nin a single flags variable? What does it mean for the flags variable\nto have both GRAPH_VERSION_1 and GRAPH_VERSION_2 bits set?\n\nWhat I am getting at is if this is better done as a n-bit bitfield\nthat represents a small unsigned integer (e.g. \"unsigned char\" that\nlets you play with up to 255 versions, or \"unsigned version : 3\"\nthat limits you to up to 7 versions).\n\nYou use an 8-bit byte in the file format anyway, so it might not be\nso bad to have a separate version parameter that is not mixed with\nthe flag bits, perhaps?\n"},{"id":"374494","messageId":"eb1ec575-601c-d50e-dfd1-beab1debcc96@gmail.com","threadId":"50302","inReplyTo":"xmqqftq6y7nv.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 3/5] commit-graph: create new version flags","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-04-25T11:09:09Z","receivedAt":"2019-04-25T11:09:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/25/2019 1:29 AM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> +\tint version = 0;\n>> ...\n>> +\tif (flags & COMMIT_GRAPH_VERSION_1)\n>> +\t\tversion = 1;\n>> +\tif (!version)\n>> +\t\tversion = 1;\n>> +\tif (version != 1) {\n>> +\t\terror(_(\"unsupported commit-graph version %d\"),\n>> +\t\t version);\n>> +\t\treturn 1;\n>> +\t}\n> \n> The above sequence had a certain \"Huh?\" factor before 5/5 introduced\n> the support for a later version that is in use by default.\n> \n> Is it sensible to define VERSION_$N as if they are independent bits\n> in a single flags variable? What does it mean for the flags variable\n> to have both GRAPH_VERSION_1 and GRAPH_VERSION_2 bits set?\n>\n> What I am getting at is if this is better done as a n-bit bitfield\n> that represents a small unsigned integer (e.g. \"unsigned char\" that\n> lets you play with up to 255 versions, or \"unsigned version : 3\"\n> that limits you to up to 7 versions).\n> \n> You use an 8-bit byte in the file format anyway, so it might not be\n> so bad to have a separate version parameter that is not mixed with\n> the flag bits, perhaps?\n\nThis is a reasonable idea, as this is a \"pick exactly one\" option.\nIt is still important to reduce the overall parameter count by combining\nthe other boolean options into flags.\n\nThanks,\n-Stolee\n \n"},{"id":"374530","messageId":"87bm0tsrf0.fsf@evledraar.gmail.com","threadId":"50302","inReplyTo":"4ddb82916321f093d88282ef4bcab99993339fb5.1556135881.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/5] commit-graph: create new version flags","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-04-25T21:31:31Z","receivedAt":"2019-04-25T21:31:36Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Apr 24 2019, Derrick Stolee via GitGitGadget wrote:\n\n> -\thash_version = *(unsigned char*)(data + 5);\n> -\tif (hash_version != oid_version()) {\n> -\t\terror(_(\"commit-graph hash version %X does not match version %X\"),\n> -\t\t hash_version, oid_version());\n> +\t\t graph_version, 1);\n> \t\treturn NULL;\n> \t}\n>\n> \tgraph = alloc_commit_graph();\n>\n> +\tswitch (graph_version) {\n> +\tcase 1:\n> +\t\thash_version = *(unsigned char*)(data + 5);\n> +\t\tif (hash_version != oid_version()) {\n> +\t\t\terror(_(\"commit-graph hash version %X does not match version %X\"),\n> +\t\t\t hash_version, oid_version());\n> +\t\t\treturn NULL;\n> +\t\t}\n\nThis is just munging existing code, but one thing in my series that I\ndidn't follow-up on was Duy's suggestion[1] of %X here being\nnonsensical.\n\nIt doesn't make sense to start saying \"version A\" here when we make it\nto version 10, however unlikely that is :)\n\nSo I think for the existing %X in this file it should be 0x%X as he\nsuggests, except in cases like this where we should just use %d.\n\n1. https://public-inbox.org/git/CACsJy8DgNzGK3g2P7ZyRmd7sbiSOXY07KqYEh-gSsPkEZ+D5Qw@mail.gmail.com/\n"},{"id":"374531","messageId":"87a7gdspo4.fsf@evledraar.gmail.com","threadId":"50302","inReplyTo":"pull.112.v2.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 0/5] Create commit-graph file format v2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-04-25T22:09:15Z","receivedAt":"2019-04-25T22:09:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Apr 24 2019, Derrick Stolee via GitGitGadget wrote:\n\n> NOTE: this series was rebased onto ab/commit-graph-fixes, as the conflicts\n> were significant and subtle.\n\nSorry, hopefully it helped more than it harmed :)\n\nA few unrelated things:\n\n1)\n\nFirst, before that series of mine applying this and writing a v2 file\nwould make most things (e.g. \"status\") hard error on e.g. v2.21.0:\n\n\n $ git status\n error: graph version 2 does not match version 1\n $\n\nNow as noted in my series we now on 'master' downgrade that to a warning\n(along with the rest of the errors):\n\n $ ~/g/git/git --exec-path=$PWD status\n error: commit-graph version 2 does not match version 1\n On branch master\n [...]\n\n...and this series sets the default version for all new graphs to v2.\n\nI think this is *way* too aggressive of an upgrade path. If these\npatches go into v2.22.0 then git clients on all older versions that grok\nthe commit graph (IIRC v2.18 and above) will have their git completely\nbroken if they're in a mixed-git-version environment.\n\nIs it really so important to move to v2 right away that we need to risk\nthose breakages? I think even with my ab/commit-graph-fixes it's still\ntoo annoying (I was mostly trying to fix other stuff...). If only we\ncould detect \"we should make a new graph now\" ....\n\n2)\n\n...speaking of which, digging up outstanding stuff I have on the\ncommit-graph I was reminded to finish up my \"commit graph on clone\"\npatch in:\nhttps://public-inbox.org/git/87in2hgzin.fsf@evledraar.gmail.com/\n\nAnd re #1 above: I guess we could also do that \"let's make a graph\" and\ncall \"gc --auto\" if a) we have gc.writeCommitGraph b) we see it's not\nthe \"right\" version. As long as older versions always write a \"old\" one\nif they can't grok the \"new\" one, and newer versions leave existing\ngraphs alone even if they're older versions, so we don't flip-flop.\n\nOne of the things that would make that \"graph on clone/fetch/whatever\"\neasier is having the graph store the total number of objects while it\nwas at it, you indicated in\nhttps://public-inbox.org/git/934fa00e-f6df-c333-4968-3e9acffab22d@gmail.com/\nthat you already have an internal MSFT implementation of it that does\nit.\n\nAny reason not to make it part of v2 while we're at it? We already find\nout how many (packed) objects we have in \"add_packed_commits\", we just\ndon't do anything with that information now.\n\n3)\n\nAlso (but mostly unrelated). I see that \"Future Work\" in\nDocumentation/technical/commit-graph.txt now appears to entirely\ndescribe \"Past Work\" instead :)\n\n4)\n\nThe third point in \"Design Details\" also says that the format doesn't\nneed a change for a future hash algo change, yet here we are at v2\nmaking a (small) change for that purpose :)\n"},{"id":"374536","messageId":"xmqqimv1wlqr.fsf@gitster-ct.c.googlers.com","threadId":"50302","inReplyTo":"87bm0tsrf0.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 3/5] commit-graph: create new version flags","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-04-26T02:20:28Z","receivedAt":"2019-04-26T02:20:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> It doesn't make sense to start saying \"version A\" here when we make it\n> to version 10, however unlikely that is :)\n\n;-)\n"},{"id":"374537","messageId":"xmqqef5pwldc.fsf@gitster-ct.c.googlers.com","threadId":"50302","inReplyTo":"87a7gdspo4.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 0/5] Create commit-graph file format v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-04-26T02:28:31Z","receivedAt":"2019-04-26T02:28:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> On Wed, Apr 24 2019, Derrick Stolee via GitGitGadget wrote:\n>\n>> NOTE: this series was rebased onto ab/commit-graph-fixes, as the conflicts\n>> were significant and subtle.\n>\n> Sorry, hopefully it helped more than it harmed :)\n>\n> A few unrelated things:\n\nThanks always for your careful review and thoughtful comments, by\nthe way.\n\n> Now as noted in my series we now on 'master' downgrade that to a warning\n> (along with the rest of the errors):\n>\n> $ ~/g/git/git --exec-path=$PWD status\n> error: commit-graph version 2 does not match version 1\n> On branch master\n> [...]\n>\n> ...and this series sets the default version for all new graphs to v2.\n\nThe phrasing seems odd. It is unclear, even to me who is vaguely\nfamiliar with the word \"commit-graph\" and is aware of the fact that\nthe file format is being updated, what\n\n \"commit-graph version 2 does not match version 1\" \n\nwants to say. Do I have version #2 on disk and the running binary\nonly understands version #1? Or do I have version #1 on disk and\nthe binary expected version #2? How would I get out of this\nsituation? Is it sufficient to do \"rm -f .git/info/commit-graph*\"\nand is it safe?\n\n> I think this is *way* too aggressive of an upgrade path. If these\n> patches go into v2.22.0 then git clients on all older versions that grok\n> the commit graph (IIRC v2.18 and above) will have their git completely\n> broken if they're in a mixed-git-version environment.\n>\n> Is it really so important to move to v2 right away that we need to risk\n> those breakages? I think even with my ab/commit-graph-fixes it's still\n> too annoying (I was mostly trying to fix other stuff...). If only we\n> could detect \"we should make a new graph now\" ....\n\nTrue.\n"},{"id":"374549","messageId":"878svxrwsh.fsf@evledraar.gmail.com","threadId":"50302","inReplyTo":"xmqqef5pwldc.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 0/5] Create commit-graph file format v2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-04-26T08:33:02Z","receivedAt":"2019-04-26T08:33:09Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Apr 26 2019, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>> On Wed, Apr 24 2019, Derrick Stolee via GitGitGadget wrote:\n>>\n>>> NOTE: this series was rebased onto ab/commit-graph-fixes, as the conflicts\n>>> were significant and subtle.\n>>\n>> Sorry, hopefully it helped more than it harmed :)\n>>\n>> A few unrelated things:\n>\n> Thanks always for your careful review and thoughtful comments, by\n> the way.\n>\n>> Now as noted in my series we now on 'master' downgrade that to a warning\n>> (along with the rest of the errors):\n>>\n>> $ ~/g/git/git --exec-path=$PWD status\n>> error: commit-graph version 2 does not match version 1\n>> On branch master\n>> [...]\n>>\n>> ...and this series sets the default version for all new graphs to v2.\n>\n> The phrasing seems odd. It is unclear, even to me who is vaguely\n> familiar with the word \"commit-graph\" and is aware of the fact that\n> the file format is being updated, what\n>\n> \"commit-graph version 2 does not match version 1\"\n\nYeah it should really say:\n\n \"commit-graph is of version 2, our maximum supported version is 1\"\n\nHindsight is 20/20, but more generally I wonder if we should have these\nformat versions match that of the git version (unlikely to change it\ntwice in the same release...) which would allow us to say things like:\n\n \"commit-graph needs v2.22.0 or later, we have the version written by v2.18.0..v2.21.0\"\n\nBut of course dealing with those larger integers in the code/gaps is\nalso messy :)\n\n> wants to say. Do I have version #2 on disk and the running binary\n> only understands version #1? Or do I have version #1 on disk and\n> the binary expected version #2? How would I get out of this\n> situation? Is it sufficient to do \"rm -f .git/info/commit-graph*\"\n> and is it safe?\n\nYeah. An rm of .git/info/commit-graph is safe, so is \"-c\ncore.commitGraph=false\" as a workaround.\n\nI'd say \"let's improve the error\", but that ship has sailed, and we can\ndo better than an error here, no matter how it's phrased...\n\n>> I think this is *way* too aggressive of an upgrade path. If these\n>> patches go into v2.22.0 then git clients on all older versions that grok\n>> the commit graph (IIRC v2.18 and above) will have their git completely\n>> broken if they're in a mixed-git-version environment.\n>>\n\n\nI should note that \"all older versions...\" here is those that have\ncore.commitGraph=true set. More details in 43d3561805 (\"commit-graph\nwrite: don't die if the existing graph is corrupt\", 2019-03-25).\n\n>> Is it really so important to move to v2 right away that we need to risk\n>> those breakages? I think even with my ab/commit-graph-fixes it's still\n>> too annoying (I was mostly trying to fix other stuff...). If only we\n>> could detect \"we should make a new graph now\" ....\n>\n> True.\n\nHaving slept on my earlier\nhttps://public-inbox.org/git/87a7gdspo4.fsf@evledraar.gmail.com/ I think\nI see a better way to deal with this than my earlier suggestion that we\nperform some version flip-flop dance on the single \"commit-graph\" file:\n\nHow about just writing .git/objects/info/commit-graph-v2, and for the\nupcoming plan when where they'll be split have some dir/prefix there\nwhere we include the version?\n\nThat means that:\n\n 1. If there's an existing v1 \"commit-graph\" file we don't write a v2 at\n that path in v2.22, although we might have some \"write v1 (as well\n as v2?) for old client compat\" config where we opt-in to do that.\n\n 2. By default in v2.22 we read/write a \"commit-graph-v2\" file,\n preferring it over the v1 \"commit-graph\", falling back on earlier\n versions if it's not there (until gc --auto kicks in on v2.22 and\n makes a v2 graph).\n\n 3. If you have concurrent v2.21 and v2.22 clients accessing the repo\n you might end up generating one commit-graph or the other depending\n on who happens to trigger \"gc --auto\".\n\n Hopefully that's a non-issue since an out-of-date graph isn't\n usually a big deal, and client versions mostly march forward. But\n v2.22 could also learn some \"incremental gc\" where it says \"my v2 is\n older, v1 client must have refreshed it, I'll refresh mine/both\".\n\n 4. v2.22 and newer versions will have some code in git-gc where we'll\n eventually readdir() .git/objects/info and remove graphs that are\n too old per some new config (say\n \"gc.pruneOlderCommitGraphVersions=180 days\").\n\nThis means that:\n\n A. GOOD: Now and going forward we can fearlessly create new versions of\n the graph without worrying/testing how older clients deal with it.\n\n B. BAD: We are going to eat ~2x the disk space for commit-graphs while\n such transitions are underway. I think that's fine. They're\n relatively small compared to .git/objects, and we'll eventually \"gc\"\n the old ones.\n\n C. BAD: Different versions of git might perform wildly differently (new\n version slower) since their respective preferred graph versions\n might have a very different/up-to-date number of commits v.s. what's\n in the packs.\n\nI think \"A\" outweighs \"B\" && \"C\" in this case. It's \"just\" a caching\ndata structure, and git works without it. So we can be a lot looser than\nsay updating the index/pack format.\n\nWorst case things slow down but still work, and as noted in #3 above if\nwe care it can be mitigated (but I don't, I think we can safely assume\n\"client versions march forward\").\n"},{"id":"374556","messageId":"0a8fb6ec-d35c-45d7-3442-0e8298efca4b@gmail.com","threadId":"50302","inReplyTo":"878svxrwsh.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 0/5] Create commit-graph file format v2","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-04-26T12:06:05Z","receivedAt":"2019-04-26T12:06:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/26/2019 4:33 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Fri, Apr 26 2019, Junio C Hamano wrote:\n>>\n>> Thanks always for your careful review and thoughtful comments, by\n>> the way.\n\nI agree that these comments are extremely helpful.\n\n>>> Now as noted in my series we now on 'master' downgrade that to a warning\n>>> (along with the rest of the errors):\n>>>\n>>> $ ~/g/git/git --exec-path=$PWD status\n>>> error: commit-graph version 2 does not match version 1\n>>> On branch master\n>>> [...]\n>>>\n>>> ...and this series sets the default version for all new graphs to v2.>>>> The phrasing seems odd. It is unclear, even to me who is vaguely\n>> familiar with the word \"commit-graph\" and is aware of the fact that\n>> the file format is being updated, what\n>>\n>> \"commit-graph version 2 does not match version 1\"\n> \n> Yeah it should really say:\n> \n> \"commit-graph is of version 2, our maximum supported version is 1\"\n\nI agree this phrasing is better. Please see the patch I just submitted [1]\nto try and improve these messages.\n\n[1] https://public-inbox.org/git/pull.181.git.gitgitgadget@gmail.com/\n \n> Hindsight is 20/20, but more generally I wonder if we should have these\n> format versions match that of the git version (unlikely to change it\n> twice in the same release...) which would allow us to say things like:\n> \n> \"commit-graph needs v2.22.0 or later, we have the version written by v2.18.0..v2.21.0\"\n> \n> But of course dealing with those larger integers in the code/gaps is\n> also messy :)\n\nThere are a couple issues with using the version numbers, from my\nperspective:\n\n1. We don't do that anywhere else, like the index file.\n\n2. The microsoft/git fork takes certain performance changes faster\n than core git, and frequently ships versions between major version\n updates. Our 2.17 had the commit-graph, for instance. It's also\n possible that we'd take commit-graph v2 earlier than the core Git\n major release.\n \n>> wants to say. Do I have version #2 on disk and the running binary\n>> only understands version #1? Or do I have version #1 on disk and\n>> the binary expected version #2? How would I get out of this\n>> situation? Is it sufficient to do \"rm -f .git/info/commit-graph*\"\n>> and is it safe?\n> \n> Yeah. An rm of .git/info/commit-graph is safe, so is \"-c\n> core.commitGraph=false\" as a workaround.\n\nThat is true. I'm not sure the error message is the right place to\ndescribe the workaround.\n\n> I'd say \"let's improve the error\", but that ship has sailed, and we can\n> do better than an error here, no matter how it's phrased...\n> \n>>> I think this is *way* too aggressive of an upgrade path. If these\n>>> patches go into v2.22.0 then git clients on all older versions that grok\n>>> the commit graph (IIRC v2.18 and above) will have their git completely\n>>> broken if they're in a mixed-git-version environmen.>\n> I should note that \"all older versions...\" here is those that have\n> core.commitGraph=true set. More details in 43d3561805 (\"commit-graph\n> write: don't die if the existing graph is corrupt\", 2019-03-25).\n> \n>>> Is it really so important to move to v2 right away that we need to risk\n>>> those breakages? I think even with my ab/commit-graph-fixes it's still\n>>> too annoying (I was mostly trying to fix other stuff...). If only we\n>>> could detect \"we should make a new graph now\" ....\n>>\n>> True.\n\nYou are right, this is too aggressive and I should have known better. I'll\nupdate in the next version to keep a default to v1. Not only do we have this\ndowngrade risk, there is no actual benefit in this series alone. This only\nsets up the ability for other features.\n \n> Having slept on my earlier\n> https://public-inbox.org/git/87a7gdspo4.fsf@evledraar.gmail.com/ I think\n> I see a better way to deal with this than my earlier suggestion that we\n> perform some version flip-flop dance on the single \"commit-graph\" file:\n> \n> How about just writing .git/objects/info/commit-graph-v2, and for the\n> upcoming plan when where they'll be split have some dir/prefix there\n> where we include the version?\n> \n> That means that:\n> \n> 1. If there's an existing v1 \"commit-graph\" file we don't write a v2 at\n> that path in v2.22, although we might have some \"write v1 (as well\n> as v2?) for old client compat\" config where we opt-in to do that.\n> \n> 2. By default in v2.22 we read/write a \"commit-graph-v2\" file,\n> preferring it over the v1 \"commit-graph\", falling back on earlier\n> versions if it's not there (until gc --auto kicks in on v2.22 and\n> makes a v2 graph).\n> \n> 3. If you have concurrent v2.21 and v2.22 clients accessing the repo\n> you might end up generating one commit-graph or the other depending\n> on who happens to trigger \"gc --auto\".\n> \n> Hopefully that's a non-issue since an out-of-date graph isn't\n> usually a big deal, and client versions mostly march forward. But\n> v2.22 could also learn some \"incremental gc\" where it says \"my v2 is\n> older, v1 client must have refreshed it, I'll refresh mine/both\".\n> \n> 4. v2.22 and newer versions will have some code in git-gc where we'll\n> eventually readdir() .git/objects/info and remove graphs that are\n> too old per some new config (say\n> \"gc.pruneOlderCommitGraphVersions=180 days\").\n> \n> This means that:\n> \n> A. GOOD: Now and going forward we can fearlessly create new versions of\n> the graph without worrying/testing how older clients deal with it.\n> \n> B. BAD: We are going to eat ~2x the disk space for commit-graphs while\n> such transitions are underway. I think that's fine. They're\n> relatively small compared to .git/objects, and we'll eventually \"gc\"\n> the old ones.\n\nWe could also write 'commit-graph-v2' and delete 'commit-graph' and if\nsomeone downgrades they would just have a performance issue, not a failure.\n\n> C. BAD: Different versions of git might perform wildly differently (new\n> version slower) since their respective preferred graph versions\n> might have a very different/up-to-date number of commits v.s. what's\n> in the packs.\n> \n> I think \"A\" outweighs \"B\" && \"C\" in this case. It's \"just\" a caching\n> data structure, and git works without it. So we can be a lot looser than\n> say updating the index/pack format.\n> \n> Worst case things slow down but still work, and as noted in #3 above if\n> we care it can be mitigated (but I don't, I think we can safely assume\n> \"client versions march forward\").\n\nWhile I agree that this downgrade path can be a problem, I don't like the\nidea of adding a version in the filename. The whole point of having a versioned\nfile format is so we can make these changes without changing the filename.\n\nIs it sufficient to remove the auto-upgrade path, at least for a few major\nversions? And I can learn from past mistakes and change the response to\nthe other version information in the v2 file (reach index version, hash version,\nunused value in 8th byte) and instead make them non-fatal warnings.\n\nThanks,\n-Stolee\n"},{"id":"374558","messageId":"874l6kswew.fsf@evledraar.gmail.com","threadId":"50302","inReplyTo":"0a8fb6ec-d35c-45d7-3442-0e8298efca4b@gmail.com","subject":"Re: [PATCH v2 0/5] Create commit-graph file format v2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-04-26T13:55:51Z","receivedAt":"2019-04-26T13:55:58Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Apr 26 2019, Derrick Stolee wrote:\n\n> On 4/26/2019 4:33 AM, Ævar Arnfjörð Bjarmason wrote:\n>>\n>> On Fri, Apr 26 2019, Junio C Hamano wrote:\n>>>\n>>> Thanks always for your careful review and thoughtful comments, by\n>>> the way.\n>\n> I agree that these comments are extremely helpful.\n>\n>>>> Now as noted in my series we now on 'master' downgrade that to a warning\n>>>> (along with the rest of the errors):\n>>>>\n>>>> $ ~/g/git/git --exec-path=$PWD status\n>>>> error: commit-graph version 2 does not match version 1\n>>>> On branch master\n>>>> [...]\n>>>>\n>>>> ...and this series sets the default version for all new graphs to v2.>>>> The phrasing seems odd. It is unclear, even to me who is vaguely\n>>> familiar with the word \"commit-graph\" and is aware of the fact that\n>>> the file format is being updated, what\n>>>\n>>> \"commit-graph version 2 does not match version 1\"\n>>\n>> Yeah it should really say:\n>>\n>> \"commit-graph is of version 2, our maximum supported version is 1\"\n>\n> I agree this phrasing is better. Please see the patch I just submitted [1]\n> to try and improve these messages.\n>\n> [1] https://public-inbox.org/git/pull.181.git.gitgitgadget@gmail.com/\n>\n>> Hindsight is 20/20, but more generally I wonder if we should have these\n>> format versions match that of the git version (unlikely to change it\n>> twice in the same release...) which would allow us to say things like:\n>>\n>> \"commit-graph needs v2.22.0 or later, we have the version written by v2.18.0..v2.21.0\"\n>>\n>> But of course dealing with those larger integers in the code/gaps is\n>> also messy :)\n>\n> There are a couple issues with using the version numbers, from my\n> perspective:\n>\n> 1. We don't do that anywhere else, like the index file.\n>\n> 2. The microsoft/git fork takes certain performance changes faster\n> than core git, and frequently ships versions between major version\n> updates. Our 2.17 had the commit-graph, for instance. It's also\n> possible that we'd take commit-graph v2 earlier than the core Git\n> major release.\n\nGood points. I'm just blathering on and playing architecture astronaut\n:)\n\n>>> wants to say. Do I have version #2 on disk and the running binary\n>>> only understands version #1? Or do I have version #1 on disk and\n>>> the binary expected version #2? How would I get out of this\n>>> situation? Is it sufficient to do \"rm -f .git/info/commit-graph*\"\n>>> and is it safe?\n>>\n>> Yeah. An rm of .git/info/commit-graph is safe, so is \"-c\n>> core.commitGraph=false\" as a workaround.\n>\n> That is true. I'm not sure the error message is the right place to\n> describe the workaround.\n\nYeah for sure, it should Just Work...\n\n>> I'd say \"let's improve the error\", but that ship has sailed, and we can\n>> do better than an error here, no matter how it's phrased...\n>>\n>>>> I think this is *way* too aggressive of an upgrade path. If these\n>>>> patches go into v2.22.0 then git clients on all older versions that grok\n>>>> the commit graph (IIRC v2.18 and above) will have their git completely\n>>>> broken if they're in a mixed-git-version environmen.>\n>> I should note that \"all older versions...\" here is those that have\n>> core.commitGraph=true set. More details in 43d3561805 (\"commit-graph\n>> write: don't die if the existing graph is corrupt\", 2019-03-25).\n>>\n>>>> Is it really so important to move to v2 right away that we need to risk\n>>>> those breakages? I think even with my ab/commit-graph-fixes it's still\n>>>> too annoying (I was mostly trying to fix other stuff...). If only we\n>>>> could detect \"we should make a new graph now\" ....\n>>>\n>>> True.\n>\n> You are right, this is too aggressive and I should have known better. I'll\n> update in the next version to keep a default to v1. Not only do we have this\n> downgrade risk, there is no actual benefit in this series alone. This only\n> sets up the ability for other features.\n>\n>> Having slept on my earlier\n>> https://public-inbox.org/git/87a7gdspo4.fsf@evledraar.gmail.com/ I think\n>> I see a better way to deal with this than my earlier suggestion that we\n>> perform some version flip-flop dance on the single \"commit-graph\" file:\n>>\n>> How about just writing .git/objects/info/commit-graph-v2, and for the\n>> upcoming plan when where they'll be split have some dir/prefix there\n>> where we include the version?\n>>\n>> That means that:\n>>\n>> 1. If there's an existing v1 \"commit-graph\" file we don't write a v2 at\n>> that path in v2.22, although we might have some \"write v1 (as well\n>> as v2?) for old client compat\" config where we opt-in to do that.\n>>\n>> 2. By default in v2.22 we read/write a \"commit-graph-v2\" file,\n>> preferring it over the v1 \"commit-graph\", falling back on earlier\n>> versions if it's not there (until gc --auto kicks in on v2.22 and\n>> makes a v2 graph).\n>>\n>> 3. If you have concurrent v2.21 and v2.22 clients accessing the repo\n>> you might end up generating one commit-graph or the other depending\n>> on who happens to trigger \"gc --auto\".\n>>\n>> Hopefully that's a non-issue since an out-of-date graph isn't\n>> usually a big deal, and client versions mostly march forward. But\n>> v2.22 could also learn some \"incremental gc\" where it says \"my v2 is\n>> older, v1 client must have refreshed it, I'll refresh mine/both\".\n>>\n>> 4. v2.22 and newer versions will have some code in git-gc where we'll\n>> eventually readdir() .git/objects/info and remove graphs that are\n>> too old per some new config (say\n>> \"gc.pruneOlderCommitGraphVersions=180 days\").\n>>\n>> This means that:\n>>\n>> A. GOOD: Now and going forward we can fearlessly create new versions of\n>> the graph without worrying/testing how older clients deal with it.\n>>\n>> B. BAD: We are going to eat ~2x the disk space for commit-graphs while\n>> such transitions are underway. I think that's fine. They're\n>> relatively small compared to .git/objects, and we'll eventually \"gc\"\n>> the old ones.\n>\n> We could also write 'commit-graph-v2' and delete 'commit-graph' and if\n> someone downgrades they would just have a performance issue, not a failure.\n>\n>> C. BAD: Different versions of git might perform wildly differently (new\n>> version slower) since their respective preferred graph versions\n>> might have a very different/up-to-date number of commits v.s. what's\n>> in the packs.\n>>\n>> I think \"A\" outweighs \"B\" && \"C\" in this case. It's \"just\" a caching\n>> data structure, and git works without it. So we can be a lot looser than\n>> say updating the index/pack format.\n>>\n>> Worst case things slow down but still work, and as noted in #3 above if\n>> we care it can be mitigated (but I don't, I think we can safely assume\n>> \"client versions march forward\").\n>\n> While I agree that this downgrade path can be a problem, I don't like the\n> idea of adding a version in the filename. The whole point of having a versioned\n> file format is so we can make these changes without changing the filename.\n>\n> Is it sufficient to remove the auto-upgrade path, at least for a few major\n> versions? And I can learn from past mistakes and change the response to\n> the other version information in the v2 file (reach index version, hash version,\n> unused value in 8th byte) and instead make them non-fatal warnings.\n\nI think there's two things here, and for *me* just one of them would be\nenough for \"screw it, let's write another file\":\n\n 1. Our \"you have v2\" error reporting for all versions until the one we\n haven't released yet sucks/hard errors.\n\n So right *now* I think few people turn on core.commitGraph=true, I\n do, you do, I just convinced GitLab to do it (after doing it myself\n & carrying patches...):\n https://gitlab.com/gitlab-org/gitaly/issues/1627\n\n I think we both want to get to a point where core.commitGraph=true\n is the default though, because it's cheap to make it, and it rocks\n for a lot of use-cases, meanwhile people are toggling it on manually\n at an increasing rate.\n\n So that combined with some distros/OSs upgrading at a glacial pace\n basically means that we'd *at least* need to do the equivalent of a\n one-off \"commit-graph\" -> \"commit-graph-verion-2-and-beyond\",\n because just like some OSs are still shipping with git 1.8.* or 2.11\n whatever, we're also going to have the versions where this is hard\n erroring in production somewhere for a long time.\n\n It would suck with even a \"conservative\" upgrade path to need to\n wait until 2022 or something just so I can have a commit graph that\n by default has the number of objects or whatever other small thing\n we add to it because we're paranoid that it hasn't been N versions\n since we stopped hard-erroring on v2.\n\n There's a lot of such mixed-version cases, e.g. running a server\n cluster where the git data is on NFS and you're in the process of\n updating git on some \"client\" nodes but not others, now if the git\n version is one of the ones impacted and you e.g. run gitlab the new\n version will ruin your day. Ditto a user on their laptop testing a\n git from debian testing and going back to stable etc, only to find\n that their repos broke.\n\n 2. While we *can* say a lost commit-graph is \"just a performance issue,\n not a failure\", which is easy enough to make the case for v2.22 and\n beyond if we fix a couple of things, I think this is becoming less\n and less acceptable in practice.\n\n E.g. I have some things now where I pretty much hard-rely on it if I\n don't want a CPU/IO spike as some commands that take 5ms now start\n taking 30s (e.g. \"what branch is this on\" in the GitLab UI)., which\n would happen if we have one file and switch to v2 by default at some\n point (and v3, ...).\n\n"},{"id":"374599","messageId":"87zhobr4fl.fsf@evledraar.gmail.com","threadId":"50302","inReplyTo":"87a7gdspo4.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 0/5] Create commit-graph file format v2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-04-27T12:57:50Z","receivedAt":"2019-04-27T12:57:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Apr 26 2019, Ævar Arnfjörð Bjarmason wrote:\n\n> ...speaking of which, digging up outstanding stuff I have on the\n> commit-graph I was reminded to finish up my \"commit graph on clone\"\n> patch in:\n> https://public-inbox.org/git/87in2hgzin.fsf@evledraar.gmail.com/\n>\n> And re #1 above: I guess we could also do that \"let's make a graph\" and\n> call \"gc --auto\" if a) we have gc.writeCommitGraph b) we see it's not\n> the \"right\" version. As long as older versions always write a \"old\" one\n> if they can't grok the \"new\" one, and newer versions leave existing\n> graphs alone even if they're older versions, so we don't flip-flop.\n>\n> One of the things that would make that \"graph on clone/fetch/whatever\"\n> easier is having the graph store the total number of objects while it\n> was at it, you indicated in\n> https://public-inbox.org/git/934fa00e-f6df-c333-4968-3e9acffab22d@gmail.com/\n> that you already have an internal MSFT implementation of it that does\n> it.\n>\n> Any reason not to make it part of v2 while we're at it? We already find\n> out how many (packed) objects we have in \"add_packed_commits\", we just\n> don't do anything with that information now.\n\nI hacked up this plus general tag/tree/blob count stats in the WIP/RFC\npatch below. I figured once I did objects I might as well do tags (note:\nannotated tag objects, not # tag refs)/trees/blobs as well.\n\nIt passes all tests with GIT_TEST_COMMIT_GRAPH=true, it fails on the\ncommit-graph's own test suite, but AFAICT only because the selective\ncorruption tests are thrown off by the location of this new chunk.\n\nSince we now skip some commits found in the pack(s) (just duplicates?)\nthe new \"num_commits_stat\" is not the same as the current \"num_commits\",\nbut usually really close.\n\nIt's probably best if we do something like this to make this chunk be of\ndynamic length, as long as we kept order we could keep adding new stats\nto the file even within the same \"version\".\n\nThis as (ab)using the \"commit-graph\" to start storing arbitrary stats\nabout stuff we find the the packs during gc. Maybe that sucks, but OTOH\nit's useful, and just having some new \"last gc stats\" format/file would\nbe overkill...\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 1485b4daaf..d9378f23d9 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -114,7 +114,9 @@ static int graph_read(int argc, const char **argv)\n \tprintf(\"header: %08x %d %d %d %d\\n\",\n \t\tntohl(*(uint32_t*)graph->data),\n \t\t*(unsigned char*)(graph->data + 4),\n-\t\t*(unsigned char*)(graph->data + 5),\n+\t\t(getenv(\"STAT_ME\")\n+\t\t ? *(unsigned char*)(graph->data + 5)\n+\t\t : (*(unsigned char*)(graph->data + 5) - 1)),\n \t\t*(unsigned char*)(graph->data + 6),\n \t\t*(unsigned char*)(graph->data + 7));\n\n@@ -123,8 +125,20 @@ static int graph_read(int argc, const char **argv)\n \t\t get_be32(graph->data + 8));\n\n \tprintf(\"num_commits: %u\\n\", graph->num_commits);\n+\tif (getenv(\"STAT_ME\")) {\n+\t\tprintf(\" pack num commits (st): %u\\n\", graph->num_commits_stat);\n+\t\tprintf(\" pack num commits - inferred diff (diff = duplicate (I think!)): %u\\n\", graph->num_commits_stat - graph->num_commits);\n+\t\tprintf(\" pack num objects: %u\\n\", graph->num_objects);\n+\t\tprintf(\" pack num tags: %u\\n\", graph->num_tags);\n+\t\tprintf(\" pack num trees: %u\\n\", graph->num_trees);\n+\t\tprintf(\" pack num blobs: %u\\n\", graph->num_blobs);\n+\t}\n \tprintf(\"chunks:\");\n\n+\tif (getenv(\"STAT_ME\")) {\n+\t\tif (graph->chunk_oid_numbers)\n+\t\t\tprintf(\" oid_numbers\");\n+\t}\n \tif (graph->chunk_oid_fanout)\n \t\tprintf(\" oid_fanout\");\n \tif (graph->chunk_oid_lookup)\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 14d6aebd99..3d0fb5193b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -18,6 +18,7 @@\n #include \"progress.h\"\n\n #define GRAPH_SIGNATURE 0x43475048 /* \"CGPH\" */\n+#define GRAPH_CHUNKID_OIDNUMBERS 0x4f49444e /* \"OIDN\" */\n #define GRAPH_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n #define GRAPH_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define GRAPH_CHUNKID_DATA 0x43444154 /* \"CDAT\" */\n@@ -32,6 +33,7 @@\n #define GRAPH_LAST_EDGE 0x80000000\n\n #define GRAPH_HEADER_SIZE 8\n+#define GRAPH_OIDNUMBERS_SIZE (4 * 5)\n #define GRAPH_FANOUT_SIZE (4 * 256)\n #define GRAPH_CHUNKLOOKUP_WIDTH 12\n #define GRAPH_MIN_SIZE (GRAPH_HEADER_SIZE + 4 * GRAPH_CHUNKLOOKUP_WIDTH \\\n@@ -127,6 +129,10 @@ static int verify_commit_graph_lite(struct commit_graph *g)\n \t * over g->num_commits, or runs a checksum on the commit-graph\n \t * itself.\n \t */\n+\tif (!g->chunk_oid_numbers) {\n+\t\terror(\"commit-graph is missing the OID Numbers chunk\");\n+\t\treturn 1;\n+\t}\n \tif (!g->chunk_oid_fanout) {\n \t\terror(\"commit-graph is missing the OID Fanout chunk\");\n \t\treturn 1;\n@@ -249,6 +255,18 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t\t}\n\n \t\tswitch (chunk_id) {\n+\t\tcase GRAPH_CHUNKID_OIDNUMBERS:\n+\t\t\tif (graph->chunk_oid_numbers) {\n+\t\t\t\tchunk_repeated = 1;\n+\t\t\t} else {\n+\t\t\t\tgraph->chunk_oid_numbers = data + chunk_offset;\n+\t\t\t\tgraph->num_objects = get_be32(graph->chunk_oid_numbers + 0);\n+\t\t\t\tgraph->num_commits_stat = get_be32(graph->chunk_oid_numbers + 4);\n+\t\t\t\tgraph->num_tags = get_be32(graph->chunk_oid_numbers + 8);\n+\t\t\t\tgraph->num_trees = get_be32(graph->chunk_oid_numbers + 12);\n+\t\t\t\tgraph->num_blobs = get_be32(graph->chunk_oid_numbers + 16);\n+\t\t\t}\n+\t\t\tbreak;\n \t\tcase GRAPH_CHUNKID_OIDFANOUT:\n \t\t\tif (graph->chunk_oid_fanout)\n \t\t\t\tchunk_repeated = 1;\n@@ -545,6 +563,22 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \treturn get_commit_tree_in_graph_one(r, r->objects->commit_graph, c);\n }\n\n+static void write_graph_chunk_numbers(struct hashfile *f,\n+\t\t\t\t struct progress *progress,\n+\t\t\t\t uint64_t *progress_cnt,\n+\t\t\t\t uint32_t num_objects,\n+\t\t\t\t uint32_t num_commits,\n+\t\t\t\t uint32_t num_tags,\n+\t\t\t\t uint32_t num_trees,\n+\t\t\t\t uint32_t num_blobs)\n+{\n+\thashwrite_be32(f, num_objects);\n+\thashwrite_be32(f, num_commits);\n+\thashwrite_be32(f, num_tags);\n+\thashwrite_be32(f, num_trees);\n+\thashwrite_be32(f, num_blobs);\n+}\n+\n static void write_graph_chunk_fanout(struct hashfile *f,\n \t\t\t\t struct commit **commits,\n \t\t\t\t int nr_commits,\n@@ -567,7 +601,6 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n \t\t\tcount++;\n \t\t\tlist++;\n \t\t}\n-\n \t\thashwrite_be32(f, count);\n \t}\n }\n@@ -732,6 +765,11 @@ struct packed_oid_list {\n \tint alloc;\n \tstruct progress *progress;\n \tint progress_done;\n+\tuint32_t num_objects;\n+\tuint32_t num_commits;\n+\tuint32_t num_tags;\n+\tuint32_t num_trees;\n+\tuint32_t num_blobs;\n };\n\n static int add_packed_commits(const struct object_id *oid,\n@@ -751,6 +789,21 @@ static int add_packed_commits(const struct object_id *oid,\n \tif (packed_object_info(the_repository, pack, offset, &oi) < 0)\n \t\tdie(_(\"unable to get type of object %s\"), oid_to_hex(oid));\n\n+\t/*\n+\t * Aggregate object statistics\n+\t */\n+\tlist->num_objects++;\n+\tif (type == OBJ_COMMIT)\n+\t\tlist->num_commits++;\n+\telse if (type == OBJ_TAG)\n+\t\tlist->num_tags++;\n+\telse if (type == OBJ_TREE)\n+\t\tlist->num_trees++;\n+\telse if (type == OBJ_BLOB)\n+\t\tlist->num_blobs++;\n+\telse\n+\t\tBUG(\"should not encounter internal-only object_type %d value here!\", type);\n+\n \tif (type != OBJ_COMMIT)\n \t\treturn 0;\n\n@@ -939,6 +992,8 @@ int write_commit_graph(const char *obj_dir,\n \toids.progress = NULL;\n \toids.progress_done = 0;\n \tcommits.list = NULL;\n+\toids.num_objects = oids.num_commits = oids.num_tags =\n+\t\toids.num_trees = oids.num_blobs = 0;\n\n \tif (append) {\n \t\tprepare_commit_graph_one(the_repository, obj_dir);\n@@ -1092,7 +1147,7 @@ int write_commit_graph(const char *obj_dir,\n\n \t\tcommits.nr++;\n \t}\n-\tnum_chunks = num_extra_edges ? 4 : 3;\n+\tnum_chunks = num_extra_edges ? 5 : 4;\n \tstop_progress(&progress);\n\n \tif (commits.nr >= GRAPH_EDGE_LAST_MASK) {\n@@ -1136,20 +1191,22 @@ int write_commit_graph(const char *obj_dir,\n \t\tbreak;\n \t}\n\n-\tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n-\tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n-\tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n+\tchunk_ids[0] = GRAPH_CHUNKID_OIDNUMBERS;\n+\tchunk_ids[1] = GRAPH_CHUNKID_OIDFANOUT;\n+\tchunk_ids[2] = GRAPH_CHUNKID_OIDLOOKUP;\n+\tchunk_ids[3] = GRAPH_CHUNKID_DATA;\n \tif (num_extra_edges)\n-\t\tchunk_ids[3] = GRAPH_CHUNKID_EXTRAEDGES;\n+\t\tchunk_ids[4] = GRAPH_CHUNKID_EXTRAEDGES;\n \telse\n-\t\tchunk_ids[3] = 0;\n-\tchunk_ids[4] = 0;\n+\t\tchunk_ids[4] = 0;\n+\tchunk_ids[5] = 0;\n\n \tchunk_offsets[0] = header_size + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n-\tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n-\tchunk_offsets[2] = chunk_offsets[1] + hashsz * commits.nr;\n-\tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * commits.nr;\n-\tchunk_offsets[4] = chunk_offsets[3] + 4 * num_extra_edges;\n+\tchunk_offsets[1] = chunk_offsets[0] + GRAPH_OIDNUMBERS_SIZE;\n+\tchunk_offsets[2] = chunk_offsets[1] + GRAPH_FANOUT_SIZE;\n+\tchunk_offsets[3] = chunk_offsets[2] + hashsz * commits.nr;\n+\tchunk_offsets[4] = chunk_offsets[3] + (hashsz + 16) * commits.nr;\n+\tchunk_offsets[5] = chunk_offsets[4] + 4 * num_extra_edges;\n\n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tuint32_t chunk_write[3];\n@@ -1170,6 +1227,10 @@ int write_commit_graph(const char *obj_dir,\n \t\t\tprogress_title.buf,\n \t\t\tnum_chunks * commits.nr);\n \t}\n+\twrite_graph_chunk_numbers(f, progress, &progress_cnt,\n+\t\t\t\t oids.num_objects, oids.num_commits,\n+\t\t\t\t oids.num_tags, oids.num_trees,\n+\t\t\t\t oids.num_blobs);\n \twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress, &progress_cnt);\n \twrite_graph_chunk_oids(f, hashsz, commits.list, commits.nr, progress, &progress_cnt);\n \twrite_graph_chunk_data(f, hashsz, commits.list, commits.nr, progress, &progress_cnt);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 2c461770e8..ef9eb0b6cb 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -45,10 +45,16 @@ struct commit_graph {\n\n \tunsigned char hash_len;\n \tunsigned char num_chunks;\n+\tuint32_t num_objects;\n \tuint32_t num_commits;\n+\tuint32_t num_commits_stat;\n+\tuint32_t num_tags;\n+\tuint32_t num_trees;\n+\tuint32_t num_blobs;\n \tstruct object_id oid;\n\n \tconst uint32_t *chunk_oid_fanout;\n+\tconst unsigned char *chunk_oid_numbers;\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_commit_data;\n \tconst unsigned char *chunk_extra_edges;\n\n\nOpel Vivaro\n"},{"id":"374754","messageId":"91f300ec0acd825494fa13d93cdc4770bbf2ac14.1556716272.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 1/6] commit-graph: return with errors during write","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-01T13:11:14Z","receivedAt":"2019-05-01T13:11:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method uses die() to report failure and\nexit when confronted with an unexpected condition. This use of\ndie() in a library function is incorrect and is now replaced by\nerror() statements and an int return type.\n\nNow that we use 'goto cleanup' to jump to the terminal condition\non an error, we have new paths that could lead to uninitialized\nvalues. New initializers are added to correct for this.\n\nThe builtins 'commit-graph', 'gc', and 'commit' call these methods,\nso update them to check the return value.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 19 +++++++------\n builtin/commit.c | 5 ++--\n builtin/gc.c | 7 ++---\n commit-graph.c | 60 +++++++++++++++++++++++++++++-------------\n commit-graph.h | 10 +++----\n 5 files changed, 62 insertions(+), 39 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 537fdfd0f0..2e86251f02 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -141,6 +141,7 @@ static int graph_write(int argc, const char **argv)\n \tstruct string_list *pack_indexes = NULL;\n \tstruct string_list *commit_hex = NULL;\n \tstruct string_list lines;\n+\tint result;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n@@ -168,10 +169,8 @@ static int graph_write(int argc, const char **argv)\n \n \tread_replace_refs = 0;\n \n-\tif (opts.reachable) {\n-\t\twrite_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n-\t\treturn 0;\n-\t}\n+\tif (opts.reachable)\n+\t\treturn write_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n \n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n@@ -188,14 +187,14 @@ static int graph_write(int argc, const char **argv)\n \t\tUNLEAK(buf);\n \t}\n \n-\twrite_commit_graph(opts.obj_dir,\n-\t\t\t pack_indexes,\n-\t\t\t commit_hex,\n-\t\t\t opts.append,\n-\t\t\t 1);\n+\tresult = write_commit_graph(opts.obj_dir,\n+\t\t\t\t pack_indexes,\n+\t\t\t\t commit_hex,\n+\t\t\t\t opts.append,\n+\t\t\t\t 1);\n \n \tUNLEAK(lines);\n-\treturn 0;\n+\treturn result;\n }\n \n int cmd_commit_graph(int argc, const char **argv, const char *prefix)\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 2986553d5f..b9ea7222fa 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1669,8 +1669,9 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\t \"new_index file. Check that disk is not full and quota is\\n\"\n \t\t \"not exceeded, and then \\\"git reset HEAD\\\" to recover.\"));\n \n-\tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0))\n-\t\twrite_commit_graph_reachable(get_object_directory(), 0, 0);\n+\tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n+\t write_commit_graph_reachable(get_object_directory(), 0, 0))\n+\t\treturn 1;\n \n \trepo_rerere(the_repository, 0);\n \trun_command_v_opt(argv_gc_auto, RUN_GIT_CMD);\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 020f725acc..3984addf73 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -664,9 +664,10 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tclean_pack_garbage();\n \t}\n \n-\tif (gc_write_commit_graph)\n-\t\twrite_commit_graph_reachable(get_object_directory(), 0,\n-\t\t\t\t\t !quiet && !daemonized);\n+\tif (gc_write_commit_graph &&\n+\t write_commit_graph_reachable(get_object_directory(), 0,\n+\t\t\t\t\t !quiet && !daemonized))\n+\t\treturn 1;\n \n \tif (auto_gc && too_many_loose_objects())\n \t\twarning(_(\"There are too many unreachable loose objects; \"\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 66865acbd7..ee487a364b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -851,27 +851,30 @@ static int add_ref_to_list(const char *refname,\n \treturn 0;\n }\n \n-void write_commit_graph_reachable(const char *obj_dir, int append,\n-\t\t\t\t int report_progress)\n+int write_commit_graph_reachable(const char *obj_dir, int append,\n+\t\t\t\t int report_progress)\n {\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n+\tint result;\n \n \tfor_each_ref(add_ref_to_list, &list);\n-\twrite_commit_graph(obj_dir, NULL, &list, append, report_progress);\n+\tresult = write_commit_graph(obj_dir, NULL, &list,\n+\t\t\t\t append, report_progress);\n \n \tstring_list_clear(&list, 0);\n+\treturn result;\n }\n \n-void write_commit_graph(const char *obj_dir,\n-\t\t\tstruct string_list *pack_indexes,\n-\t\t\tstruct string_list *commit_hex,\n-\t\t\tint append, int report_progress)\n+int write_commit_graph(const char *obj_dir,\n+\t\t struct string_list *pack_indexes,\n+\t\t struct string_list *commit_hex,\n+\t\t int append, int report_progress)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n \tstruct hashfile *f;\n \tuint32_t i, count_distinct = 0;\n-\tchar *graph_name;\n+\tchar *graph_name = NULL;\n \tstruct lock_file lk = LOCK_INIT;\n \tuint32_t chunk_ids[5];\n \tuint64_t chunk_offsets[5];\n@@ -883,15 +886,17 @@ void write_commit_graph(const char *obj_dir,\n \tuint64_t progress_cnt = 0;\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tunsigned long approx_nr_objects;\n+\tint res = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n-\t\treturn;\n+\t\treturn 0;\n \n \toids.nr = 0;\n \tapprox_nr_objects = approximate_object_count();\n \toids.alloc = approx_nr_objects / 32;\n \toids.progress = NULL;\n \toids.progress_done = 0;\n+\tcommits.list = NULL;\n \n \tif (append) {\n \t\tprepare_commit_graph_one(the_repository, obj_dir);\n@@ -932,10 +937,16 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tstrbuf_setlen(&packname, dirlen);\n \t\t\tstrbuf_addstr(&packname, pack_indexes->items[i].string);\n \t\t\tp = add_packed_git(packname.buf, packname.len, 1);\n-\t\t\tif (!p)\n-\t\t\t\tdie(_(\"error adding pack %s\"), packname.buf);\n-\t\t\tif (open_pack_index(p))\n-\t\t\t\tdie(_(\"error opening index for %s\"), packname.buf);\n+\t\t\tif (!p) {\n+\t\t\t\terror(_(\"error adding pack %s\"), packname.buf);\n+\t\t\t\tres = 1;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\t\t\tif (open_pack_index(p)) {\n+\t\t\t\terror(_(\"error opening index for %s\"), packname.buf);\n+\t\t\t\tres = 1;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \t\t\tfor_each_object_in_pack(p, add_packed_commits, &oids,\n \t\t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n \t\t\tclose_pack(p);\n@@ -1006,8 +1017,11 @@ void write_commit_graph(const char *obj_dir,\n \t}\n \tstop_progress(&progress);\n \n-\tif (count_distinct >= GRAPH_EDGE_LAST_MASK)\n-\t\tdie(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n+\tif (count_distinct >= GRAPH_EDGE_LAST_MASK) {\n+\t\terror(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n+\t\tres = 1;\n+\t\tgoto cleanup;\n+\t}\n \n \tcommits.nr = 0;\n \tcommits.alloc = count_distinct;\n@@ -1039,16 +1053,21 @@ void write_commit_graph(const char *obj_dir,\n \tnum_chunks = num_extra_edges ? 4 : 3;\n \tstop_progress(&progress);\n \n-\tif (commits.nr >= GRAPH_EDGE_LAST_MASK)\n-\t\tdie(_(\"too many commits to write graph\"));\n+\tif (commits.nr >= GRAPH_EDGE_LAST_MASK) {\n+\t\terror(_(\"too many commits to write graph\"));\n+\t\tres = 1;\n+\t\tgoto cleanup;\n+\t}\n \n \tcompute_generation_numbers(&commits, report_progress);\n \n \tgraph_name = get_commit_graph_filename(obj_dir);\n \tif (safe_create_leading_directories(graph_name)) {\n \t\tUNLEAK(graph_name);\n-\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n-\t\t\t graph_name);\n+\t\terror(_(\"unable to create leading directories of %s\"),\n+\t\t\tgraph_name);\n+\t\tres = errno;\n+\t\tgoto cleanup;\n \t}\n \n \thold_lock_file_for_update(&lk, graph_name, LOCK_DIE_ON_ERROR);\n@@ -1107,9 +1126,12 @@ void write_commit_graph(const char *obj_dir,\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n \tcommit_lock_file(&lk);\n \n+cleanup:\n \tfree(graph_name);\n \tfree(commits.list);\n \tfree(oids.list);\n+\n+\treturn res;\n }\n \n #define VERIFY_COMMIT_GRAPH_ERROR_HASH 2\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 7dfb8c896f..d15670bf46 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -65,12 +65,12 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n */\n int generation_numbers_enabled(struct repository *r);\n \n-void write_commit_graph_reachable(const char *obj_dir, int append,\n+int write_commit_graph_reachable(const char *obj_dir, int append,\n \t\t\t\t int report_progress);\n-void write_commit_graph(const char *obj_dir,\n-\t\t\tstruct string_list *pack_indexes,\n-\t\t\tstruct string_list *commit_hex,\n-\t\t\tint append, int report_progress);\n+int write_commit_graph(const char *obj_dir,\n+\t\t struct string_list *pack_indexes,\n+\t\t struct string_list *commit_hex,\n+\t\t int append, int report_progress);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \n-- \ngitgitgadget\n\n"},{"id":"374755","messageId":"pull.112.v3.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v3 0/6] Create commit-graph file format v2","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-01T13:11:13Z","receivedAt":"2019-05-01T13:11:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit-graph file format has some shortcomings that were discussed\non-list:\n\n 1. It doesn't use the 4-byte format ID from the_hash_algo.\n \n \n 2. There is no way to change the reachability index from generation numbers\n to corrected commit date [1].\n \n \n 3. The unused byte in the format could be used to signal the file is\n incremental, but current clients ignore the value even if it is\n non-zero.\n \n \n\nThis series adds a new version (2) to the commit-graph file. The fifth byte\nalready specified the file format, so existing clients will gracefully\nrespond to files with a different version number. The only real change now\nis that the header takes 12 bytes instead of 8, due to using the 4-byte\nformat ID for the hash algorithm.\n\nThe new bytes reserved for the reachability index version and incremental\nfile formats are now expected to be equal to the defaults. When we update\nthese values to be flexible in the future, if a client understands\ncommit-graph v2 but not those new values, then it will fail gracefully.\n\nNOTE: this series was rebased onto ab/commit-graph-fixes, as the conflicts\nwere significant and subtle.\n\nUpdates in V3: Thanks for all the feedback so far!\n\n * Moved the version information into an unsigned char parameter, instead of\n a flag.\n \n \n * We no longer default to the v2 file format, as that will break users who\n downgrade. This required some changes to the test script.\n \n \n * Removed the \"Future work\" section from the commit-graph design document\n in a new patch.\n \n \n * I did not change the file name for v2 file formats, as Ævar suggested.\n I'd like the discussion to continue on this topic.\n \n \n\nThanks, -Stolee\n\n[1] \nhttps://public-inbox.org/git/6367e30a-1b3a-4fe9-611b-d931f51effef@gmail.com/\n\nDerrick Stolee (6):\n commit-graph: return with errors during write\n commit-graph: collapse parameters into flags\n commit-graph: create new version parameter\n commit-graph: add --version= option\n commit-graph: implement file format version 2\n commit-graph: remove Future Work section\n\n Documentation/git-commit-graph.txt | 3 +\n .../technical/commit-graph-format.txt | 26 ++-\n Documentation/technical/commit-graph.txt | 17 --\n builtin/commit-graph.c | 33 ++--\n builtin/commit.c | 5 +-\n builtin/gc.c | 8 +-\n commit-graph.c | 153 +++++++++++++-----\n commit-graph.h | 16 +-\n t/t5318-commit-graph.sh | 75 ++++++++-\n 9 files changed, 250 insertions(+), 86 deletions(-)\n\n\nbase-commit: 93b4405ffe4ad9308740e7c1c71383bfc369baaa\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-112%2Fderrickstolee%2Fgraph%2Fv2-head-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-112/derrickstolee/graph/v2-head-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/112\n\nRange-diff vs v2:\n\n 1: 91f300ec0a = 1: 91f300ec0a commit-graph: return with errors during write\n 2: 04f5df1135 ! 2: 924b22f990 commit-graph: collapse parameters into flags\n @@ -86,7 +86,7 @@\n \n -int write_commit_graph_reachable(const char *obj_dir, int append,\n -\t\t\t\t int report_progress)\n -+int write_commit_graph_reachable(const char *obj_dir, int flags)\n ++int write_commit_graph_reachable(const char *obj_dir, unsigned int flags)\n {\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n \tint result;\n @@ -103,7 +103,7 @@\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n -\t\t int append, int report_progress)\n -+\t\t int flags)\n ++\t\t unsigned int flags)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n @@ -129,12 +129,12 @@\n +#define COMMIT_GRAPH_APPEND (1 << 0)\n +#define COMMIT_GRAPH_PROGRESS (1 << 1)\n +\n -+int write_commit_graph_reachable(const char *obj_dir, int flags);\n ++int write_commit_graph_reachable(const char *obj_dir, unsigned int flags);\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n -\t\t int append, int report_progress);\n -+\t\t int flags);\n ++\t\t unsigned int flags);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \n 3: 4ddb829163 ! 3: 8446011a43 commit-graph: create new version flags\n @@ -1,12 +1,12 @@\n Author: Derrick Stolee \n \n - commit-graph: create new version flags\n + commit-graph: create new version parameter\n \n In anticipation of a new commit-graph file format version, create\n - a flag for the write_commit_graph() and write_commit_graph_reachable()\n + a parameter for the write_commit_graph() and write_commit_graph_reachable()\n methods to take a version number.\n \n - When there is no specified version, the implementation selects a\n + When the given version is zero, the implementation selects a\n default value. Currently, the only valid value is 1.\n \n The file format will change the header information, so place the\n @@ -14,6 +14,55 @@\n \n Signed-off-by: Derrick Stolee \n \n + diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n + --- a/builtin/commit-graph.c\n + +++ b/builtin/commit-graph.c\n +@@\n + \tread_replace_refs = 0;\n + \n + \tif (opts.reachable)\n +-\t\treturn write_commit_graph_reachable(opts.obj_dir, flags);\n ++\t\treturn write_commit_graph_reachable(opts.obj_dir, flags, 0);\n + \n + \tstring_list_init(&lines, 0);\n + \tif (opts.stdin_packs || opts.stdin_commits) {\n +@@\n + \tresult = write_commit_graph(opts.obj_dir,\n + \t\t\t\t pack_indexes,\n + \t\t\t\t commit_hex,\n +-\t\t\t\t flags);\n ++\t\t\t\t flags, 0);\n + \n + \tUNLEAK(lines);\n + \treturn result;\n +\n + diff --git a/builtin/commit.c b/builtin/commit.c\n + --- a/builtin/commit.c\n + +++ b/builtin/commit.c\n +@@\n + \t\t \"not exceeded, and then \\\"git reset HEAD\\\" to recover.\"));\n + \n + \tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n +-\t write_commit_graph_reachable(get_object_directory(), 0))\n ++\t write_commit_graph_reachable(get_object_directory(), 0, 0))\n + \t\treturn 1;\n + \n + \trepo_rerere(the_repository, 0);\n +\n + diff --git a/builtin/gc.c b/builtin/gc.c\n + --- a/builtin/gc.c\n + +++ b/builtin/gc.c\n +@@\n + \n + \tif (gc_write_commit_graph &&\n + \t write_commit_graph_reachable(get_object_directory(),\n +-\t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_PROGRESS : 0))\n ++\t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_PROGRESS : 0,\n ++\t\t\t\t\t 0))\n + \t\treturn 1;\n + \n + \tif (auto_gc && too_many_loose_objects())\n +\n diff --git a/commit-graph.c b/commit-graph.c\n --- a/commit-graph.c\n +++ b/commit-graph.c\n @@ -74,18 +123,43 @@\n \tfor (i = 0; i < graph->num_chunks; i++) {\n \t\tuint32_t chunk_id;\n \t\tuint64_t chunk_offset;\n +@@\n + \treturn 0;\n + }\n + \n +-int write_commit_graph_reachable(const char *obj_dir, unsigned int flags)\n ++int write_commit_graph_reachable(const char *obj_dir, unsigned int flags,\n ++\t\t\t\t unsigned char version)\n + {\n + \tstruct string_list list = STRING_LIST_INIT_DUP;\n + \tint result;\n + \n + \tfor_each_ref(add_ref_to_list, &list);\n + \tresult = write_commit_graph(obj_dir, NULL, &list,\n +-\t\t\t\t flags);\n ++\t\t\t\t flags, version);\n + \n + \tstring_list_clear(&list, 0);\n + \treturn result;\n +@@\n + int write_commit_graph(const char *obj_dir,\n + \t\t struct string_list *pack_indexes,\n + \t\t struct string_list *commit_hex,\n +-\t\t unsigned int flags)\n ++\t\t unsigned int flags,\n ++\t\t unsigned char version)\n + {\n + \tstruct packed_oid_list oids;\n + \tstruct packed_commit_list commits;\n @@\n \tint res = 0;\n \tint append = flags & COMMIT_GRAPH_APPEND;\n \tint report_progress = flags & COMMIT_GRAPH_PROGRESS;\n -+\tint version = 0;\n +\tint header_size = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn 0;\n \n -+\tif (flags & COMMIT_GRAPH_VERSION_1)\n -+\t\tversion = 1;\n +\tif (!version)\n +\t\tversion = 1;\n +\tif (version != 1) {\n @@ -132,10 +206,18 @@\n --- a/commit-graph.h\n +++ b/commit-graph.h\n @@\n - \n #define COMMIT_GRAPH_APPEND (1 << 0)\n #define COMMIT_GRAPH_PROGRESS (1 << 1)\n -+#define COMMIT_GRAPH_VERSION_1 (1 << 2)\n \n - int write_commit_graph_reachable(const char *obj_dir, int flags);\n +-int write_commit_graph_reachable(const char *obj_dir, unsigned int flags);\n ++int write_commit_graph_reachable(const char *obj_dir, unsigned int flags,\n ++\t\t\t\t unsigned char version);\n int write_commit_graph(const char *obj_dir,\n + \t\t struct string_list *pack_indexes,\n + \t\t struct string_list *commit_hex,\n +-\t\t unsigned int flags);\n ++\t\t unsigned int flags,\n ++\t\t unsigned char version);\n + \n + int verify_commit_graph(struct repository *r, struct commit_graph *g);\n + \n 4: b1b0c76eb4 ! 4: 6a0e99f9f9 commit-graph: add --version= option\n @@ -62,18 +62,29 @@\n \t};\n \n @@\n + \tif (!opts.obj_dir)\n + \t\topts.obj_dir = get_object_directory();\n \tif (opts.append)\n - \t\tflags |= COMMIT_GRAPH_APPEND;\n +-\t\tflags |= COMMIT_GRAPH_APPEND;\n ++\t\tflags |= COMMIT_GRAPH_APPEND;\t\n \n -+\tswitch (opts.version) {\n -+\tcase 1:\n -+\t\tflags |= COMMIT_GRAPH_VERSION_1;\n -+\t\tbreak;\n -+\t}\n -+\n \tread_replace_refs = 0;\n \n \tif (opts.reachable)\n +-\t\treturn write_commit_graph_reachable(opts.obj_dir, flags, 0);\n ++\t\treturn write_commit_graph_reachable(opts.obj_dir, flags, opts.version);\n + \n + \tstring_list_init(&lines, 0);\n + \tif (opts.stdin_packs || opts.stdin_commits) {\n +@@\n + \tresult = write_commit_graph(opts.obj_dir,\n + \t\t\t\t pack_indexes,\n + \t\t\t\t commit_hex,\n +-\t\t\t\t flags, 0);\n ++\t\t\t\t flags, opts.version);\n + \n + \tUNLEAK(lines);\n + \treturn result;\n \n diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n --- a/t/t5318-commit-graph.sh\n 5: 09362bda1b ! 5: cca8267dfe commit-graph: implement file format version 2\n @@ -23,14 +23,8 @@\n non-zero.\n \n Update the 'git commit-graph read' subcommand to display the new\n - data.\n -\n - Set the default file format version to 2, and adjust the tests to\n - expect the new 'git commit-graph read' output.\n -\n - Add explicit tests for the upgrade path from version 1 to 2. Users\n - with an existing commit-graph with version 1 will seamlessly\n - upgrade to version 2 on their next write.\n + data, and check this output in the test that explicitly writes a\n + v2 commit-graph file.\n \n While we converted the existing 'verify' tests to use a version 1\n file to avoid recalculating data offsets, add explicit 'verify'\n @@ -103,17 +97,6 @@\n \tprintf(\"num_commits: %u\\n\", graph->num_commits);\n \tprintf(\"chunks:\");\n \n -@@\n - \tcase 1:\n - \t\tflags |= COMMIT_GRAPH_VERSION_1;\n - \t\tbreak;\n -+\n -+\tcase 2:\n -+\t\tflags |= COMMIT_GRAPH_VERSION_2;\n -+\t\tbreak;\n - \t}\n - \n - \tread_replace_refs = 0;\n \n diff --git a/commit-graph.c b/commit-graph.c\n --- a/commit-graph.c\n @@ -170,14 +153,9 @@\n \tgraph->hash_len = the_hash_algo->rawsz;\n @@\n \n - \tif (flags & COMMIT_GRAPH_VERSION_1)\n - \t\tversion = 1;\n -+\tif (flags & COMMIT_GRAPH_VERSION_2)\n -+\t\tversion = 2;\n \tif (!version)\n --\t\tversion = 1;\n + \t\tversion = 1;\n -\tif (version != 1) {\n -+\t\tversion = 2;\n +\tif (version <= 0 || version > 2) {\n \t\terror(_(\"unsupported commit-graph version %d\"),\n \t\t version);\n @@ -198,18 +176,6 @@\n \n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \n - diff --git a/commit-graph.h b/commit-graph.h\n - --- a/commit-graph.h\n - +++ b/commit-graph.h\n -@@\n - #define COMMIT_GRAPH_APPEND (1 << 0)\n - #define COMMIT_GRAPH_PROGRESS (1 << 1)\n - #define COMMIT_GRAPH_VERSION_1 (1 << 2)\n -+#define COMMIT_GRAPH_VERSION_2 (1 << 3)\n - \n - int write_commit_graph_reachable(const char *obj_dir, int flags);\n - int write_commit_graph(const char *obj_dir,\n -\n diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n --- a/t/t5318-commit-graph.sh\n +++ b/t/t5318-commit-graph.sh\n @@ -238,37 +204,37 @@\n \tOPTIONAL=\"\"\n \tNUM_CHUNKS=3\n \tif test ! -z $2\n -@@\n - \t\tNUM_CHUNKS=$((3 + $(echo \"$2\" | wc -w)))\n - \tfi\n - \tcat >expect <<- EOF\n --\theader: 43475048 1 1 $NUM_CHUNKS 0\n -+\theader: 43475048 2 $NUM_CHUNKS 1 0\n -+\thash algorithm: 73686131\n - \tnum_commits: $1\n - \tchunks: oid_fanout oid_lookup commit_metadata$OPTIONAL\n - \tEOF\n @@\n \t)\n '\n \n -+test_expect_success 'write v1 graph' '\n -+\tgit commit-graph write --reachable --version=1 &&\n -+\tgit commit-graph verify\n -+'\n -+\n -+graph_git_behavior 'version 1 graph, commit 8 vs merge 2' full commits/8 merge/2\n -+graph_git_behavior 'version 1 graph, commit 8 vs merge 2' full commits/8 merge/2\n ++graph_read_expect_v2 () {\n ++\tOPTIONAL=\"\"\n ++\tNUM_CHUNKS=3\n ++\tif test ! -z $2\n ++\tthen\n ++\t\tOPTIONAL=\" $2\"\n ++\t\tNUM_CHUNKS=$((3 + $(echo \"$2\" | wc -w)))\n ++\tfi\t\n ++\tcat >expect <<- EOF\n ++\theader: 43475048 2 $NUM_CHUNKS 1 0\n ++\thash algorithm: 73686131\n ++\tnum_commits: $1\n ++\tchunks: oid_fanout oid_lookup commit_metadata$OPTIONAL\n ++\tEOF\n ++\tgit commit-graph read >output &&\n ++\ttest_cmp expect output\n ++}\n +\n -+test_expect_success 'upgrade from v1 to v2' '\n -+\tgit checkout -b new-commit-for-upgrade &&\n -+\ttest_commit force-upgrade &&\n ++test_expect_success 'write v2 graph' '\n ++\tcd \"$TRASH_DIRECTORY/full\" &&\n +\tgit commit-graph write --reachable --version=2 &&\n ++\tgraph_read_expect_v2 11 extra_edges &&\n +\tgit commit-graph verify\n +'\n +\n -+graph_git_behavior 'upgraded graph, commit 8 vs merge 2' full commits/8 merge/2\n -+graph_git_behavior 'upgraded graph, commit 8 vs merge 2' full commits/8 merge/2\n ++graph_git_behavior 'version 2 graph, commit 8 vs merge 2' full commits/8 merge/2\n ++graph_git_behavior 'version 2 graph, commit 8 vs merge 2' full commits/8 merge/2\n +\n # the verify tests below expect the commit-graph to contain\n # exactly the commits reachable from the commits/8 branch.\n -: ---------- > 6: e72bca6c78 commit-graph: remove Future Work section\n\n-- \ngitgitgadget\n"},{"id":"374756","messageId":"924b22f990afa750b5e1aa61f4495efc65540107.1556716272.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 2/6] commit-graph: collapse parameters into flags","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-01T13:11:15Z","receivedAt":"2019-05-01T13:11:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() and write_commit_graph_reachable() methods\ncurrently take two boolean parameters: 'append' and 'report_progress'.\nWe will soon expand the possible options to send to these methods, so\ninstead of complicating the parameter list, first simplify it.\n\nCollapse these parameters into a 'flags' parameter, and adjust the\ncallers to provide flags as necessary.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 8 +++++---\n builtin/commit.c | 2 +-\n builtin/gc.c | 4 ++--\n commit-graph.c | 9 +++++----\n commit-graph.h | 8 +++++---\n 5 files changed, 18 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 2e86251f02..828b1a713f 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -142,6 +142,7 @@ static int graph_write(int argc, const char **argv)\n \tstruct string_list *commit_hex = NULL;\n \tstruct string_list lines;\n \tint result;\n+\tint flags = COMMIT_GRAPH_PROGRESS;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n@@ -166,11 +167,13 @@ static int graph_write(int argc, const char **argv)\n \t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n+\tif (opts.append)\n+\t\tflags |= COMMIT_GRAPH_APPEND;\n \n \tread_replace_refs = 0;\n \n \tif (opts.reachable)\n-\t\treturn write_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n+\t\treturn write_commit_graph_reachable(opts.obj_dir, flags);\n \n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n@@ -190,8 +193,7 @@ static int graph_write(int argc, const char **argv)\n \tresult = write_commit_graph(opts.obj_dir,\n \t\t\t\t pack_indexes,\n \t\t\t\t commit_hex,\n-\t\t\t\t opts.append,\n-\t\t\t\t 1);\n+\t\t\t\t flags);\n \n \tUNLEAK(lines);\n \treturn result;\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex b9ea7222fa..b001ef565d 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1670,7 +1670,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\t \"not exceeded, and then \\\"git reset HEAD\\\" to recover.\"));\n \n \tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n-\t write_commit_graph_reachable(get_object_directory(), 0, 0))\n+\t write_commit_graph_reachable(get_object_directory(), 0))\n \t\treturn 1;\n \n \trepo_rerere(the_repository, 0);\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 3984addf73..df2573f124 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -665,8 +665,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (gc_write_commit_graph &&\n-\t write_commit_graph_reachable(get_object_directory(), 0,\n-\t\t\t\t\t !quiet && !daemonized))\n+\t write_commit_graph_reachable(get_object_directory(),\n+\t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_PROGRESS : 0))\n \t\treturn 1;\n \n \tif (auto_gc && too_many_loose_objects())\ndiff --git a/commit-graph.c b/commit-graph.c\nindex ee487a364b..8bbd50658c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -851,15 +851,14 @@ static int add_ref_to_list(const char *refname,\n \treturn 0;\n }\n \n-int write_commit_graph_reachable(const char *obj_dir, int append,\n-\t\t\t\t int report_progress)\n+int write_commit_graph_reachable(const char *obj_dir, unsigned int flags)\n {\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n \tint result;\n \n \tfor_each_ref(add_ref_to_list, &list);\n \tresult = write_commit_graph(obj_dir, NULL, &list,\n-\t\t\t\t append, report_progress);\n+\t\t\t\t flags);\n \n \tstring_list_clear(&list, 0);\n \treturn result;\n@@ -868,7 +867,7 @@ int write_commit_graph_reachable(const char *obj_dir, int append,\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t int append, int report_progress)\n+\t\t unsigned int flags)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n@@ -887,6 +886,8 @@ int write_commit_graph(const char *obj_dir,\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tunsigned long approx_nr_objects;\n \tint res = 0;\n+\tint append = flags & COMMIT_GRAPH_APPEND;\n+\tint report_progress = flags & COMMIT_GRAPH_PROGRESS;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn 0;\ndiff --git a/commit-graph.h b/commit-graph.h\nindex d15670bf46..70f4caf0c7 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -65,12 +65,14 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n */\n int generation_numbers_enabled(struct repository *r);\n \n-int write_commit_graph_reachable(const char *obj_dir, int append,\n-\t\t\t\t int report_progress);\n+#define COMMIT_GRAPH_APPEND (1 << 0)\n+#define COMMIT_GRAPH_PROGRESS (1 << 1)\n+\n+int write_commit_graph_reachable(const char *obj_dir, unsigned int flags);\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t int append, int report_progress);\n+\t\t unsigned int flags);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \n-- \ngitgitgadget\n\n"},{"id":"374757","messageId":"8446011a43db7482c1858861d516749847555c05.1556716273.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 3/6] commit-graph: create new version parameter","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-01T13:11:16Z","receivedAt":"2019-05-01T13:11:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn anticipation of a new commit-graph file format version, create\na parameter for the write_commit_graph() and write_commit_graph_reachable()\nmethods to take a version number.\n\nWhen the given version is zero, the implementation selects a\ndefault value. Currently, the only valid value is 1.\n\nThe file format will change the header information, so place the\nexisting header logic inside a switch statement with only one case.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 4 +--\n builtin/commit.c | 2 +-\n builtin/gc.c | 3 +-\n commit-graph.c | 63 +++++++++++++++++++++++++++---------------\n commit-graph.h | 6 ++--\n 5 files changed, 50 insertions(+), 28 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 828b1a713f..7d9185dfc2 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -173,7 +173,7 @@ static int graph_write(int argc, const char **argv)\n \tread_replace_refs = 0;\n \n \tif (opts.reachable)\n-\t\treturn write_commit_graph_reachable(opts.obj_dir, flags);\n+\t\treturn write_commit_graph_reachable(opts.obj_dir, flags, 0);\n \n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n@@ -193,7 +193,7 @@ static int graph_write(int argc, const char **argv)\n \tresult = write_commit_graph(opts.obj_dir,\n \t\t\t\t pack_indexes,\n \t\t\t\t commit_hex,\n-\t\t\t\t flags);\n+\t\t\t\t flags, 0);\n \n \tUNLEAK(lines);\n \treturn result;\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex b001ef565d..b9ea7222fa 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1670,7 +1670,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\t \"not exceeded, and then \\\"git reset HEAD\\\" to recover.\"));\n \n \tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n-\t write_commit_graph_reachable(get_object_directory(), 0))\n+\t write_commit_graph_reachable(get_object_directory(), 0, 0))\n \t\treturn 1;\n \n \trepo_rerere(the_repository, 0);\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex df2573f124..41637242b1 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -666,7 +666,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \n \tif (gc_write_commit_graph &&\n \t write_commit_graph_reachable(get_object_directory(),\n-\t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_PROGRESS : 0))\n+\t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_PROGRESS : 0,\n+\t\t\t\t\t 0))\n \t\treturn 1;\n \n \tif (auto_gc && too_many_loose_objects())\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8bbd50658c..b6f09f1be2 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -25,9 +25,6 @@\n \n #define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n \n-#define GRAPH_VERSION_1 0x1\n-#define GRAPH_VERSION GRAPH_VERSION_1\n-\n #define GRAPH_EXTRA_EDGES_NEEDED 0x80000000\n #define GRAPH_EDGE_LAST_MASK 0x7fffffff\n #define GRAPH_PARENT_NONE 0x70000000\n@@ -173,30 +170,35 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t}\n \n \tgraph_version = *(unsigned char*)(data + 4);\n-\tif (graph_version != GRAPH_VERSION) {\n+\tif (graph_version != 1) {\n \t\terror(_(\"commit-graph version %X does not match version %X\"),\n-\t\t graph_version, GRAPH_VERSION);\n-\t\treturn NULL;\n-\t}\n-\n-\thash_version = *(unsigned char*)(data + 5);\n-\tif (hash_version != oid_version()) {\n-\t\terror(_(\"commit-graph hash version %X does not match version %X\"),\n-\t\t hash_version, oid_version());\n+\t\t graph_version, 1);\n \t\treturn NULL;\n \t}\n \n \tgraph = alloc_commit_graph();\n \n+\tswitch (graph_version) {\n+\tcase 1:\n+\t\thash_version = *(unsigned char*)(data + 5);\n+\t\tif (hash_version != oid_version()) {\n+\t\t\terror(_(\"commit-graph hash version %X does not match version %X\"),\n+\t\t\t hash_version, oid_version());\n+\t\t\treturn NULL;\n+\t\t}\n+\n+\t\tgraph->num_chunks = *(unsigned char*)(data + 6);\n+\t\tchunk_lookup = data + 8;\n+\t\tbreak;\n+\t}\n+\n \tgraph->hash_len = the_hash_algo->rawsz;\n-\tgraph->num_chunks = *(unsigned char*)(data + 6);\n \tgraph->graph_fd = fd;\n \tgraph->data = graph_map;\n \tgraph->data_len = graph_size;\n \n \tlast_chunk_id = 0;\n \tlast_chunk_offset = 8;\n-\tchunk_lookup = data + 8;\n \tfor (i = 0; i < graph->num_chunks; i++) {\n \t\tuint32_t chunk_id;\n \t\tuint64_t chunk_offset;\n@@ -851,14 +853,15 @@ static int add_ref_to_list(const char *refname,\n \treturn 0;\n }\n \n-int write_commit_graph_reachable(const char *obj_dir, unsigned int flags)\n+int write_commit_graph_reachable(const char *obj_dir, unsigned int flags,\n+\t\t\t\t unsigned char version)\n {\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n \tint result;\n \n \tfor_each_ref(add_ref_to_list, &list);\n \tresult = write_commit_graph(obj_dir, NULL, &list,\n-\t\t\t\t flags);\n+\t\t\t\t flags, version);\n \n \tstring_list_clear(&list, 0);\n \treturn result;\n@@ -867,7 +870,8 @@ int write_commit_graph_reachable(const char *obj_dir, unsigned int flags)\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t unsigned int flags)\n+\t\t unsigned int flags,\n+\t\t unsigned char version)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n@@ -888,10 +892,19 @@ int write_commit_graph(const char *obj_dir,\n \tint res = 0;\n \tint append = flags & COMMIT_GRAPH_APPEND;\n \tint report_progress = flags & COMMIT_GRAPH_PROGRESS;\n+\tint header_size = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn 0;\n \n+\tif (!version)\n+\t\tversion = 1;\n+\tif (version != 1) {\n+\t\terror(_(\"unsupported commit-graph version %d\"),\n+\t\t version);\n+\t\treturn 1;\n+\t}\n+\n \toids.nr = 0;\n \tapprox_nr_objects = approximate_object_count();\n \toids.alloc = approx_nr_objects / 32;\n@@ -1076,10 +1089,16 @@ int write_commit_graph(const char *obj_dir,\n \n \thashwrite_be32(f, GRAPH_SIGNATURE);\n \n-\thashwrite_u8(f, GRAPH_VERSION);\n-\thashwrite_u8(f, oid_version());\n-\thashwrite_u8(f, num_chunks);\n-\thashwrite_u8(f, 0); /* unused padding byte */\n+\thashwrite_u8(f, version);\n+\n+\tswitch (version) {\n+\tcase 1:\n+\t\thashwrite_u8(f, oid_version());\n+\t\thashwrite_u8(f, num_chunks);\n+\t\thashwrite_u8(f, 0); /* unused padding byte */\n+\t\theader_size = 8;\n+\t\tbreak;\n+\t}\n \n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n@@ -1090,7 +1109,7 @@ int write_commit_graph(const char *obj_dir,\n \t\tchunk_ids[3] = 0;\n \tchunk_ids[4] = 0;\n \n-\tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n+\tchunk_offsets[0] = header_size + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n \tchunk_offsets[2] = chunk_offsets[1] + hashsz * commits.nr;\n \tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * commits.nr;\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 70f4caf0c7..d64a2cc78c 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -68,11 +68,13 @@ int generation_numbers_enabled(struct repository *r);\n #define COMMIT_GRAPH_APPEND (1 << 0)\n #define COMMIT_GRAPH_PROGRESS (1 << 1)\n \n-int write_commit_graph_reachable(const char *obj_dir, unsigned int flags);\n+int write_commit_graph_reachable(const char *obj_dir, unsigned int flags,\n+\t\t\t\t unsigned char version);\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t unsigned int flags);\n+\t\t unsigned int flags,\n+\t\t unsigned char version);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \n-- \ngitgitgadget\n\n"},{"id":"374759","messageId":"cca8267dfeb382005ae3c5a4935f2281dabf26b4.1556716273.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 5/6] commit-graph: implement file format version 2","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-01T13:11:18Z","receivedAt":"2019-05-01T13:11:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe commit-graph file format had some shortcomings which we now\ncorrect:\n\n 1. The hash algorithm was determined by a single byte, instead\n of the 4-byte format identifier.\n\n 2. There was no way to update the reachability index we used.\n We currently only support generation numbers, but that will\n change in the future.\n\n 3. Git did not fail with error if the unused eighth byte was\n non-zero, so we could not use that to indicate an incremental\n file format without breaking compatibility across versions.\n\nThe new format modifies the header of the commit-graph to solve\nthese problems. We use the 4-byte hash format id, freeing up a byte\nin our 32-bit alignment to introduce a reachability index version.\nWe can also fail to read the commit-graph if the eighth byte is\nnon-zero.\n\nUpdate the 'git commit-graph read' subcommand to display the new\ndata, and check this output in the test that explicitly writes a\nv2 commit-graph file.\n\nWhile we converted the existing 'verify' tests to use a version 1\nfile to avoid recalculating data offsets, add explicit 'verify'\ntests on a version 2 file that corrupt the new header values.\n\nSigned-off-by: Derrick Stolee \n---\n .../technical/commit-graph-format.txt | 26 ++++++-\n builtin/commit-graph.c | 5 ++\n commit-graph.c | 39 +++++++++-\n t/t5318-commit-graph.sh | 73 +++++++++++++++++--\n 4 files changed, 134 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex 16452a0504..e367aa94b1 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -31,13 +31,22 @@ and hash type.\n \n All 4-byte numbers are in network order.\n \n+There are two versions available, 1 and 2. These currently differ only in\n+the header.\n+\n HEADER:\n \n+All commit-graph files use the first five bytes for the same purpose.\n+\n 4-byte signature:\n The signature is: {'C', 'G', 'P', 'H'}\n \n 1-byte version number:\n- Currently, the only valid version is 1.\n+ Currently, the valid version numbers are 1 and 2.\n+\n+The remainder of the header changes depending on the version.\n+\n+Version 1:\n \n 1-byte Hash Version (1 = SHA-1)\n We infer the hash length (H) from this value.\n@@ -47,6 +56,21 @@ HEADER:\n 1-byte (reserved for later use)\n Current clients should ignore this value.\n \n+Version 2:\n+\n+ 1-byte number (C) of \"chunks\"\n+\n+ 1-byte reachability index version number:\n+ Currently, the only valid number is 1.\n+\n+ 1-byte (reserved for later use)\n+ Current clients expect this value to be zero, and will not\n+ try to read the commit-graph file if it is non-zero.\n+\n+ 4-byte format identifier for the hash algorithm:\n+ If this identifier does not agree with the repository's current\n+ hash algorithm, then the client will not read the commit graph.\n+\n CHUNK LOOKUP:\n \n (C + 1) * 12 bytes listing the table of contents for the chunks:\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex e766dd076e..7df6688b08 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -117,6 +117,11 @@ static int graph_read(int argc, const char **argv)\n \t\t*(unsigned char*)(graph->data + 5),\n \t\t*(unsigned char*)(graph->data + 6),\n \t\t*(unsigned char*)(graph->data + 7));\n+\n+\tif (*(unsigned char *)(graph->data + 4) == 2)\n+\t\tprintf(\"hash algorithm: %X\\n\",\n+\t\t get_be32(graph->data + 8));\n+\n \tprintf(\"num_commits: %u\\n\", graph->num_commits);\n \tprintf(\"chunks:\");\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex b6f09f1be2..5eebba6a0f 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -152,7 +152,8 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \tuint64_t last_chunk_offset;\n \tuint32_t last_chunk_id;\n \tuint32_t graph_signature;\n-\tunsigned char graph_version, hash_version;\n+\tunsigned char graph_version, hash_version, reach_index_version;\n+\tuint32_t hash_id;\n \n \tif (!graph_map)\n \t\treturn NULL;\n@@ -170,7 +171,7 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t}\n \n \tgraph_version = *(unsigned char*)(data + 4);\n-\tif (graph_version != 1) {\n+\tif (!graph_version || graph_version > 2) {\n \t\terror(_(\"commit-graph version %X does not match version %X\"),\n \t\t graph_version, 1);\n \t\treturn NULL;\n@@ -190,6 +191,30 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t\tgraph->num_chunks = *(unsigned char*)(data + 6);\n \t\tchunk_lookup = data + 8;\n \t\tbreak;\n+\n+\tcase 2:\n+\t\tgraph->num_chunks = *(unsigned char *)(data + 5);\n+\n+\t\treach_index_version = *(unsigned char *)(data + 6);\n+\t\tif (reach_index_version != 1) {\n+\t\t\terror(_(\"unsupported reachability index version %d\"),\n+\t\t\t reach_index_version);\n+\t\t\treturn NULL;\n+\t\t}\n+\n+\t\tif (*(unsigned char*)(data + 7)) {\n+\t\t\terror(_(\"unsupported value in commit-graph header\"));\n+\t\t\treturn NULL;\n+\t\t}\n+\n+\t\thash_id = get_be32(data + 8);\n+\t\tif (hash_id != the_hash_algo->format_id) {\n+\t\t\terror(_(\"commit-graph hash algorithm does not match current algorithm\"));\n+\t\t\treturn NULL;\n+\t\t}\n+\n+\t\tchunk_lookup = data + 12;\n+\t\tbreak;\n \t}\n \n \tgraph->hash_len = the_hash_algo->rawsz;\n@@ -899,7 +924,7 @@ int write_commit_graph(const char *obj_dir,\n \n \tif (!version)\n \t\tversion = 1;\n-\tif (version != 1) {\n+\tif (version <= 0 || version > 2) {\n \t\terror(_(\"unsupported commit-graph version %d\"),\n \t\t version);\n \t\treturn 1;\n@@ -1098,6 +1123,14 @@ int write_commit_graph(const char *obj_dir,\n \t\thashwrite_u8(f, 0); /* unused padding byte */\n \t\theader_size = 8;\n \t\tbreak;\n+\n+\tcase 2:\n+\t\thashwrite_u8(f, num_chunks);\n+\t\thashwrite_u8(f, 1); /* reachability index version */\n+\t\thashwrite_u8(f, 0); /* unused padding byte */\n+\t\thashwrite_be32(f, the_hash_algo->format_id);\n+\t\theader_size = 12;\n+\t\tbreak;\n \t}\n \n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 4eb5a09ef3..373a6cd0d4 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -33,13 +33,13 @@ test_expect_success 'create commits and repack' '\n \tgit repack\n '\n \n-graph_git_two_modes() {\n+graph_git_two_modes () {\n \tgit -c core.commitGraph=true $1 >output\n \tgit -c core.commitGraph=false $1 >expect\n \ttest_cmp expect output\n }\n \n-graph_git_behavior() {\n+graph_git_behavior () {\n \tMSG=$1\n \tDIR=$2\n \tBRANCH=$3\n@@ -56,7 +56,7 @@ graph_git_behavior() {\n \n graph_git_behavior 'no graph' full commits/3 commits/1\n \n-graph_read_expect() {\n+graph_read_expect () {\n \tOPTIONAL=\"\"\n \tNUM_CHUNKS=3\n \tif test ! -z $2\n@@ -320,6 +320,34 @@ test_expect_success 'replace-objects invalidates commit-graph' '\n \t)\n '\n \n+graph_read_expect_v2 () {\n+\tOPTIONAL=\"\"\n+\tNUM_CHUNKS=3\n+\tif test ! -z $2\n+\tthen\n+\t\tOPTIONAL=\" $2\"\n+\t\tNUM_CHUNKS=$((3 + $(echo \"$2\" | wc -w)))\n+\tfi\t\n+\tcat >expect <<- EOF\n+\theader: 43475048 2 $NUM_CHUNKS 1 0\n+\thash algorithm: 73686131\n+\tnum_commits: $1\n+\tchunks: oid_fanout oid_lookup commit_metadata$OPTIONAL\n+\tEOF\n+\tgit commit-graph read >output &&\n+\ttest_cmp expect output\n+}\n+\n+test_expect_success 'write v2 graph' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\tgit commit-graph write --reachable --version=2 &&\n+\tgraph_read_expect_v2 11 extra_edges &&\n+\tgit commit-graph verify\n+'\n+\n+graph_git_behavior 'version 2 graph, commit 8 vs merge 2' full commits/8 merge/2\n+graph_git_behavior 'version 2 graph, commit 8 vs merge 2' full commits/8 merge/2\n+\n # the verify tests below expect the commit-graph to contain\n # exactly the commits reachable from the commits/8 branch.\n # If the file changes the set of commits in the list, then the\n@@ -392,7 +420,7 @@ corrupt_graph_verify() {\n # starting at , then runs 'git commit-graph verify'\n # and places the output in the file 'err'. Test 'err' for\n # the given string.\n-corrupt_graph_and_verify() {\n+corrupt_graph_and_verify () {\n \tpos=$1\n \tdata=\"${2:-\\0}\"\n \tgrepstr=$3\n@@ -424,10 +452,14 @@ test_expect_success 'detect bad signature' '\n '\n \n test_expect_success 'detect bad version' '\n-\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\02\" \\\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\03\" \\\n \t\t\"graph version\"\n '\n \n+test_expect_success 'detect version 2 with version 1 data' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_VERSION \"\\02\" \\\n+\t\t\"reachability index version\"\n+'\n test_expect_success 'detect bad hash version' '\n \tcorrupt_graph_and_verify $GRAPH_BYTE_HASH \"\\02\" \\\n \t\t\"hash version\"\n@@ -532,6 +564,37 @@ test_expect_success 'git fsck (checks commit-graph)' '\n \ttest_must_fail git fsck\n '\n \n+test_expect_success 'rewrite commmit-graph with version 2' '\n+\trm -f .git/objects/info/commit-graph &&\n+\tgit commit-graph write --reachable --version=2 &&\n+\tgit commit-graph verify\n+'\n+\n+GRAPH_BYTE_CHUNK_COUNT=5\n+GRAPH_BYTE_REACH_INDEX=6\n+GRAPH_BYTE_UNUSED=7\n+GRAPH_BYTE_HASH=8\n+\n+test_expect_success 'detect low chunk count (v2)' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_CHUNK_COUNT \"\\02\" \\\n+\t\t\"missing the .* chunk\"\n+'\n+\n+test_expect_success 'detect incorrect reachability index' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_REACH_INDEX \"\\03\" \\\n+\t\t\"reachability index version\"\n+'\n+\n+test_expect_success 'detect non-zero unused byte' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_UNUSED \"\\01\" \\\n+\t\t\"unsupported value\"\n+'\n+\n+test_expect_success 'detect bad hash version (v2)' '\n+\tcorrupt_graph_and_verify $GRAPH_BYTE_HASH \"\\00\" \\\n+\t\t\"hash algorithm\"\n+'\n+\n test_expect_success 'setup non-the_repository tests' '\n \trm -rf repo &&\n \tgit init repo &&\n-- \ngitgitgadget\n\n"},{"id":"374758","messageId":"6a0e99f9f9d9027f95a47ec97b3bbafda4235522.1556716273.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 4/6] commit-graph: add --version= option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-01T13:11:17Z","receivedAt":"2019-05-01T13:11:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAllow the commit-graph builtin to specify the file format version\nusing the '--version=' option. Specify the version exactly in\nthe verification tests as using a different version would change\nthe offsets used in those tests.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 3 +++\n builtin/commit-graph.c | 13 ++++++++-----\n t/t5318-commit-graph.sh | 2 +-\n 3 files changed, 12 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 624470e198..1d1cc70de4 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -51,6 +51,9 @@ or `--stdin-packs`.)\n +\n With the `--append` option, include all commits that are present in the\n existing commit-graph file.\n++\n+With the `--version=` option, specify the file format version. Used\n+only for testing.\n \n 'read'::\n \ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 7d9185dfc2..e766dd076e 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -10,7 +10,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph verify [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits] [--version=]\"),\n \tNULL\n };\n \n@@ -25,7 +25,7 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits] [--version=]\"),\n \tNULL\n };\n \n@@ -35,6 +35,7 @@ static struct opts_commit_graph {\n \tint stdin_packs;\n \tint stdin_commits;\n \tint append;\n+\tint version;\n } opts;\n \n \n@@ -156,6 +157,8 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"start walk at commits listed by stdin\")),\n \t\tOPT_BOOL(0, \"append\", &opts.append,\n \t\t\tN_(\"include all commits already in the commit-graph file\")),\n+\t\tOPT_INTEGER(0, \"version\", &opts.version,\n+\t\t\tN_(\"specify the file format version\")),\n \t\tOPT_END(),\n \t};\n \n@@ -168,12 +171,12 @@ static int graph_write(int argc, const char **argv)\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n \tif (opts.append)\n-\t\tflags |= COMMIT_GRAPH_APPEND;\n+\t\tflags |= COMMIT_GRAPH_APPEND;\t\n \n \tread_replace_refs = 0;\n \n \tif (opts.reachable)\n-\t\treturn write_commit_graph_reachable(opts.obj_dir, flags, 0);\n+\t\treturn write_commit_graph_reachable(opts.obj_dir, flags, opts.version);\n \n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n@@ -193,7 +196,7 @@ static int graph_write(int argc, const char **argv)\n \tresult = write_commit_graph(opts.obj_dir,\n \t\t\t\t pack_indexes,\n \t\t\t\t commit_hex,\n-\t\t\t\t flags, 0);\n+\t\t\t\t flags, opts.version);\n \n \tUNLEAK(lines);\n \treturn result;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex e80c1cac02..4eb5a09ef3 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -328,7 +328,7 @@ test_expect_success 'replace-objects invalidates commit-graph' '\n \n test_expect_success 'git commit-graph verify' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n-\tgit rev-parse commits/8 | git commit-graph write --stdin-commits &&\n+\tgit rev-parse commits/8 | git commit-graph write --stdin-commits --version=1 &&\n \tgit commit-graph verify >output\n '\n \n-- \ngitgitgadget\n\n"},{"id":"374760","messageId":"e72bca6c784c5fc727ae4cf661526eec346c478c.1556716273.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 6/6] commit-graph: remove Future Work section","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-01T13:11:19Z","receivedAt":"2019-05-01T13:11:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe commit-graph feature began with a long list of planned\nbenefits, most of which are now complete. The future work\nsection has only a few items left.\n\nAs for making more algorithms aware of generation numbers,\nsome are only waiting for generation number v2 to ensure the\nperformance matches the existing behavior using commit date.\n\nIt is unlikely that we will ever send a commit-graph file\nas part of the protocol, since we would need to verify the\ndata, and that is as expensive as writing a commit-graph from\nscratch. If we want to start trusting remote content, then\nthat item can be investigated again.\n\nWhile there is more work to be done on the feature, having\na section of the docs devoted to a TODO list is wasteful and\nhard to keep up-to-date.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 17 -----------------\n 1 file changed, 17 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex 7805b0968c..fb53341d5e 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -127,23 +127,6 @@ Design Details\n helpful for these clones, anyway. The commit-graph will not be read or\n written when shallow commits are present.\n \n-Future Work\n------------\n-\n-- After computing and storing generation numbers, we must make graph\n- walks aware of generation numbers to gain the performance benefits they\n- enable. This will mostly be accomplished by swapping a commit-date-ordered\n- priority queue with one ordered by generation number. The following\n- operations are important candidates:\n-\n- - 'log --topo-order'\n- - 'tag --merged'\n-\n-- A server could provide a commit-graph file as part of the network protocol\n- to avoid extra calculations by clients. This feature is only of benefit if\n- the user is willing to trust the file, because verifying the file is correct\n- is as hard as computing it from scratch.\n-\n Related Links\n -------------\n [0] https://bugs.chromium.org/p/git/issues/detail?id=8\n-- \ngitgitgadget\n"},{"id":"374764","messageId":"87pnp2qlkv.fsf@evledraar.gmail.com","threadId":"50302","inReplyTo":"91f300ec0acd825494fa13d93cdc4770bbf2ac14.1556716272.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/6] commit-graph: return with errors during write","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-05-01T14:46:24Z","receivedAt":"2019-05-01T14:46:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, May 01 2019, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> The write_commit_graph() method uses die() to report failure and\n> exit when confronted with an unexpected condition. This use of\n> die() in a library function is incorrect and is now replaced by\n> error() statements and an int return type.\n>\n> Now that we use 'goto cleanup' to jump to the terminal condition\n> on an error, we have new paths that could lead to uninitialized\n> values. New initializers are added to correct for this.\n>\n> The builtins 'commit-graph', 'gc', and 'commit' call these methods,\n> so update them to check the return value.\n\nSeems good to have a test to check for some of this behavior. I see that\ncan be done as just:\n\n echo doesnotexist | git commit-graph write --stdin-packs\n\nAnd checking the exit code is 1 as it is now, not 128.\n"},{"id":"374766","messageId":"87o94mql0a.fsf@evledraar.gmail.com","threadId":"50302","inReplyTo":"e72bca6c784c5fc727ae4cf661526eec346c478c.1556716273.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 6/6] commit-graph: remove Future Work section","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-05-01T14:58:45Z","receivedAt":"2019-05-01T14:58:50Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, May 01 2019, Derrick Stolee via GitGitGadget wrote:\n\n> The commit-graph feature began with a long list of planned\n> benefits, most of which are now complete. The future work\n> section has only a few items left.\n>\n> As for making more algorithms aware of generation numbers,\n> some are only waiting for generation number v2 to ensure the\n> performance matches the existing behavior using commit date.\n>\n> It is unlikely that we will ever send a commit-graph file\n> as part of the protocol, since we would need to verify the\n> data, and that is as expensive as writing a commit-graph from\n> scratch. If we want to start trusting remote content, then\n> that item can be investigated again.\n\nMy best of 3 times for \"write\" followed by \"verify\" on linux.git are\n8.7/7.9 real/user for \"write\" and 5.2/4.9 real/user for \"write\".\n\nSo that's a reduction of ~40%. I have another big in-house repo where I\nget similar numbers of 17/16 for \"write\" and 10/9 for \"verify\". Both for\na commit-graph file on the order of 50MB where it would be quicker for\nme to download and verify it if the protocol supported it.\n\nI'm not clamoring to make it part of the protocol, but the claim that\n\"verify\" needs to do the equivalent of \"write\" seems to be demonstrably\nwrong, or perhaps \"verify\" isn't doing all the work it should be doing?\n\n> While there is more work to be done on the feature, having\n> a section of the docs devoted to a TODO list is wasteful and\n> hard to keep up-to-date.\n\nAgreed, whatever we decide to do in the future I think it makes sense to\nremove this section from the docs, although perhaps the commit message\nshould be amended per the above :)\n\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/technical/commit-graph.txt | 17 -----------------\n> 1 file changed, 17 deletions(-)\n>\n> diff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\n> index 7805b0968c..fb53341d5e 100644\n> --- a/Documentation/technical/commit-graph.txt\n> +++ b/Documentation/technical/commit-graph.txt\n> @@ -127,23 +127,6 @@ Design Details\n> helpful for these clones, anyway. The commit-graph will not be read or\n> written when shallow commits are present.\n>\n> -Future Work\n> ------------\n> -\n> -- After computing and storing generation numbers, we must make graph\n> - walks aware of generation numbers to gain the performance benefits they\n> - enable. This will mostly be accomplished by swapping a commit-date-ordered\n> - priority queue with one ordered by generation number. The following\n> - operations are important candidates:\n> -\n> - - 'log --topo-order'\n> - - 'tag --merged'\n> -\n> -- A server could provide a commit-graph file as part of the network protocol\n> - to avoid extra calculations by clients. This feature is only of benefit if\n> - the user is willing to trust the file, because verifying the file is correct\n> - is as hard as computing it from scratch.\n> -\n> Related Links\n> -------------\n> [0] https://bugs.chromium.org/p/git/issues/detail?id=8\n"},{"id":"374781","messageId":"87muk6q98k.fsf@evledraar.gmail.com","threadId":"50302","inReplyTo":"cca8267dfeb382005ae3c5a4935f2281dabf26b4.1556716273.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 5/6] commit-graph: implement file format version 2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-05-01T19:12:59Z","receivedAt":"2019-05-01T19:13:04Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, May 01 2019, Derrick Stolee via GitGitGadget wrote:\n\n> 3. Git did not fail with error if the unused eighth byte was\n> non-zero, so we could not use that to indicate an incremental\n> file format without breaking compatibility across versions.\n\nThis isn't new, I just missed this the last time around. I don't see how\nthis makes any sense.\n\nOn the current v1 graph code you can apply this patch and everything\ncontinues to work because we ignore this padding byte:\n\n - hashwrite_u8(f, 0); /* unused padding byte */\n + hashwrite_u8(f, 1); /* unused padding byte */\n\nBut now after ab/commit-graph-fixes just got finished fixed the version\nbump being a hard error of:\n\n error: graph version 2 does not match version 1\n\nThis v2 code is basically, as I understand it, introducing two ways of\nexpressing the version, so e.g. we might have v2 graphs with \"0\" here\nchanged to \"1\" for an incremental version of \"2.1\".\n\nOK, let's try that then, on top of this series:\n\n diff --git a/commit-graph.c b/commit-graph.c\n index 5eebba6a0f..36c8cdb950 100644\n --- a/commit-graph.c\n +++ b/commit-graph.c\n @@ -1127,7 +1127,7 @@ int write_commit_graph(const char *obj_dir,\n case 2:\n hashwrite_u8(f, num_chunks);\n hashwrite_u8(f, 1); /* reachability index version */\n - hashwrite_u8(f, 0); /* unused padding byte */\n + hashwrite_u8(f, 1); /* unused padding byte */\n\nThen:\n\n $ ~/g/git/git --exec-path=$PWD commit-graph write --version=2; ~/g/git/git --exec-path=$PWD status\n Expanding reachable commits in commit graph: 100% (201645/201645), done.\n Computing commit graph generation numbers: 100% (200556/200556), done.\n error: unsupported value in commit-graph header\n HEAD detached at pr-112/derrickstolee/graph/v2-head-v3\n\nSo we'll error out in the same way as if \"2.0\" was changed to \"3.0\" with\nthis \"2.1\" change, just with a more cryptic error message on this new v2\ncode.\n\nI don't see how this is meaningfully different from just bumping the\nversion to \"3\". We abort parsing the graph just like with major version\nchanges.\n"},{"id":"374782","messageId":"4cf34654-4b2d-ab5c-40ae-b47dc9502135@gmail.com","threadId":"50302","inReplyTo":"87muk6q98k.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 5/6] commit-graph: implement file format version 2","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-01T19:56:08Z","receivedAt":"2019-05-01T19:56:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/1/2019 3:12 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> OK, let's try that then, on top of this series:\n> \n> diff --git a/commit-graph.c b/commit-graph.c\n> index 5eebba6a0f..36c8cdb950 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -1127,7 +1127,7 @@ int write_commit_graph(const char *obj_dir,\n> case 2:\n> hashwrite_u8(f, num_chunks);\n> hashwrite_u8(f, 1); /* reachability index version */\n> - hashwrite_u8(f, 0); /* unused padding byte */\n> + hashwrite_u8(f, 1); /* unused padding byte */\n> \n> Then:\n> \n> $ ~/g/git/git --exec-path=$PWD commit-graph write --version=2; ~/g/git/git --exec-path=$PWD status\n> Expanding reachable commits in commit graph: 100% (201645/201645), done.\n> Computing commit graph generation numbers: 100% (200556/200556), done.\n> error: unsupported value in commit-graph header\n> HEAD detached at pr-112/derrickstolee/graph/v2-head-v3\n> \n> So we'll error out in the same way as if \"2.0\" was changed to \"3.0\" with\n> this \"2.1\" change, just with a more cryptic error message on this new v2\n> code.\n> \n> I don't see how this is meaningfully different from just bumping the\n> version to \"3\". We abort parsing the graph just like with major version\n> changes.\n\nHaving a non-zero value here doesn't really mean \"2.1\" or \"3\". But I understand\nyour apprehension.\n\nI'm currently working on building the incremental file format, based on\nthis series. This \"unused\" byte will be used to say \"how many base commit-graph\nfiles does this graph have?\" If non-zero, we do not currently understand\nhow to stitch these files together into a \"combined\" graph at run time,\nso we should fail.\n\nIf we should never have an intermediate version of Git that doesn't\nunderstand this byte, then this series can wait until that feature is ready.\n\nThanks,\n-Stolee\n\n"},{"id":"374783","messageId":"c1c91314-477e-ffc7-25f5-fb46bc64db4e@gmail.com","threadId":"50302","inReplyTo":"87o94mql0a.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 6/6] commit-graph: remove Future Work section","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-01T19:59:36Z","receivedAt":"2019-05-01T19:59:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/1/2019 10:58 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, May 01 2019, Derrick Stolee via GitGitGadget wrote:\n> \n>> The commit-graph feature began with a long list of planned\n>> benefits, most of which are now complete. The future work\n>> section has only a few items left.\n>>\n>> As for making more algorithms aware of generation numbers,\n>> some are only waiting for generation number v2 to ensure the\n>> performance matches the existing behavior using commit date.\n>>\n>> It is unlikely that we will ever send a commit-graph file\n>> as part of the protocol, since we would need to verify the\n>> data, and that is as expensive as writing a commit-graph from\n>> scratch. If we want to start trusting remote content, then\n>> that item can be investigated again.\n> \n> My best of 3 times for \"write\" followed by \"verify\" on linux.git are\n> 8.7/7.9 real/user for \"write\" and 5.2/4.9 real/user for \"write\".\n> \n> So that's a reduction of ~40%. I have another big in-house repo where I\n> get similar numbers of 17/16 for \"write\" and 10/9 for \"verify\". Both for\n> a commit-graph file on the order of 50MB where it would be quicker for\n> me to download and verify it if the protocol supported it.\n\nKeep in mind that your first \"write\" may have warmed up the file system\nand your pack-files parsed faster the second time around.\n\nYou are right though, 'verify' doesn't do these things:\n\n1. Sort a list of OIDs.\n2. Write a file.\n\nAnd perhaps some other things. I should mean that \"the main task of\n'git commit-graph verify' is to parse commits from the object store,\nand this is the most expensive operation in 'git commit-graph write'.\"\n\nThanks,\n-Stolee\n\n"},{"id":"374784","messageId":"87lfzprkfc.fsf@evledraar.gmail.com","threadId":"50302","inReplyTo":"pull.112.v3.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 0/6] Create commit-graph file format v2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-05-01T20:25:59Z","receivedAt":"2019-05-01T20:26:05Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, May 01 2019, Derrick Stolee via GitGitGadget wrote:\n\n> The commit-graph file format has some shortcomings that were discussed\n> on-list:\n>\n> 1. It doesn't use the 4-byte format ID from the_hash_algo.\n>\n> 2. There is no way to change the reachability index from generation numbers\n> to corrected commit date [1].\n>\n> 3. The unused byte in the format could be used to signal the file is\n> incremental, but current clients ignore the value even if it is\n> non-zero.\n>\n> This series adds a new version (2) to the commit-graph file. The fifth byte\n> already specified the file format, so existing clients will gracefully\n> respond to files with a different version number. The only real change now\n> is that the header takes 12 bytes instead of 8, due to using the 4-byte\n> format ID for the hash algorithm.\n>\n> The new bytes reserved for the reachability index version and incremental\n> file formats are now expected to be equal to the defaults. When we update\n> these values to be flexible in the future, if a client understands\n> commit-graph v2 but not those new values, then it will fail gracefully.\n>\n> NOTE: this series was rebased onto ab/commit-graph-fixes, as the conflicts\n> were significant and subtle.\n>\n> Updates in V3: Thanks for all the feedback so far!\n>\n> * Moved the version information into an unsigned char parameter, instead of\n> a flag.\n>\n> * We no longer default to the v2 file format, as that will break users who\n> downgrade. This required some changes to the test script.\n>\n> * Removed the \"Future work\" section from the commit-graph design document\n> in a new patch.\n>\n> * I did not change the file name for v2 file formats, as Ævar suggested.\n> I'd like the discussion to continue on this topic.\n\nI won't repeat my outstanding v2 feedback about v1 & v2\nincompatibilities, except to say that I'd in principle be fine with\nhaving a v2 format the way this series is adding it. I.e. saying \"here\nit is, it's never written by default, we'll figure out these compat\nissues later\".\n\nMy only objection/nit on that point would be that the current\ndocs/commit messages should make some mention of the really bad\ninteractions between v1 and v2 on different git versions.\n\nHowever, having written this again I really don't understand why we need\na v2 of this format at all.\n\nThe current format is:\n\n \n \n \n \n \n [chunk offsets for our $num_chunks]\n [arbitrary chunk data for our $num_chunks]\n\nAnd you want to change it to:\n\n \n \n \n \n \n \n [chunk offsets for our $num_chunks]\n [arbitrary chunk data for our $num_chunks]\n\nWhere \"chunks\" in the v1 format has always been a non-exhaustive list of\nthings *where we ignore anything we don't know about*.\n\nSo given our really bad compatibility issues with any non-v1 format I\nsuggested \"let's use a different filename\". But on closer look I retract\nthat.\n\nHow about we instead just don't change the header? I.e.:\n\n * Let's just live with \"1\" as the marker for SHA-1.\n\n Yeah it would be cute to use 0x73686131 instead like \"struct\n git_hash_algo\", but we can live with a 1=0x73686131 (\"sha1\"),\n 2=0x73323536 (\"s256\") mapping somewhere. It's not like we're going to\n be running into the 255 limit of hash algorithms Git will support any\n time soon.\n\n * Don't add the reachability index version *to the header* or change\n the reserved byte to be an error (see [1] again).\n\nInstead we just add these things to new \"chunks\" as appropriate. As this\npatch of mine shows we can easily do that, and it doesn't error out on\nany existing version of git:\nhttps://github.com/avar/git/commit/3fca63e12a9d38867d4bc0a8a25d419c00a09d95\n\nI now can't imagine a situation where we'd ever need to change the\nformat. We have 32 bits of chunk ids to play with, and can have 255 of\nthese chunks at a time, and unknown chunks are ignored by existing\nversions and future version.\n\nWe can even have more than 255 if it comes to that by having a special\n\"extension\" chunk, or even use the existing reserved byte for that and\npull the nasty trick of putting another set after the existing file\nchecksum, but I digress.\n\nIf we ever find that we e.g. don't want to write SHA-1 data anymore but\njust want SHA-256 we just write a tiny amount of dummy data. Older git\nversions will shrug at what looks like a really incomplete commit graph\ndata, but newer versions will know the real data is in some other chunk\nthey know about.\n\nDitto this \"gen numbers or adjusted timestamps?\" plan in\nhttps://public-inbox.org/git/6367e30a-1b3a-4fe9-611b-d931f51effef@gmail.com/\nWe can have a chunk of adjusted timestamps into the generation number\nchunk, and even start adding chunks of other side-data, e.g. the path\nbloom filters...\n\nThis E-Mail needs to stop at some point, but as a brief aside I don't\nsee how this die(\"commit-graph hash algorithm does not match current\nalgorithm\") plan makes sense either.\n\nThe hash-function-transition.txt plan describes how we'll have an index\nof SHA-1<->SHA-256 object names. Why would it be an error to read a\nSHA-1 commit-graph under SHA-256? Won't we just say \"oh this graph lists\nthe SHA-1s\" and then use that lookup table to resolve them as SHA-256s?\n\nAnd as discussed once we go for extending things with chunks we can do a\nlot better. We'd just keep the SHA-1 data segment, and perhaps (for\n\"just commits\" cache locality) have another chunk of SHA-256 mappings to\nthose SHA-1s in the commit-graph file.\n\n1. See feedback on the v2 patch in\n https://public-inbox.org/git/87muk6q98k.fsf@evledraar.gmail.com/\n"},{"id":"374816","messageId":"bb0c22f9-9d0b-0fa6-e826-8e2ac146c6f9@gmail.com","threadId":"50302","inReplyTo":"87lfzprkfc.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 0/6] Create commit-graph file format v2","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-02T13:26:04Z","receivedAt":"2019-05-02T13:26:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/1/2019 4:25 PM, Ævar Arnfjörð Bjarmason wrote:\n> I won't repeat my outstanding v2 feedback about v1 & v2\n> incompatibilities, except to say that I'd in principle be fine with\n> having a v2 format the way this series is adding it. I.e. saying \"here\n> it is, it's never written by default, we'll figure out these compat\n> issues later\".\n> \n> My only objection/nit on that point would be that the current\n> docs/commit messages should make some mention of the really bad\n> interactions between v1 and v2 on different git versions.\n\nGood idea to add some warnings in the docs to say something like\n\"version 2 is not supported by Git 2.2x and earlier\".\n\n> However, having written this again I really don't understand why we need\n> a v2 of this format at all.\n\n[snip]\n\n> How about we instead just don't change the header? I.e.:\n> \n> * Let's just live with \"1\" as the marker for SHA-1.\n> \n> Yeah it would be cute to use 0x73686131 instead like \"struct\n> git_hash_algo\", but we can live with a 1=0x73686131 (\"sha1\"),\n> 2=0x73323536 (\"s256\") mapping somewhere. It's not like we're going to\n> be running into the 255 limit of hash algorithms Git will support any\n> time soon.\n\nThis was the intended direction of using the 1-byte value before, but\nwe have a preferred plan to use the 4-byte value in all future file formats.\n\n> * Don't add the reachability index version *to the header* or change\n> the reserved byte to be an error (see [1] again).\n\nSince we can make the \"corrected commit date\" offset for a commit be\nstrictly larger than the offset of a parent, we can make it so an old client\nwill not give incorrect values when we use the new values. The only downside\nwould be that we would fail on 'git commit-graph verify' since the offsets\nare not actually generation numbers in all cases.\n\n> Instead we just add these things to new \"chunks\" as appropriate. As this\n> patch of mine shows we can easily do that, and it doesn't error out on\n> any existing version of git:\n> https://github.com/avar/git/commit/3fca63e12a9d38867d4bc0a8a25d419c00a09d95\n\nI like the idea of a \"metadata\" chunk. This can be useful for a lot of things.\nIf we start the chunk with a \"number of items\" and only append items to the\nlist, we can dynamically grow the chunk as we add values.\n\n> I now can't imagine a situation where we'd ever need to change the\n> format. We have 32 bits of chunk ids to play with, and can have 255 of\n> these chunks at a time, and unknown chunks are ignored by existing\n> versions and future version.\n\nThe solutions you have discussed work for 2 of the 3 problems at hand.\nThe incremental file format is a high-value feature, but _would_ break\nexisting clients if they don't understand the extra data. Unless I am\nmissing something for how to succeed here.\n\n> 1. See feedback on the v2 patch in\n> https://public-inbox.org/git/87muk6q98k.fsf@evledraar.gmail.com/\n\nMy response [2] to that message includes the discussion of the\nincremental file format.\n\n[2] https://public-inbox.org/git/87muk6q98k.fsf@evledraar.gmail.com/\n\n\n"},{"id":"374840","messageId":"87h8acivkh.fsf@evledraar.gmail.com","threadId":"50302","inReplyTo":"bb0c22f9-9d0b-0fa6-e826-8e2ac146c6f9@gmail.com","subject":"Re: [PATCH v3 0/6] Create commit-graph file format v2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-05-02T18:02:22Z","receivedAt":"2019-05-02T18:02:29Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, May 02 2019, Derrick Stolee wrote:\n\n> On 5/1/2019 4:25 PM, Ævar Arnfjörð Bjarmason wrote:\n>> I won't repeat my outstanding v2 feedback about v1 & v2\n>> incompatibilities, except to say that I'd in principle be fine with\n>> having a v2 format the way this series is adding it. I.e. saying \"here\n>> it is, it's never written by default, we'll figure out these compat\n>> issues later\".\n>>\n>> My only objection/nit on that point would be that the current\n>> docs/commit messages should make some mention of the really bad\n>> interactions between v1 and v2 on different git versions.\n>\n> Good idea to add some warnings in the docs to say something like\n> \"version 2 is not supported by Git 2.2x and earlier\".\n>\n>> However, having written this again I really don't understand why we need\n>> a v2 of this format at all.\n>\n> [snip]\n>\n>> How about we instead just don't change the header? I.e.:\n>>\n>> * Let's just live with \"1\" as the marker for SHA-1.\n>>\n>> Yeah it would be cute to use 0x73686131 instead like \"struct\n>> git_hash_algo\", but we can live with a 1=0x73686131 (\"sha1\"),\n>> 2=0x73323536 (\"s256\") mapping somewhere. It's not like we're going to\n>> be running into the 255 limit of hash algorithms Git will support any\n>> time soon.\n>\n> This was the intended direction of using the 1-byte value before, but\n> we have a preferred plan to use the 4-byte value in all future file formats.\n\nRight, and I wouldn't argue about such a pointless thing for a future\nfile format.\n\nBut since the v1->v2 migration story is so unfriendly already for\nreasons that can't be helped at this point (existing released versions)\nI think we need to weigh the trade-offs of changing the header v.s. just\ndoing the conceptually less clean thing that allows existing clients a\npainless transition.\n\n>> * Don't add the reachability index version *to the header* or change\n>> the reserved byte to be an error (see [1] again).\n>\n> Since we can make the \"corrected commit date\" offset for a commit be\n> strictly larger than the offset of a parent, we can make it so an old client\n> will not give incorrect values when we use the new values. The only downside\n> would be that we would fail on 'git commit-graph verify' since the offsets\n> are not actually generation numbers in all cases.\n\nAren't you talking about how the *content* (presumably in the chunk part\nof the graph) is going to look like? I just mean these couple of bytes\nin the header, again as a proxy discussion for \"do we *really* need to\nchange this?\".\n\n>> Instead we just add these things to new \"chunks\" as appropriate. As this\n>> patch of mine shows we can easily do that, and it doesn't error out on\n>> any existing version of git:\n>> https://github.com/avar/git/commit/3fca63e12a9d38867d4bc0a8a25d419c00a09d95\n>\n> I like the idea of a \"metadata\" chunk. This can be useful for a lot of things.\n> If we start the chunk with a \"number of items\" and only append items to the\n> list, we can dynamically grow the chunk as we add values.\n\nRight. I like it too :) But right now I'm just using it as a demo for\nhow new arbitrary chunk data can be added to the v1 format in backwards\ncompatible ways.\n\nMy inclination for an actual version of that patch would be to make it\neasier to read/extend (even just dump JSON there, or a series of\nkey/values) over micro-optimizing the storage size. Such metadata will\nalways be tiny v.s. the rest, but that's for later bikeshedding...\n\n>> I now can't imagine a situation where we'd ever need to change the\n>> format. We have 32 bits of chunk ids to play with, and can have 255 of\n>> these chunks at a time, and unknown chunks are ignored by existing\n>> versions and future version.\n>\n> The solutions you have discussed work for 2 of the 3 problems at hand.\n> The incremental file format is a high-value feature, but _would_ break\n> existing clients if they don't understand the extra data. Unless I am\n> missing something for how to succeed here.\n\nWe would write out a file like this:\n\n \n \n \n \n \n\nWhere one of the new chunks could be INCC (\"incremental count\") or\nwhatever, serving the same purpose as the v2 modification of the header\nto use the padding byte for the count. Then we'd have more chunks with\nthe incremental data (or pack it into one \"magic\" chunk with its own\nformat...).\n\nExisting clients deal with the graph being incomplete, so the writer\ncould just not bother to update that part of the data and a newer\nclients would know to find the rest in a series of incremental updates.\n\nIOW an \"empty\" commit-graph now is 1100 bytes. Worst case we'd be\nwriting at least that number of bytes that would be mostly or entirely\nuseless to older clients, with the rest being new stuff newer clients\nunderstand.\n\nOn the incremental format: I don't like:\n\n 1) The idea that an incremental format would involve in-place\n modification of an existing file (or would we write a completely new\n one and move it in-place?).\n\n If it's in-place modification we get away a lot of things/avoid\n complexity with \"we might delete, but we never modify existing\" on\n existing *.{idx,bitmap} formats. E.g. mmap() is a royal PITA\n (more-so than now) once you need to deal with modifications.\n\n Also, if it's in-place we'd need to fully recompute the checksum\n SHA-1 as we modify the file.\n\n 2) The assumption that we'd just have 255 of these, wouldn't it be\n reasonable to have a MIDX-like for it & write it along with packs as\n they come in? I.e. eventually have PACK.{pack,idx,bitmap,graph}.\n\n We support more than 255 packs, and it seems likely that we'd\n eventually want to generate/coalesce packs/idx and any other\n side-indexes on the same \"gc\" schedule.\n\nBut those are separate from any back-compat concerns, which is what I\nthink makes sense to focus on now.\n\n>> 1. See feedback on the v2 patch in\n>> https://public-inbox.org/git/87muk6q98k.fsf@evledraar.gmail.com/\n>\n> My response [2] to that message includes the discussion of the\n> incremental file format.\n>\n> [2] https://public-inbox.org/git/87muk6q98k.fsf@evledraar.gmail.com/\n"},{"id":"374874","messageId":"04ac97c5-ab2b-2b23-4c84-8ed8aab0e2f5@gmail.com","threadId":"50302","inReplyTo":"87h8acivkh.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 0/6] Create commit-graph file format v2","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-03T12:47:25Z","receivedAt":"2019-05-03T12:47:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/2/2019 2:02 PM, Ævar Arnfjörð Bjarmason wrote:\n>\n> But those are separate from any back-compat concerns, which is what I\n> think makes sense to focus on now.\n\nThinking more on this topic, I think I have a way to satisfy _all_ of\nyour concerns by simplifying the plan for incremental commit-graph files.\n\nMy initial plan was to have the \"commit-graph\" file always be the \"tip\"\nfile, and it would point to some number of \"graph-{hash}.graph\" files.\nThen, we would have some set of strategies to decide when we create a new\n.graph file or when we compact the files down into the \"commit-graph\"\nfile.\n\nThis has several issues regarding race conditions that I had not yet\nresolved (and maybe would always have problems).\n\nIt would be much simpler to restrict the model. Your idea of changing\nthe file name is the inspiration here.\n\n* The \"commit-graph\" file is the base commit graph. It is always\n closed under reachability (if a commit exists in this file, then\n its parents are also in this file). We will also consider this to\n be \"commit-graph-0\".\n\n* A commit-graph- exists, then we check for the existence of\n commit-graph-. This file can contain commits whose parents\n are in any smaller file.\n\nI think this resolves the issue of back-compat without updating\nthe file format:\n\n1. Old clients will never look at commit-graph-N, so they will\n never complain about an \"incomplete\" file.\n\n2. If we always open a read handle as we move up the list, then\n a \"merge and collapse\" write to commit-graph-N will not\n interrupt an existing process reading that file.\n\nI'll start hacking on this model.\n\nThanks,\n-Stolee\n"},{"id":"374875","messageId":"87bm0jirjj.fsf@evledraar.gmail.com","threadId":"50302","inReplyTo":"04ac97c5-ab2b-2b23-4c84-8ed8aab0e2f5@gmail.com","subject":"Re: [PATCH v3 0/6] Create commit-graph file format v2","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-05-03T13:41:36Z","receivedAt":"2019-05-03T13:41:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, May 03 2019, Derrick Stolee wrote:\n\n> On 5/2/2019 2:02 PM, Ævar Arnfjörð Bjarmason wrote:\n>>\n>> But those are separate from any back-compat concerns, which is what I\n>> think makes sense to focus on now.\n>\n> Thinking more on this topic, I think I have a way to satisfy _all_ of\n> your concerns by simplifying the plan for incremental commit-graph files.\n>\n> My initial plan was to have the \"commit-graph\" file always be the \"tip\"\n> file, and it would point to some number of \"graph-{hash}.graph\" files.\n> Then, we would have some set of strategies to decide when we create a new\n> .graph file or when we compact the files down into the \"commit-graph\"\n> file.\n>\n> This has several issues regarding race conditions that I had not yet\n> resolved (and maybe would always have problems).\n>\n> It would be much simpler to restrict the model. Your idea of changing\n> the file name is the inspiration here.\n\nI still have some questions about SHA-256, how we'll eventually change\nthe format of these new files etc, but those can wait...\n\n> * The \"commit-graph\" file is the base commit graph. It is always\n> closed under reachability (if a commit exists in this file, then\n> its parents are also in this file). We will also consider this to\n> be \"commit-graph-0\".\n>\n> * A commit-graph- exists, then we check for the existence of\n> commit-graph-. This file can contain commits whose parents\n> are in any smaller file.\n>\n> I think this resolves the issue of back-compat without updating\n> the file format:\n>\n> 1. Old clients will never look at commit-graph-N, so they will\n> never complain about an \"incomplete\" file.\n>\n> 2. If we always open a read handle as we move up the list, then\n> a \"merge and collapse\" write to commit-graph-N will not\n> interrupt an existing process reading that file.\n>\n> I'll start hacking on this model.\n\nJust on this, consider storing them in\n.git/objects/info/commit-graphs/commit-graph-,\nbecause:\n\n1) We can stat() the \"commit-graphs\" directory to see if there's any\n new/deleted ones (dir mtime changed), similar to what we do for the\n untracked cache, and can (but I don't think we do...) do for packs/*\n and objects/??/.\n\n As opposed to \".git/objects/info\" itself which e.g. has the \"packs\",\n \"alternates\" etc. files (can still do it, but more false positives)\n\n2) If these things are \"chained\" supersets of one another anyway we have\n a big caveat that we don't have with repacking the *.pack\n files.\n\n Those you can do in any order, as long as you write a new one it\n doesn't matter if the data is already elsewhere stored. Just repack\n however many N at a time, then unlink() all the old ones later.\n\n If you have commit-graph-, commit-graph- you can only munge\n them in that sequence, and you get to do a much more careful fsync()\n dance around with both the directory entry and the file itself, which\n has a much higher chance of breaking things.\n\n I.e. the client might see a change to \"N\" before \"N+1\", and now we're\n screwed. Dealing with this is a giant but avoidable PITA. See\n https://public-inbox.org/git/20180117184828.31816-1-hch@lst.de/ and\n http://blog.httrack.com/blog/2013/11/15/everything-you-always-wanted-to-know-about-fsync/\n\n If instead you name the files commit-graph-, and add a\n new (v1 backwards-compatible) chunk for \"and the next file's SHA-1\n is...\" we can more aggressively repack these.\n\n Worst case the directory entries will sync in the wrong order and the\n chain will be broken, but as long as we gracefully handle the chain\n abruptly ending that's OK.\n\n You can also do things like rewrite the middle \"[]\" of a \"N, [N+1,\n N+2], N+3\" sequence to have a single file for what was previously\n within those brackets (again, like \"repack\"). You'd just need to drop\n in a new \"N\" that points to the replacement for \"[N+1, N+2]\", but\n *don't* need to touch \"N+3\".\n\n Whereas if they're numbered you need to also move \"N+3\", and you're\n back to juggling both fsync() on files and directory entries.\n"},{"id":"374878","messageId":"20190503141613.GG14763@szeder.dev","threadId":"50302","inReplyTo":"04ac97c5-ab2b-2b23-4c84-8ed8aab0e2f5@gmail.com","subject":"Re: [PATCH v3 0/6] Create commit-graph file format v2","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-05-03T14:16:14Z","receivedAt":"2019-05-03T14:16:21Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, May 03, 2019 at 08:47:25AM -0400, Derrick Stolee wrote:\n> It would be much simpler to restrict the model. Your idea of changing\n> the file name is the inspiration here.\n> \n> * The \"commit-graph\" file is the base commit graph. It is always\n> closed under reachability (if a commit exists in this file, then\n> its parents are also in this file). We will also consider this to\n> be \"commit-graph-0\".\n> \n> * A commit-graph- exists, then we check for the existence of\n> commit-graph-. This file can contain commits whose parents\n> are in any smaller file.\n> \n> I think this resolves the issue of back-compat without updating\n> the file format:\n> \n> 1. Old clients will never look at commit-graph-N, so they will\n> never complain about an \"incomplete\" file.\n> \n> 2. If we always open a read handle as we move up the list, then\n> a \"merge and collapse\" write to commit-graph-N will not\n> interrupt an existing process reading that file.\n\nWhat if a process reading the commit-graph files runs short on file\ndescriptors and has to close some of them, while a second process is\nmerging commit-graph files?\n\n\n> I'll start hacking on this model.\n\nHave fun! :)\n\n\nSemi-related, but I'm curious: what are your plans for 'struct\ncommit's 'graph_pos' field, and how will it work with multiple\ncommit-graph files?\n\nIn particular: currently we use this 'graph_pos' field as an index\ninto the Commit Data chunk to find the metadata associated with a\ngiven commit object. But we could add any commit-specific metadata in\na new chunk, being an array ordered by commit OID, and then use\n'graph_pos' as an index into this chunk as well. I find this quite\nconvenient. However, with mulitple commit-graph files there will be\nmultiple arrays...\n\n"},{"id":"374882","messageId":"284b3398-0b7f-c51b-3417-af7ab521cf9d@gmail.com","threadId":"50302","inReplyTo":"20190503141613.GG14763@szeder.dev","subject":"Re: [PATCH v3 0/6] Create commit-graph file format v2","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-03T15:11:36Z","receivedAt":"2019-05-03T15:12:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/3/2019 10:16 AM, SZEDER Gábor wrote:\n> On Fri, May 03, 2019 at 08:47:25AM -0400, Derrick Stolee wrote:\n>> It would be much simpler to restrict the model. Your idea of changing\n>> the file name is the inspiration here.\n>>\n>> * The \"commit-graph\" file is the base commit graph. It is always\n>> closed under reachability (if a commit exists in this file, then\n>> its parents are also in this file). We will also consider this to\n>> be \"commit-graph-0\".\n>>\n>> * A commit-graph- exists, then we check for the existence of\n>> commit-graph-. This file can contain commits whose parents\n>> are in any smaller file.\n>>\n>> I think this resolves the issue of back-compat without updating\n>> the file format:\n>>\n>> 1. Old clients will never look at commit-graph-N, so they will\n>> never complain about an \"incomplete\" file.\n>>\n>> 2. If we always open a read handle as we move up the list, then\n>> a \"merge and collapse\" write to commit-graph-N will not\n>> interrupt an existing process reading that file.\n> \n> What if a process reading the commit-graph files runs short on file\n> descriptors and has to close some of them, while a second process is\n> merging commit-graph files?\n\nWe will want to keep the number small so we never recycle the file\nhandles. Instead, we will keep them open for the entire process.\n\nThe strategies for creating these graphs should include a \"merge\"\nstrategy that keeps the number of commit-graph files very small\n(fewer than 5 should be sufficient).\n \n>> I'll start hacking on this model.\n> \n> Have fun! :)\n> \n> \n> Semi-related, but I'm curious: what are your plans for 'struct\n> commit's 'graph_pos' field, and how will it work with multiple\n> commit-graph files?\n\nSince we have a predefined \"sequence\" of graphs, the graph_pos\nwill be the position in the \"meta-order\" given by concatenating\nthe commit lists from each commit-graph. We then navigate to a commit\nin O(num graphs) instead of O(1).\n\nIn the commit-graph format, we will use this \"meta-order\" number\nto refer to parent positions.\n\n> In particular: currently we use this 'graph_pos' field as an index\n> into the Commit Data chunk to find the metadata associated with a\n> given commit object. But we could add any commit-specific metadata in\n> a new chunk, being an array ordered by commit OID, and then use\n> 'graph_pos' as an index into this chunk as well. I find this quite\n> convenient. However, with mulitple commit-graph files there will be\n> multiple arrays...\n\nYes, this will continue to be useful*. To find the position inside\na specific commit-graph-N file, take the graph_pos and subtract the\nnumber of commits in the \"lower\" commit-graph files.\n\n* For example, this meta-data information is necessary for the Bloom\nfilter data [1].\n\nThanks,\n-Stolee\n\n[1] https://public-inbox.org/git/61559c5b-546e-d61b-d2e1-68de692f5972@gmail.com/\n"},{"id":"374939","messageId":"CAP8UFD2tnkNfgjm=H__=YPG-+BuOHwHs9=OoXnL9wnj_eOtbQg@mail.gmail.com","threadId":"50302","inReplyTo":"87bm0jirjj.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 0/6] Create commit-graph file format v2","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2019-05-06T08:27:38Z","receivedAt":"2019-05-06T08:27:53Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Fri, May 3, 2019 at 3:44 PM Ævar Arnfjörð Bjarmason wrote:\n>\n>\n> On Fri, May 03 2019, Derrick Stolee wrote:\n>\n> > On 5/2/2019 2:02 PM, Ævar Arnfjörð Bjarmason wrote:\n> >>\n> >> But those are separate from any back-compat concerns, which is what I\n> >> think makes sense to focus on now.\n> >\n> > Thinking more on this topic, I think I have a way to satisfy _all_ of\n> > your concerns by simplifying the plan for incremental commit-graph files.\n\n[...]\n\n> Just on this, consider storing them in\n> .git/objects/info/commit-graphs/commit-graph-,\n> because:\n>\n> 1) We can stat() the \"commit-graphs\" directory to see if there's any\n> new/deleted ones (dir mtime changed), similar to what we do for the\n> untracked cache, and can (but I don't think we do...) do for packs/*\n> and objects/??/.\n>\n> As opposed to \".git/objects/info\" itself which e.g. has the \"packs\",\n> \"alternates\" etc. files (can still do it, but more false positives)\n\nAbout incremental commit-graph files and alternates, I wonder if they\ncould work well together. The main use case would be for servers that\nuse a common repo for all the forks.\n"},{"id":"374944","messageId":"c827ccb9-4278-0630-ac38-6cf597437678@gmail.com","threadId":"50302","inReplyTo":"CAP8UFD2tnkNfgjm=H__=YPG-+BuOHwHs9=OoXnL9wnj_eOtbQg@mail.gmail.com","subject":"Re: [PATCH v3 0/6] Create commit-graph file format v2","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-06T13:47:55Z","receivedAt":"2019-05-06T13:47:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/6/2019 4:27 AM, Christian Couder wrote:\n> On Fri, May 3, 2019 at 3:44 PM Ævar Arnfjörð Bjarmason wrote:\n>> 1) We can stat() the \"commit-graphs\" directory to see if there's any\n>> new/deleted ones (dir mtime changed), similar to what we do for the\n>> untracked cache, and can (but I don't think we do...) do for packs/*\n>> and objects/??/.\n>>\n>> As opposed to \".git/objects/info\" itself which e.g. has the \"packs\",\n>> \"alternates\" etc. files (can still do it, but more false positives)\n> \n> About incremental commit-graph files and alternates, I wonder if they\n> could work well together. The main use case would be for servers that\n> use a common repo for all the forks.\n\nWe use a \"shared object cache\" in VFS for Git, implemented as an alternate,\nso all enlistments share prefetch packs, multi-pack-indexes, and commit-graph\nfiles. This is something we are very focused on.\n\nThanks,\n-Stolee\n\n"},{"id":"375228","messageId":"pull.112.v4.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v4 00/11] Commit-graph write refactor (was: Create commit-graph file format v2)","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-09T14:22:30Z","receivedAt":"2019-05-09T14:22:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This series replaces ds/commit-graph-file-v2, and I'm using the same\ngitgitgadget PR to continue the version numbers and hopefully make that\nclear. This is a slight modification on patches 1-11 from the incremental\nfile format RFC [0].\n\nThe commit-graph feature is growing, thanks to all of the contributions by\nseveral community members. This also means that the write_commit_graph()\nmethod is a bit unwieldy now. This series refactors that method to use a\nwrite_commit_graph_context struct that is passed between several smaller\nmethods. The final result should be a write_commit_graph() method that has a\nclear set of steps. Future changes should then be easier to understand.\n\n * Patches 1-4: these are small changes which either fix issues or just\n provide clean-up. These are mostly borrowed from\n ds/commit-graph-format-v2. \n \n \n * Patches 5-11: these provide a non-functional refactor of\n write_commit_graph() into several methods using a \"struct\n write_commit_graph_context\" to share across the methods.\n \n \n\nUpdates to commits previously in this thread:\n\n * \"commit-graph: remove Future Work section\" no longer says that 'verify'\n takes as long as 'write'. [1]\n \n \n * \"commit-graph: return with errors during write\" now has a test to check\n we don't die(). [2]\n \n \n\nÆvar: Looking at the old thread, I only saw two comments that still apply to\nthis series [1] [2]. Please point me to any comments I have missed.\n\nThanks, -Stolee\n\n[0] https://public-inbox.org/git/pull.184.git.gitgitgadget@gmail.com/\n\n[1] https://public-inbox.org/git/87o94mql0a.fsf@evledraar.gmail.com/\n\n[2] https://public-inbox.org/git/87pnp2qlkv.fsf@evledraar.gmail.com/\n\nDerrick Stolee (11):\n commit-graph: fix the_repository reference\n commit-graph: return with errors during write\n commit-graph: collapse parameters into flags\n commit-graph: remove Future Work section\n commit-graph: create write_commit_graph_context\n commit-graph: extract fill_oids_from_packs()\n commit-graph: extract fill_oids_from_commit_hex()\n commit-graph: extract fill_oids_from_all_packs()\n commit-graph: extract count_distinct_commits()\n commit-graph: extract copy_oids_to_commits()\n commit-graph: extract write_commit_graph_file()\n\n Documentation/technical/commit-graph.txt | 17 -\n builtin/commit-graph.c | 21 +-\n builtin/commit.c | 5 +-\n builtin/gc.c | 7 +-\n commit-graph.c | 607 +++++++++++++----------\n commit-graph.h | 14 +-\n commit.c | 2 +-\n t/t5318-commit-graph.sh | 8 +\n 8 files changed, 371 insertions(+), 310 deletions(-)\n\n\nbase-commit: 93b4405ffe4ad9308740e7c1c71383bfc369baaa\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-112%2Fderrickstolee%2Fgraph%2Fv2-head-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-112/derrickstolee/graph/v2-head-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/112\n\nRange-diff vs v3:\n\n -: ---------- > 1: 0be7713a25 commit-graph: fix the_repository reference\n 1: 91f300ec0a ! 2: a4082b827e commit-graph: return with errors during write\n @@ -253,3 +253,22 @@\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \n +\n + diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n + --- a/t/t5318-commit-graph.sh\n + +++ b/t/t5318-commit-graph.sh\n +@@\n + \ttest_path_is_file info/commit-graph\n + '\n + \n ++test_expect_success 'close with correct error on bad input' '\n ++\tcd \"$TRASH_DIRECTORY/full\" &&\n ++\techo doesnotexist >in &&\n ++\t{ git commit-graph write --stdin-packs stderr; ret=$?; } &&\n ++\ttest \"$ret\" = 1 &&\n ++\ttest_i18ngrep \"error adding pack\" stderr\n ++'\n ++\n + test_expect_success 'create commits and repack' '\n + \tcd \"$TRASH_DIRECTORY/full\" &&\n + \tfor i in $(test_seq 3)\n 2: 924b22f990 = 3: 469d0c9a32 commit-graph: collapse parameters into flags\n 3: 8446011a43 < -: ---------- commit-graph: create new version parameter\n 4: 6a0e99f9f9 < -: ---------- commit-graph: add --version= option\n 5: cca8267dfe < -: ---------- commit-graph: implement file format version 2\n 6: e72bca6c78 ! 4: 130007d0e1 commit-graph: remove Future Work section\n @@ -12,9 +12,8 @@\n \n It is unlikely that we will ever send a commit-graph file\n as part of the protocol, since we would need to verify the\n - data, and that is as expensive as writing a commit-graph from\n - scratch. If we want to start trusting remote content, then\n - that item can be investigated again.\n + data, and that is expensive. If we want to start trusting\n + remote content, then that item can be investigated again.\n \n While there is more work to be done on the feature, having\n a section of the docs devoted to a TODO list is wasteful and\n -: ---------- > 5: 0ca4e18e98 commit-graph: create write_commit_graph_context\n -: ---------- > 6: 30c1b618b1 commit-graph: extract fill_oids_from_packs()\n -: ---------- > 7: 8cb2613dfa commit-graph: extract fill_oids_from_commit_hex()\n -: ---------- > 8: 8f7129672a commit-graph: extract fill_oids_from_all_packs()\n -: ---------- > 9: a37548745b commit-graph: extract count_distinct_commits()\n -: ---------- > 10: 57366ffdaa commit-graph: extract copy_oids_to_commits()\n -: ---------- > 11: fc81c8946d commit-graph: extract write_commit_graph_file()\n\n-- \ngitgitgadget\n"},{"id":"375229","messageId":"0be7713a255d6f52c51ccb19299d0c9797e9a03f.1557411749.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 01/11] commit-graph: fix the_repository reference","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-09T14:22:31Z","receivedAt":"2019-05-09T14:22:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe parse_commit_buffer() method takes a repository pointer, so it\nshould not refer to the_repository anymore.\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/commit.c b/commit.c\nindex a5333c7ac6..e4d1233226 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -443,7 +443,7 @@ int parse_commit_buffer(struct repository *r, struct commit *item, const void *b\n \titem->date = parse_commit_date(bufptr, tail);\n \n \tif (check_graph)\n-\t\tload_commit_graph_info(the_repository, item);\n+\t\tload_commit_graph_info(r, item);\n \n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"375230","messageId":"a4082b827eb1999f6e46780b4147962d6dd87975.1557411749.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 02/11] commit-graph: return with errors during write","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-09T14:22:32Z","receivedAt":"2019-05-09T14:22:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method uses die() to report failure and\nexit when confronted with an unexpected condition. This use of\ndie() in a library function is incorrect and is now replaced by\nerror() statements and an int return type.\n\nNow that we use 'goto cleanup' to jump to the terminal condition\non an error, we have new paths that could lead to uninitialized\nvalues. New initializers are added to correct for this.\n\nThe builtins 'commit-graph', 'gc', and 'commit' call these methods,\nso update them to check the return value.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 19 +++++++------\n builtin/commit.c | 5 ++--\n builtin/gc.c | 7 ++---\n commit-graph.c | 60 ++++++++++++++++++++++++++++-------------\n commit-graph.h | 10 +++----\n t/t5318-commit-graph.sh | 8 ++++++\n 6 files changed, 70 insertions(+), 39 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 537fdfd0f0..2e86251f02 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -141,6 +141,7 @@ static int graph_write(int argc, const char **argv)\n \tstruct string_list *pack_indexes = NULL;\n \tstruct string_list *commit_hex = NULL;\n \tstruct string_list lines;\n+\tint result;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n@@ -168,10 +169,8 @@ static int graph_write(int argc, const char **argv)\n \n \tread_replace_refs = 0;\n \n-\tif (opts.reachable) {\n-\t\twrite_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n-\t\treturn 0;\n-\t}\n+\tif (opts.reachable)\n+\t\treturn write_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n \n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n@@ -188,14 +187,14 @@ static int graph_write(int argc, const char **argv)\n \t\tUNLEAK(buf);\n \t}\n \n-\twrite_commit_graph(opts.obj_dir,\n-\t\t\t pack_indexes,\n-\t\t\t commit_hex,\n-\t\t\t opts.append,\n-\t\t\t 1);\n+\tresult = write_commit_graph(opts.obj_dir,\n+\t\t\t\t pack_indexes,\n+\t\t\t\t commit_hex,\n+\t\t\t\t opts.append,\n+\t\t\t\t 1);\n \n \tUNLEAK(lines);\n-\treturn 0;\n+\treturn result;\n }\n \n int cmd_commit_graph(int argc, const char **argv, const char *prefix)\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 2986553d5f..b9ea7222fa 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1669,8 +1669,9 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\t \"new_index file. Check that disk is not full and quota is\\n\"\n \t\t \"not exceeded, and then \\\"git reset HEAD\\\" to recover.\"));\n \n-\tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0))\n-\t\twrite_commit_graph_reachable(get_object_directory(), 0, 0);\n+\tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n+\t write_commit_graph_reachable(get_object_directory(), 0, 0))\n+\t\treturn 1;\n \n \trepo_rerere(the_repository, 0);\n \trun_command_v_opt(argv_gc_auto, RUN_GIT_CMD);\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 020f725acc..3984addf73 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -664,9 +664,10 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tclean_pack_garbage();\n \t}\n \n-\tif (gc_write_commit_graph)\n-\t\twrite_commit_graph_reachable(get_object_directory(), 0,\n-\t\t\t\t\t !quiet && !daemonized);\n+\tif (gc_write_commit_graph &&\n+\t write_commit_graph_reachable(get_object_directory(), 0,\n+\t\t\t\t\t !quiet && !daemonized))\n+\t\treturn 1;\n \n \tif (auto_gc && too_many_loose_objects())\n \t\twarning(_(\"There are too many unreachable loose objects; \"\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 66865acbd7..ee487a364b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -851,27 +851,30 @@ static int add_ref_to_list(const char *refname,\n \treturn 0;\n }\n \n-void write_commit_graph_reachable(const char *obj_dir, int append,\n-\t\t\t\t int report_progress)\n+int write_commit_graph_reachable(const char *obj_dir, int append,\n+\t\t\t\t int report_progress)\n {\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n+\tint result;\n \n \tfor_each_ref(add_ref_to_list, &list);\n-\twrite_commit_graph(obj_dir, NULL, &list, append, report_progress);\n+\tresult = write_commit_graph(obj_dir, NULL, &list,\n+\t\t\t\t append, report_progress);\n \n \tstring_list_clear(&list, 0);\n+\treturn result;\n }\n \n-void write_commit_graph(const char *obj_dir,\n-\t\t\tstruct string_list *pack_indexes,\n-\t\t\tstruct string_list *commit_hex,\n-\t\t\tint append, int report_progress)\n+int write_commit_graph(const char *obj_dir,\n+\t\t struct string_list *pack_indexes,\n+\t\t struct string_list *commit_hex,\n+\t\t int append, int report_progress)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n \tstruct hashfile *f;\n \tuint32_t i, count_distinct = 0;\n-\tchar *graph_name;\n+\tchar *graph_name = NULL;\n \tstruct lock_file lk = LOCK_INIT;\n \tuint32_t chunk_ids[5];\n \tuint64_t chunk_offsets[5];\n@@ -883,15 +886,17 @@ void write_commit_graph(const char *obj_dir,\n \tuint64_t progress_cnt = 0;\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tunsigned long approx_nr_objects;\n+\tint res = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n-\t\treturn;\n+\t\treturn 0;\n \n \toids.nr = 0;\n \tapprox_nr_objects = approximate_object_count();\n \toids.alloc = approx_nr_objects / 32;\n \toids.progress = NULL;\n \toids.progress_done = 0;\n+\tcommits.list = NULL;\n \n \tif (append) {\n \t\tprepare_commit_graph_one(the_repository, obj_dir);\n@@ -932,10 +937,16 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tstrbuf_setlen(&packname, dirlen);\n \t\t\tstrbuf_addstr(&packname, pack_indexes->items[i].string);\n \t\t\tp = add_packed_git(packname.buf, packname.len, 1);\n-\t\t\tif (!p)\n-\t\t\t\tdie(_(\"error adding pack %s\"), packname.buf);\n-\t\t\tif (open_pack_index(p))\n-\t\t\t\tdie(_(\"error opening index for %s\"), packname.buf);\n+\t\t\tif (!p) {\n+\t\t\t\terror(_(\"error adding pack %s\"), packname.buf);\n+\t\t\t\tres = 1;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\t\t\tif (open_pack_index(p)) {\n+\t\t\t\terror(_(\"error opening index for %s\"), packname.buf);\n+\t\t\t\tres = 1;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \t\t\tfor_each_object_in_pack(p, add_packed_commits, &oids,\n \t\t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n \t\t\tclose_pack(p);\n@@ -1006,8 +1017,11 @@ void write_commit_graph(const char *obj_dir,\n \t}\n \tstop_progress(&progress);\n \n-\tif (count_distinct >= GRAPH_EDGE_LAST_MASK)\n-\t\tdie(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n+\tif (count_distinct >= GRAPH_EDGE_LAST_MASK) {\n+\t\terror(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n+\t\tres = 1;\n+\t\tgoto cleanup;\n+\t}\n \n \tcommits.nr = 0;\n \tcommits.alloc = count_distinct;\n@@ -1039,16 +1053,21 @@ void write_commit_graph(const char *obj_dir,\n \tnum_chunks = num_extra_edges ? 4 : 3;\n \tstop_progress(&progress);\n \n-\tif (commits.nr >= GRAPH_EDGE_LAST_MASK)\n-\t\tdie(_(\"too many commits to write graph\"));\n+\tif (commits.nr >= GRAPH_EDGE_LAST_MASK) {\n+\t\terror(_(\"too many commits to write graph\"));\n+\t\tres = 1;\n+\t\tgoto cleanup;\n+\t}\n \n \tcompute_generation_numbers(&commits, report_progress);\n \n \tgraph_name = get_commit_graph_filename(obj_dir);\n \tif (safe_create_leading_directories(graph_name)) {\n \t\tUNLEAK(graph_name);\n-\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n-\t\t\t graph_name);\n+\t\terror(_(\"unable to create leading directories of %s\"),\n+\t\t\tgraph_name);\n+\t\tres = errno;\n+\t\tgoto cleanup;\n \t}\n \n \thold_lock_file_for_update(&lk, graph_name, LOCK_DIE_ON_ERROR);\n@@ -1107,9 +1126,12 @@ void write_commit_graph(const char *obj_dir,\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n \tcommit_lock_file(&lk);\n \n+cleanup:\n \tfree(graph_name);\n \tfree(commits.list);\n \tfree(oids.list);\n+\n+\treturn res;\n }\n \n #define VERIFY_COMMIT_GRAPH_ERROR_HASH 2\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 7dfb8c896f..d15670bf46 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -65,12 +65,12 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n */\n int generation_numbers_enabled(struct repository *r);\n \n-void write_commit_graph_reachable(const char *obj_dir, int append,\n+int write_commit_graph_reachable(const char *obj_dir, int append,\n \t\t\t\t int report_progress);\n-void write_commit_graph(const char *obj_dir,\n-\t\t\tstruct string_list *pack_indexes,\n-\t\t\tstruct string_list *commit_hex,\n-\t\t\tint append, int report_progress);\n+int write_commit_graph(const char *obj_dir,\n+\t\t struct string_list *pack_indexes,\n+\t\t struct string_list *commit_hex,\n+\t\t int append, int report_progress);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex e80c1cac02..3b6fd0d728 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -23,6 +23,14 @@ test_expect_success 'write graph with no packs' '\n \ttest_path_is_file info/commit-graph\n '\n \n+test_expect_success 'close with correct error on bad input' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\techo doesnotexist >in &&\n+\t{ git commit-graph write --stdin-packs stderr; ret=$?; } &&\n+\ttest \"$ret\" = 1 &&\n+\ttest_i18ngrep \"error adding pack\" stderr\n+'\n+\n test_expect_success 'create commits and repack' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tfor i in $(test_seq 3)\n-- \ngitgitgadget\n\n"},{"id":"375231","messageId":"130007d0e1ba274e5260aacd510c8104073fd7d1.1557411749.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 04/11] commit-graph: remove Future Work section","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-09T14:22:34Z","receivedAt":"2019-05-09T14:22:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe commit-graph feature began with a long list of planned\nbenefits, most of which are now complete. The future work\nsection has only a few items left.\n\nAs for making more algorithms aware of generation numbers,\nsome are only waiting for generation number v2 to ensure the\nperformance matches the existing behavior using commit date.\n\nIt is unlikely that we will ever send a commit-graph file\nas part of the protocol, since we would need to verify the\ndata, and that is expensive. If we want to start trusting\nremote content, then that item can be investigated again.\n\nWhile there is more work to be done on the feature, having\na section of the docs devoted to a TODO list is wasteful and\nhard to keep up-to-date.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 17 -----------------\n 1 file changed, 17 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex 7805b0968c..fb53341d5e 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -127,23 +127,6 @@ Design Details\n helpful for these clones, anyway. The commit-graph will not be read or\n written when shallow commits are present.\n \n-Future Work\n------------\n-\n-- After computing and storing generation numbers, we must make graph\n- walks aware of generation numbers to gain the performance benefits they\n- enable. This will mostly be accomplished by swapping a commit-date-ordered\n- priority queue with one ordered by generation number. The following\n- operations are important candidates:\n-\n- - 'log --topo-order'\n- - 'tag --merged'\n-\n-- A server could provide a commit-graph file as part of the network protocol\n- to avoid extra calculations by clients. This feature is only of benefit if\n- the user is willing to trust the file, because verifying the file is correct\n- is as hard as computing it from scratch.\n-\n Related Links\n -------------\n [0] https://bugs.chromium.org/p/git/issues/detail?id=8\n-- \ngitgitgadget\n\n"},{"id":"375232","messageId":"469d0c9a32ba5a3f797b5bb50f37d6ac7746385b.1557411749.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 03/11] commit-graph: collapse parameters into flags","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-09T14:22:33Z","receivedAt":"2019-05-09T14:22:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() and write_commit_graph_reachable() methods\ncurrently take two boolean parameters: 'append' and 'report_progress'.\nWe will soon expand the possible options to send to these methods, so\ninstead of complicating the parameter list, first simplify it.\n\nCollapse these parameters into a 'flags' parameter, and adjust the\ncallers to provide flags as necessary.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 8 +++++---\n builtin/commit.c | 2 +-\n builtin/gc.c | 4 ++--\n commit-graph.c | 9 +++++----\n commit-graph.h | 8 +++++---\n 5 files changed, 18 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 2e86251f02..828b1a713f 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -142,6 +142,7 @@ static int graph_write(int argc, const char **argv)\n \tstruct string_list *commit_hex = NULL;\n \tstruct string_list lines;\n \tint result;\n+\tint flags = COMMIT_GRAPH_PROGRESS;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n@@ -166,11 +167,13 @@ static int graph_write(int argc, const char **argv)\n \t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n+\tif (opts.append)\n+\t\tflags |= COMMIT_GRAPH_APPEND;\n \n \tread_replace_refs = 0;\n \n \tif (opts.reachable)\n-\t\treturn write_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n+\t\treturn write_commit_graph_reachable(opts.obj_dir, flags);\n \n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n@@ -190,8 +193,7 @@ static int graph_write(int argc, const char **argv)\n \tresult = write_commit_graph(opts.obj_dir,\n \t\t\t\t pack_indexes,\n \t\t\t\t commit_hex,\n-\t\t\t\t opts.append,\n-\t\t\t\t 1);\n+\t\t\t\t flags);\n \n \tUNLEAK(lines);\n \treturn result;\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex b9ea7222fa..b001ef565d 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1670,7 +1670,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\t \"not exceeded, and then \\\"git reset HEAD\\\" to recover.\"));\n \n \tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n-\t write_commit_graph_reachable(get_object_directory(), 0, 0))\n+\t write_commit_graph_reachable(get_object_directory(), 0))\n \t\treturn 1;\n \n \trepo_rerere(the_repository, 0);\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 3984addf73..df2573f124 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -665,8 +665,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (gc_write_commit_graph &&\n-\t write_commit_graph_reachable(get_object_directory(), 0,\n-\t\t\t\t\t !quiet && !daemonized))\n+\t write_commit_graph_reachable(get_object_directory(),\n+\t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_PROGRESS : 0))\n \t\treturn 1;\n \n \tif (auto_gc && too_many_loose_objects())\ndiff --git a/commit-graph.c b/commit-graph.c\nindex ee487a364b..8bbd50658c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -851,15 +851,14 @@ static int add_ref_to_list(const char *refname,\n \treturn 0;\n }\n \n-int write_commit_graph_reachable(const char *obj_dir, int append,\n-\t\t\t\t int report_progress)\n+int write_commit_graph_reachable(const char *obj_dir, unsigned int flags)\n {\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n \tint result;\n \n \tfor_each_ref(add_ref_to_list, &list);\n \tresult = write_commit_graph(obj_dir, NULL, &list,\n-\t\t\t\t append, report_progress);\n+\t\t\t\t flags);\n \n \tstring_list_clear(&list, 0);\n \treturn result;\n@@ -868,7 +867,7 @@ int write_commit_graph_reachable(const char *obj_dir, int append,\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t int append, int report_progress)\n+\t\t unsigned int flags)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n@@ -887,6 +886,8 @@ int write_commit_graph(const char *obj_dir,\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tunsigned long approx_nr_objects;\n \tint res = 0;\n+\tint append = flags & COMMIT_GRAPH_APPEND;\n+\tint report_progress = flags & COMMIT_GRAPH_PROGRESS;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn 0;\ndiff --git a/commit-graph.h b/commit-graph.h\nindex d15670bf46..70f4caf0c7 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -65,12 +65,14 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n */\n int generation_numbers_enabled(struct repository *r);\n \n-int write_commit_graph_reachable(const char *obj_dir, int append,\n-\t\t\t\t int report_progress);\n+#define COMMIT_GRAPH_APPEND (1 << 0)\n+#define COMMIT_GRAPH_PROGRESS (1 << 1)\n+\n+int write_commit_graph_reachable(const char *obj_dir, unsigned int flags);\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t int append, int report_progress);\n+\t\t unsigned int flags);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \n-- \ngitgitgadget\n\n"},{"id":"375233","messageId":"8f7129672a15514138a3d677ce22522f21b43ec0.1557411749.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 08/11] commit-graph: extract fill_oids_from_all_packs()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-09T14:22:37Z","receivedAt":"2019-05-09T14:22:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too complex, so we are\nextracting methods one by one.\n\nExtract fill_oids_from_all_packs() that reads all pack-files\nfor commits and fills the oid list in the context.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 26 +++++++++++++++-----------\n 1 file changed, 15 insertions(+), 11 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex fb25280df1..730d529815 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -950,6 +950,19 @@ static void fill_oids_from_commit_hex(struct write_commit_graph_context *ctx,\n \tstrbuf_release(&progress_title);\n }\n \n+static void fill_oids_from_all_packs(struct write_commit_graph_context *ctx)\n+{\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t_(\"Finding commits for commit graph among packed objects\"),\n+\t\t\tctx->approx_nr_objects);\n+\tfor_each_packed_object(add_packed_commits, ctx,\n+\t\t\t FOR_EACH_OBJECT_PACK_ORDER);\n+\tif (ctx->progress_done < ctx->approx_nr_objects)\n+\t\tdisplay_progress(ctx->progress, ctx->approx_nr_objects);\n+\tstop_progress(&ctx->progress);\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -1006,17 +1019,8 @@ int write_commit_graph(const char *obj_dir,\n \tif (commit_hex)\n \t\tfill_oids_from_commit_hex(ctx, commit_hex);\n \n-\tif (!pack_indexes && !commit_hex) {\n-\t\tif (ctx->report_progress)\n-\t\t\tctx->progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph among packed objects\"),\n-\t\t\t\tctx->approx_nr_objects);\n-\t\tfor_each_packed_object(add_packed_commits, ctx,\n-\t\t\t\t FOR_EACH_OBJECT_PACK_ORDER);\n-\t\tif (ctx->progress_done < ctx->approx_nr_objects)\n-\t\t\tdisplay_progress(ctx->progress, ctx->approx_nr_objects);\n-\t\tstop_progress(&ctx->progress);\n-\t}\n+\tif (!pack_indexes && !commit_hex)\n+\t\tfill_oids_from_all_packs(ctx);\n \n \tclose_reachable(ctx);\n \n-- \ngitgitgadget\n\n"},{"id":"375234","messageId":"30c1b618b139e637122d9476f54d5eb8ce8e0d57.1557411749.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 06/11] commit-graph: extract fill_oids_from_packs()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-09T14:22:36Z","receivedAt":"2019-05-09T14:22:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too complex, so we are\nextracting methods one by one.\n\nThis extracts fill_oids_from_packs() that reads the given\npack-file list and fills the oid list in the context.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 83 ++++++++++++++++++++++++++++----------------------\n 1 file changed, 47 insertions(+), 36 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 58f0f0ae34..80c7069aaa 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -867,6 +867,51 @@ int write_commit_graph_reachable(const char *obj_dir, unsigned int flags)\n \treturn result;\n }\n \n+static int fill_oids_from_packs(struct write_commit_graph_context *ctx,\n+\t\t\t\tstruct string_list *pack_indexes)\n+{\n+\tuint32_t i;\n+\tstruct strbuf progress_title = STRBUF_INIT;\n+\tstruct strbuf packname = STRBUF_INIT;\n+\tint dirlen;\n+\n+\tstrbuf_addf(&packname, \"%s/pack/\", ctx->obj_dir);\n+\tdirlen = packname.len;\n+\tif (ctx->report_progress) {\n+\t\tstrbuf_addf(&progress_title,\n+\t\t\t Q_(\"Finding commits for commit graph in %d pack\",\n+\t\t\t \"Finding commits for commit graph in %d packs\",\n+\t\t\t pack_indexes->nr),\n+\t\t\t pack_indexes->nr);\n+\t\tctx->progress = start_delayed_progress(progress_title.buf, 0);\n+\t\tctx->progress_done = 0;\n+\t}\n+\tfor (i = 0; i < pack_indexes->nr; i++) {\n+\t\tstruct packed_git *p;\n+\t\tstrbuf_setlen(&packname, dirlen);\n+\t\tstrbuf_addstr(&packname, pack_indexes->items[i].string);\n+\t\tp = add_packed_git(packname.buf, packname.len, 1);\n+\t\tif (!p) {\n+\t\t\terror(_(\"error adding pack %s\"), packname.buf);\n+\t\t\treturn 1;\n+\t\t}\n+\t\tif (open_pack_index(p)) {\n+\t\t\terror(_(\"error opening index for %s\"), packname.buf);\n+\t\t\treturn 1;\n+\t\t}\n+\t\tfor_each_object_in_pack(p, add_packed_commits, ctx,\n+\t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n+\t\tclose_pack(p);\n+\t\tfree(p);\n+\t}\n+\n+\tstop_progress(&ctx->progress);\n+\tstrbuf_reset(&progress_title);\n+\tstrbuf_release(&packname);\n+\n+\treturn 0;\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -916,42 +961,8 @@ int write_commit_graph(const char *obj_dir,\n \t}\n \n \tif (pack_indexes) {\n-\t\tstruct strbuf packname = STRBUF_INIT;\n-\t\tint dirlen;\n-\t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n-\t\tdirlen = packname.len;\n-\t\tif (ctx->report_progress) {\n-\t\t\tstrbuf_addf(&progress_title,\n-\t\t\t\t Q_(\"Finding commits for commit graph in %d pack\",\n-\t\t\t\t \"Finding commits for commit graph in %d packs\",\n-\t\t\t\t pack_indexes->nr),\n-\t\t\t\t pack_indexes->nr);\n-\t\t\tctx->progress = start_delayed_progress(progress_title.buf, 0);\n-\t\t\tctx->progress_done = 0;\n-\t\t}\n-\t\tfor (i = 0; i < pack_indexes->nr; i++) {\n-\t\t\tstruct packed_git *p;\n-\t\t\tstrbuf_setlen(&packname, dirlen);\n-\t\t\tstrbuf_addstr(&packname, pack_indexes->items[i].string);\n-\t\t\tp = add_packed_git(packname.buf, packname.len, 1);\n-\t\t\tif (!p) {\n-\t\t\t\terror(_(\"error adding pack %s\"), packname.buf);\n-\t\t\t\tres = 1;\n-\t\t\t\tgoto cleanup;\n-\t\t\t}\n-\t\t\tif (open_pack_index(p)) {\n-\t\t\t\terror(_(\"error opening index for %s\"), packname.buf);\n-\t\t\t\tres = 1;\n-\t\t\t\tgoto cleanup;\n-\t\t\t}\n-\t\t\tfor_each_object_in_pack(p, add_packed_commits, ctx,\n-\t\t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n-\t\t\tclose_pack(p);\n-\t\t\tfree(p);\n-\t\t}\n-\t\tstop_progress(&ctx->progress);\n-\t\tstrbuf_reset(&progress_title);\n-\t\tstrbuf_release(&packname);\n+\t\tif ((res = fill_oids_from_packs(ctx, pack_indexes)))\n+\t\t\tgoto cleanup;\n \t}\n \n \tif (commit_hex) {\n-- \ngitgitgadget\n\n"},{"id":"375235","messageId":"0ca4e18e98dafe8d87c634a4988f8bb9474e4beb.1557411749.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 05/11] commit-graph: create write_commit_graph_context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-09T14:22:35Z","receivedAt":"2019-05-09T14:22:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too large and complex. To simplify\nit, we should extract several small methods. However, we will risk\nrepeating a lot of declarations related to progress incidators and\nobject id or commit lists.\n\nCreate a new write_commit_graph_context struct that contains the\ncore data structures used in this process. Replace the other local\nvariables with the values inside the context object. Following this\nchange, we will start to lift code segments wholesale out of the\nwrite_commit_graph() method and into their own methods.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 390 ++++++++++++++++++++++++-------------------------\n 1 file changed, 194 insertions(+), 196 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8bbd50658c..58f0f0ae34 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -518,14 +518,38 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \treturn get_commit_tree_in_graph_one(r, r->objects->commit_graph, c);\n }\n \n+struct packed_commit_list {\n+\tstruct commit **list;\n+\tint nr;\n+\tint alloc;\n+};\n+\n+struct packed_oid_list {\n+\tstruct object_id *list;\n+\tint nr;\n+\tint alloc;\n+};\n+\n+struct write_commit_graph_context {\n+\tstruct repository *r;\n+\tconst char *obj_dir;\n+\tchar *graph_name;\n+\tstruct packed_oid_list oids;\n+\tstruct packed_commit_list commits;\n+\tint num_extra_edges;\n+\tunsigned long approx_nr_objects;\n+\tstruct progress *progress;\n+\tint progress_done;\n+\tuint64_t progress_cnt;\n+\tunsigned append:1,\n+\t\t report_progress:1;\n+};\n+\n static void write_graph_chunk_fanout(struct hashfile *f,\n-\t\t\t\t struct commit **commits,\n-\t\t\t\t int nr_commits,\n-\t\t\t\t struct progress *progress,\n-\t\t\t\t uint64_t *progress_cnt)\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tint i, count = 0;\n-\tstruct commit **list = commits;\n+\tstruct commit **list = ctx->commits.list;\n \n \t/*\n \t * Write the first-level table (the list is sorted,\n@@ -533,10 +557,10 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n \t * having to do eight extra binary search iterations).\n \t */\n \tfor (i = 0; i < 256; i++) {\n-\t\twhile (count < nr_commits) {\n+\t\twhile (count < ctx->commits.nr) {\n \t\t\tif ((*list)->object.oid.hash[0] != i)\n \t\t\t\tbreak;\n-\t\t\tdisplay_progress(progress, ++*progress_cnt);\n+\t\t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n \t\t\tcount++;\n \t\t\tlist++;\n \t\t}\n@@ -546,14 +570,12 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n }\n \n static void write_graph_chunk_oids(struct hashfile *f, int hash_len,\n-\t\t\t\t struct commit **commits, int nr_commits,\n-\t\t\t\t struct progress *progress,\n-\t\t\t\t uint64_t *progress_cnt)\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n-\tstruct commit **list = commits;\n+\tstruct commit **list = ctx->commits.list;\n \tint count;\n-\tfor (count = 0; count < nr_commits; count++, list++) {\n-\t\tdisplay_progress(progress, ++*progress_cnt);\n+\tfor (count = 0; count < ctx->commits.nr; count++, list++) {\n+\t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n \t\thashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n \t}\n }\n@@ -565,19 +587,17 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n }\n \n static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n-\t\t\t\t struct commit **commits, int nr_commits,\n-\t\t\t\t struct progress *progress,\n-\t\t\t\t uint64_t *progress_cnt)\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n-\tstruct commit **list = commits;\n-\tstruct commit **last = commits + nr_commits;\n+\tstruct commit **list = ctx->commits.list;\n+\tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tuint32_t num_extra_edges = 0;\n \n \twhile (list < last) {\n \t\tstruct commit_list *parent;\n \t\tint edge_value;\n \t\tuint32_t packedDate[2];\n-\t\tdisplay_progress(progress, ++*progress_cnt);\n+\t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n \n \t\tparse_commit_no_graph(*list);\n \t\thashwrite(f, get_commit_tree_oid(*list)->hash, hash_len);\n@@ -588,8 +608,8 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\tedge_value = GRAPH_PARENT_NONE;\n \t\telse {\n \t\t\tedge_value = sha1_pos(parent->item->object.oid.hash,\n-\t\t\t\t\t commits,\n-\t\t\t\t\t nr_commits,\n+\t\t\t\t\t ctx->commits.list,\n+\t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t commit_to_sha1);\n \n \t\t\tif (edge_value < 0)\n@@ -609,8 +629,8 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\tedge_value = GRAPH_EXTRA_EDGES_NEEDED | num_extra_edges;\n \t\telse {\n \t\t\tedge_value = sha1_pos(parent->item->object.oid.hash,\n-\t\t\t\t\t commits,\n-\t\t\t\t\t nr_commits,\n+\t\t\t\t\t ctx->commits.list,\n+\t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t commit_to_sha1);\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n@@ -642,19 +662,16 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n }\n \n static void write_graph_chunk_extra_edges(struct hashfile *f,\n-\t\t\t\t\t struct commit **commits,\n-\t\t\t\t\t int nr_commits,\n-\t\t\t\t\t struct progress *progress,\n-\t\t\t\t\t uint64_t *progress_cnt)\n+\t\t\t\t\t struct write_commit_graph_context *ctx)\n {\n-\tstruct commit **list = commits;\n-\tstruct commit **last = commits + nr_commits;\n+\tstruct commit **list = ctx->commits.list;\n+\tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tstruct commit_list *parent;\n \n \twhile (list < last) {\n \t\tint num_parents = 0;\n \n-\t\tdisplay_progress(progress, ++*progress_cnt);\n+\t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n \n \t\tfor (parent = (*list)->parents; num_parents < 3 && parent;\n \t\t parent = parent->next)\n@@ -668,8 +685,8 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n \t\t/* Since num_parents > 2, this initializer is safe. */\n \t\tfor (parent = (*list)->parents->next; parent; parent = parent->next) {\n \t\t\tint edge_value = sha1_pos(parent->item->object.oid.hash,\n-\t\t\t\t\t\t commits,\n-\t\t\t\t\t\t nr_commits,\n+\t\t\t\t\t\t ctx->commits.list,\n+\t\t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t\t commit_to_sha1);\n \n \t\t\tif (edge_value < 0)\n@@ -693,125 +710,111 @@ static int commit_compare(const void *_a, const void *_b)\n \treturn oidcmp(a, b);\n }\n \n-struct packed_commit_list {\n-\tstruct commit **list;\n-\tint nr;\n-\tint alloc;\n-};\n-\n-struct packed_oid_list {\n-\tstruct object_id *list;\n-\tint nr;\n-\tint alloc;\n-\tstruct progress *progress;\n-\tint progress_done;\n-};\n-\n static int add_packed_commits(const struct object_id *oid,\n \t\t\t struct packed_git *pack,\n \t\t\t uint32_t pos,\n \t\t\t void *data)\n {\n-\tstruct packed_oid_list *list = (struct packed_oid_list*)data;\n+\tstruct write_commit_graph_context *ctx = (struct write_commit_graph_context*)data;\n \tenum object_type type;\n \toff_t offset = nth_packed_object_offset(pack, pos);\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \n-\tif (list->progress)\n-\t\tdisplay_progress(list->progress, ++list->progress_done);\n+\tif (ctx->progress)\n+\t\tdisplay_progress(ctx->progress, ++ctx->progress_done);\n \n \toi.typep = &type;\n-\tif (packed_object_info(the_repository, pack, offset, &oi) < 0)\n+\tif (packed_object_info(ctx->r, pack, offset, &oi) < 0)\n \t\tdie(_(\"unable to get type of object %s\"), oid_to_hex(oid));\n \n \tif (type != OBJ_COMMIT)\n \t\treturn 0;\n \n-\tALLOC_GROW(list->list, list->nr + 1, list->alloc);\n-\toidcpy(&(list->list[list->nr]), oid);\n-\tlist->nr++;\n+\tALLOC_GROW(ctx->oids.list, ctx->oids.nr + 1, ctx->oids.alloc);\n+\toidcpy(&(ctx->oids.list[ctx->oids.nr]), oid);\n+\tctx->oids.nr++;\n \n \treturn 0;\n }\n \n-static void add_missing_parents(struct packed_oid_list *oids, struct commit *commit)\n+static void add_missing_parents(struct write_commit_graph_context *ctx, struct commit *commit)\n {\n \tstruct commit_list *parent;\n \tfor (parent = commit->parents; parent; parent = parent->next) {\n \t\tif (!(parent->item->object.flags & UNINTERESTING)) {\n-\t\t\tALLOC_GROW(oids->list, oids->nr + 1, oids->alloc);\n-\t\t\toidcpy(&oids->list[oids->nr], &(parent->item->object.oid));\n-\t\t\toids->nr++;\n+\t\t\tALLOC_GROW(ctx->oids.list, ctx->oids.nr + 1, ctx->oids.alloc);\n+\t\t\toidcpy(&ctx->oids.list[ctx->oids.nr], &(parent->item->object.oid));\n+\t\t\tctx->oids.nr++;\n \t\t\tparent->item->object.flags |= UNINTERESTING;\n \t\t}\n \t}\n }\n \n-static void close_reachable(struct packed_oid_list *oids, int report_progress)\n+static void close_reachable(struct write_commit_graph_context *ctx)\n {\n \tint i;\n \tstruct commit *commit;\n-\tstruct progress *progress = NULL;\n \n-\tif (report_progress)\n-\t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Loading known commits in commit graph\"), oids->nr);\n-\tfor (i = 0; i < oids->nr; i++) {\n-\t\tdisplay_progress(progress, i + 1);\n-\t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"Loading known commits in commit graph\"),\n+\t\t\t\t\tctx->oids.nr);\n+\tfor (i = 0; i < ctx->oids.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tcommit = lookup_commit(ctx->r, &ctx->oids.list[i]);\n \t\tif (commit)\n \t\t\tcommit->object.flags |= UNINTERESTING;\n \t}\n-\tstop_progress(&progress);\n+\tstop_progress(&ctx->progress);\n \n \t/*\n-\t * As this loop runs, oids->nr may grow, but not more\n+\t * As this loop runs, ctx->oids.nr may grow, but not more\n \t * than the number of missing commits in the reachable\n \t * closure.\n \t */\n-\tif (report_progress)\n-\t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Expanding reachable commits in commit graph\"), oids->nr);\n-\tfor (i = 0; i < oids->nr; i++) {\n-\t\tdisplay_progress(progress, i + 1);\n-\t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"Expanding reachable commits in commit graph\"),\n+\t\t\t\t\tctx->oids.nr);\n+\tfor (i = 0; i < ctx->oids.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tcommit = lookup_commit(ctx->r, &ctx->oids.list[i]);\n \n \t\tif (commit && !parse_commit_no_graph(commit))\n-\t\t\tadd_missing_parents(oids, commit);\n+\t\t\tadd_missing_parents(ctx, commit);\n \t}\n-\tstop_progress(&progress);\n+\tstop_progress(&ctx->progress);\n \n-\tif (report_progress)\n-\t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Clearing commit marks in commit graph\"), oids->nr);\n-\tfor (i = 0; i < oids->nr; i++) {\n-\t\tdisplay_progress(progress, i + 1);\n-\t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"Clearing commit marks in commit graph\"),\n+\t\t\t\t\tctx->oids.nr);\n+\tfor (i = 0; i < ctx->oids.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tcommit = lookup_commit(ctx->r, &ctx->oids.list[i]);\n \n \t\tif (commit)\n \t\t\tcommit->object.flags &= ~UNINTERESTING;\n \t}\n-\tstop_progress(&progress);\n+\tstop_progress(&ctx->progress);\n }\n \n-static void compute_generation_numbers(struct packed_commit_list* commits,\n-\t\t\t\t int report_progress)\n+static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n {\n \tint i;\n \tstruct commit_list *list = NULL;\n-\tstruct progress *progress = NULL;\n \n-\tif (report_progress)\n-\t\tprogress = start_progress(\n-\t\t\t_(\"Computing commit graph generation numbers\"),\n-\t\t\tcommits->nr);\n-\tfor (i = 0; i < commits->nr; i++) {\n-\t\tdisplay_progress(progress, i + 1);\n-\t\tif (commits->list[i]->generation != GENERATION_NUMBER_INFINITY &&\n-\t\t commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_progress(\n+\t\t\t\t\t_(\"Computing commit graph generation numbers\"),\n+\t\t\t\t\tctx->commits.nr);\n+\tfor (i = 0; i < ctx->commits.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tif (ctx->commits.list[i]->generation != GENERATION_NUMBER_INFINITY &&\n+\t\t ctx->commits.list[i]->generation != GENERATION_NUMBER_ZERO)\n \t\t\tcontinue;\n \n-\t\tcommit_list_insert(commits->list[i], &list);\n+\t\tcommit_list_insert(ctx->commits.list[i], &list);\n \t\twhile (list) {\n \t\t\tstruct commit *current = list->item;\n \t\t\tstruct commit_list *parent;\n@@ -838,7 +841,7 @@ static void compute_generation_numbers(struct packed_commit_list* commits,\n \t\t\t}\n \t\t}\n \t}\n-\tstop_progress(&progress);\n+\tstop_progress(&ctx->progress);\n }\n \n static int add_ref_to_list(const char *refname,\n@@ -869,8 +872,7 @@ int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *commit_hex,\n \t\t unsigned int flags)\n {\n-\tstruct packed_oid_list oids;\n-\tstruct packed_commit_list commits;\n+\tstruct write_commit_graph_context *ctx;\n \tstruct hashfile *f;\n \tuint32_t i, count_distinct = 0;\n \tchar *graph_name = NULL;\n@@ -878,44 +880,38 @@ int write_commit_graph(const char *obj_dir,\n \tuint32_t chunk_ids[5];\n \tuint64_t chunk_offsets[5];\n \tint num_chunks;\n-\tint num_extra_edges;\n \tstruct commit_list *parent;\n-\tstruct progress *progress = NULL;\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n-\tuint64_t progress_cnt = 0;\n \tstruct strbuf progress_title = STRBUF_INIT;\n-\tunsigned long approx_nr_objects;\n \tint res = 0;\n-\tint append = flags & COMMIT_GRAPH_APPEND;\n-\tint report_progress = flags & COMMIT_GRAPH_PROGRESS;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn 0;\n \n-\toids.nr = 0;\n-\tapprox_nr_objects = approximate_object_count();\n-\toids.alloc = approx_nr_objects / 32;\n-\toids.progress = NULL;\n-\toids.progress_done = 0;\n-\tcommits.list = NULL;\n-\n-\tif (append) {\n-\t\tprepare_commit_graph_one(the_repository, obj_dir);\n-\t\tif (the_repository->objects->commit_graph)\n-\t\t\toids.alloc += the_repository->objects->commit_graph->num_commits;\n+\tctx = xcalloc(1, sizeof(struct write_commit_graph_context));\n+\tctx->r = the_repository;\n+\tctx->obj_dir = obj_dir;\n+\tctx->append = flags & COMMIT_GRAPH_APPEND ? 1 : 0;\n+\tctx->report_progress = flags & COMMIT_GRAPH_PROGRESS ? 1 : 0;\n+\n+\tctx->approx_nr_objects = approximate_object_count();\n+\tctx->oids.alloc = ctx->approx_nr_objects / 32;\n+\n+\tif (ctx->append) {\n+\t\tprepare_commit_graph_one(ctx->r, ctx->obj_dir);\n+\t\tif (ctx->r->objects->commit_graph)\n+\t\t\tctx->oids.alloc += ctx->r->objects->commit_graph->num_commits;\n \t}\n \n-\tif (oids.alloc < 1024)\n-\t\toids.alloc = 1024;\n-\tALLOC_ARRAY(oids.list, oids.alloc);\n-\n-\tif (append && the_repository->objects->commit_graph) {\n-\t\tstruct commit_graph *commit_graph =\n-\t\t\tthe_repository->objects->commit_graph;\n-\t\tfor (i = 0; i < commit_graph->num_commits; i++) {\n-\t\t\tconst unsigned char *hash = commit_graph->chunk_oid_lookup +\n-\t\t\t\tcommit_graph->hash_len * i;\n-\t\t\thashcpy(oids.list[oids.nr++].hash, hash);\n+\tif (ctx->oids.alloc < 1024)\n+\t\tctx->oids.alloc = 1024;\n+\tALLOC_ARRAY(ctx->oids.list, ctx->oids.alloc);\n+\n+\tif (ctx->append && ctx->r->objects->commit_graph) {\n+\t\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\t\tfor (i = 0; i < g->num_commits; i++) {\n+\t\t\tconst unsigned char *hash = g->chunk_oid_lookup + g->hash_len * i;\n+\t\t\thashcpy(ctx->oids.list[ctx->oids.nr++].hash, hash);\n \t\t}\n \t}\n \n@@ -924,14 +920,14 @@ int write_commit_graph(const char *obj_dir,\n \t\tint dirlen;\n \t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n \t\tdirlen = packname.len;\n-\t\tif (report_progress) {\n+\t\tif (ctx->report_progress) {\n \t\t\tstrbuf_addf(&progress_title,\n \t\t\t\t Q_(\"Finding commits for commit graph in %d pack\",\n \t\t\t\t \"Finding commits for commit graph in %d packs\",\n \t\t\t\t pack_indexes->nr),\n \t\t\t\t pack_indexes->nr);\n-\t\t\toids.progress = start_delayed_progress(progress_title.buf, 0);\n-\t\t\toids.progress_done = 0;\n+\t\t\tctx->progress = start_delayed_progress(progress_title.buf, 0);\n+\t\t\tctx->progress_done = 0;\n \t\t}\n \t\tfor (i = 0; i < pack_indexes->nr; i++) {\n \t\t\tstruct packed_git *p;\n@@ -948,75 +944,76 @@ int write_commit_graph(const char *obj_dir,\n \t\t\t\tres = 1;\n \t\t\t\tgoto cleanup;\n \t\t\t}\n-\t\t\tfor_each_object_in_pack(p, add_packed_commits, &oids,\n+\t\t\tfor_each_object_in_pack(p, add_packed_commits, ctx,\n \t\t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n \t\t\tclose_pack(p);\n \t\t\tfree(p);\n \t\t}\n-\t\tstop_progress(&oids.progress);\n+\t\tstop_progress(&ctx->progress);\n \t\tstrbuf_reset(&progress_title);\n \t\tstrbuf_release(&packname);\n \t}\n \n \tif (commit_hex) {\n-\t\tif (report_progress) {\n+\t\tif (ctx->report_progress) {\n \t\t\tstrbuf_addf(&progress_title,\n \t\t\t\t Q_(\"Finding commits for commit graph from %d ref\",\n \t\t\t\t \"Finding commits for commit graph from %d refs\",\n \t\t\t\t commit_hex->nr),\n \t\t\t\t commit_hex->nr);\n-\t\t\tprogress = start_delayed_progress(progress_title.buf,\n-\t\t\t\t\t\t\t commit_hex->nr);\n+\t\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t\tprogress_title.buf,\n+\t\t\t\t\t\tcommit_hex->nr);\n \t\t}\n \t\tfor (i = 0; i < commit_hex->nr; i++) {\n \t\t\tconst char *end;\n \t\t\tstruct object_id oid;\n \t\t\tstruct commit *result;\n \n-\t\t\tdisplay_progress(progress, i + 1);\n+\t\t\tdisplay_progress(ctx->progress, i + 1);\n \t\t\tif (commit_hex->items[i].string &&\n \t\t\t parse_oid_hex(commit_hex->items[i].string, &oid, &end))\n \t\t\t\tcontinue;\n \n-\t\t\tresult = lookup_commit_reference_gently(the_repository, &oid, 1);\n+\t\t\tresult = lookup_commit_reference_gently(ctx->r, &oid, 1);\n \n \t\t\tif (result) {\n-\t\t\t\tALLOC_GROW(oids.list, oids.nr + 1, oids.alloc);\n-\t\t\t\toidcpy(&oids.list[oids.nr], &(result->object.oid));\n-\t\t\t\toids.nr++;\n+\t\t\t\tALLOC_GROW(ctx->oids.list, ctx->oids.nr + 1, ctx->oids.alloc);\n+\t\t\t\toidcpy(&ctx->oids.list[ctx->oids.nr], &(result->object.oid));\n+\t\t\t\tctx->oids.nr++;\n \t\t\t}\n \t\t}\n-\t\tstop_progress(&progress);\n+\t\tstop_progress(&ctx->progress);\n \t\tstrbuf_reset(&progress_title);\n \t}\n \n \tif (!pack_indexes && !commit_hex) {\n-\t\tif (report_progress)\n-\t\t\toids.progress = start_delayed_progress(\n+\t\tif (ctx->report_progress)\n+\t\t\tctx->progress = start_delayed_progress(\n \t\t\t\t_(\"Finding commits for commit graph among packed objects\"),\n-\t\t\t\tapprox_nr_objects);\n-\t\tfor_each_packed_object(add_packed_commits, &oids,\n+\t\t\t\tctx->approx_nr_objects);\n+\t\tfor_each_packed_object(add_packed_commits, ctx,\n \t\t\t\t FOR_EACH_OBJECT_PACK_ORDER);\n-\t\tif (oids.progress_done < approx_nr_objects)\n-\t\t\tdisplay_progress(oids.progress, approx_nr_objects);\n-\t\tstop_progress(&oids.progress);\n+\t\tif (ctx->progress_done < ctx->approx_nr_objects)\n+\t\t\tdisplay_progress(ctx->progress, ctx->approx_nr_objects);\n+\t\tstop_progress(&ctx->progress);\n \t}\n \n-\tclose_reachable(&oids, report_progress);\n+\tclose_reachable(ctx);\n \n-\tif (report_progress)\n-\t\tprogress = start_delayed_progress(\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n \t\t\t_(\"Counting distinct commits in commit graph\"),\n-\t\t\toids.nr);\n-\tdisplay_progress(progress, 0); /* TODO: Measure QSORT() progress */\n-\tQSORT(oids.list, oids.nr, commit_compare);\n+\t\t\tctx->oids.nr);\n+\tdisplay_progress(ctx->progress, 0); /* TODO: Measure QSORT() progress */\n+\tQSORT(ctx->oids.list, ctx->oids.nr, commit_compare);\n \tcount_distinct = 1;\n-\tfor (i = 1; i < oids.nr; i++) {\n-\t\tdisplay_progress(progress, i + 1);\n-\t\tif (!oideq(&oids.list[i - 1], &oids.list[i]))\n+\tfor (i = 1; i < ctx->oids.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n \t\t\tcount_distinct++;\n \t}\n-\tstop_progress(&progress);\n+\tstop_progress(&ctx->progress);\n \n \tif (count_distinct >= GRAPH_EDGE_LAST_MASK) {\n \t\terror(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n@@ -1024,54 +1021,54 @@ int write_commit_graph(const char *obj_dir,\n \t\tgoto cleanup;\n \t}\n \n-\tcommits.nr = 0;\n-\tcommits.alloc = count_distinct;\n-\tALLOC_ARRAY(commits.list, commits.alloc);\n+\tctx->commits.alloc = count_distinct;\n+\tALLOC_ARRAY(ctx->commits.list, ctx->commits.alloc);\n \n-\tnum_extra_edges = 0;\n-\tif (report_progress)\n-\t\tprogress = start_delayed_progress(\n+\tctx->num_extra_edges = 0;\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n \t\t\t_(\"Finding extra edges in commit graph\"),\n-\t\t\toids.nr);\n-\tfor (i = 0; i < oids.nr; i++) {\n+\t\t\tctx->oids.nr);\n+\tfor (i = 0; i < ctx->oids.nr; i++) {\n \t\tint num_parents = 0;\n-\t\tdisplay_progress(progress, i + 1);\n-\t\tif (i > 0 && oideq(&oids.list[i - 1], &oids.list[i]))\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tif (i > 0 && oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n \t\t\tcontinue;\n \n-\t\tcommits.list[commits.nr] = lookup_commit(the_repository, &oids.list[i]);\n-\t\tparse_commit_no_graph(commits.list[commits.nr]);\n+\t\tctx->commits.list[ctx->commits.nr] = lookup_commit(ctx->r, &ctx->oids.list[i]);\n+\t\tparse_commit_no_graph(ctx->commits.list[ctx->commits.nr]);\n \n-\t\tfor (parent = commits.list[commits.nr]->parents;\n+\t\tfor (parent = ctx->commits.list[ctx->commits.nr]->parents;\n \t\t parent; parent = parent->next)\n \t\t\tnum_parents++;\n \n \t\tif (num_parents > 2)\n-\t\t\tnum_extra_edges += num_parents - 1;\n+\t\t\tctx->num_extra_edges += num_parents - 1;\n \n-\t\tcommits.nr++;\n+\t\tctx->commits.nr++;\n \t}\n-\tnum_chunks = num_extra_edges ? 4 : 3;\n-\tstop_progress(&progress);\n+\tstop_progress(&ctx->progress);\n \n-\tif (commits.nr >= GRAPH_EDGE_LAST_MASK) {\n+\tif (ctx->commits.nr >= GRAPH_EDGE_LAST_MASK) {\n \t\terror(_(\"too many commits to write graph\"));\n \t\tres = 1;\n \t\tgoto cleanup;\n \t}\n \n-\tcompute_generation_numbers(&commits, report_progress);\n+\tcompute_generation_numbers(ctx);\n \n-\tgraph_name = get_commit_graph_filename(obj_dir);\n-\tif (safe_create_leading_directories(graph_name)) {\n-\t\tUNLEAK(graph_name);\n+\tnum_chunks = ctx->num_extra_edges ? 4 : 3;\n+\n+\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n+\tif (safe_create_leading_directories(ctx->graph_name)) {\n+\t\tUNLEAK(ctx->graph_name);\n \t\terror(_(\"unable to create leading directories of %s\"),\n-\t\t\tgraph_name);\n+\t\t\tctx->graph_name);\n \t\tres = errno;\n \t\tgoto cleanup;\n \t}\n \n-\thold_lock_file_for_update(&lk, graph_name, LOCK_DIE_ON_ERROR);\n+\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \n \thashwrite_be32(f, GRAPH_SIGNATURE);\n@@ -1084,7 +1081,7 @@ int write_commit_graph(const char *obj_dir,\n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n \tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n-\tif (num_extra_edges)\n+\tif (ctx->num_extra_edges)\n \t\tchunk_ids[3] = GRAPH_CHUNKID_EXTRAEDGES;\n \telse\n \t\tchunk_ids[3] = 0;\n@@ -1092,9 +1089,9 @@ int write_commit_graph(const char *obj_dir,\n \n \tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n-\tchunk_offsets[2] = chunk_offsets[1] + hashsz * commits.nr;\n-\tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * commits.nr;\n-\tchunk_offsets[4] = chunk_offsets[3] + 4 * num_extra_edges;\n+\tchunk_offsets[2] = chunk_offsets[1] + hashsz * ctx->commits.nr;\n+\tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * ctx->commits.nr;\n+\tchunk_offsets[4] = chunk_offsets[3] + 4 * ctx->num_extra_edges;\n \n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tuint32_t chunk_write[3];\n@@ -1105,32 +1102,33 @@ int write_commit_graph(const char *obj_dir,\n \t\thashwrite(f, chunk_write, 12);\n \t}\n \n-\tif (report_progress) {\n+\tif (ctx->report_progress) {\n \t\tstrbuf_addf(&progress_title,\n \t\t\t Q_(\"Writing out commit graph in %d pass\",\n \t\t\t \"Writing out commit graph in %d passes\",\n \t\t\t num_chunks),\n \t\t\t num_chunks);\n-\t\tprogress = start_delayed_progress(\n+\t\tctx->progress = start_delayed_progress(\n \t\t\tprogress_title.buf,\n-\t\t\tnum_chunks * commits.nr);\n+\t\t\tnum_chunks * ctx->commits.nr);\n \t}\n-\twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress, &progress_cnt);\n-\twrite_graph_chunk_oids(f, hashsz, commits.list, commits.nr, progress, &progress_cnt);\n-\twrite_graph_chunk_data(f, hashsz, commits.list, commits.nr, progress, &progress_cnt);\n-\tif (num_extra_edges)\n-\t\twrite_graph_chunk_extra_edges(f, commits.list, commits.nr, progress, &progress_cnt);\n-\tstop_progress(&progress);\n+\twrite_graph_chunk_fanout(f, ctx);\n+\twrite_graph_chunk_oids(f, hashsz, ctx);\n+\twrite_graph_chunk_data(f, hashsz, ctx);\n+\tif (ctx->num_extra_edges)\n+\t\twrite_graph_chunk_extra_edges(f, ctx);\n+\tstop_progress(&ctx->progress);\n \tstrbuf_release(&progress_title);\n \n-\tclose_commit_graph(the_repository);\n+\tclose_commit_graph(ctx->r);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n \tcommit_lock_file(&lk);\n \n cleanup:\n \tfree(graph_name);\n-\tfree(commits.list);\n-\tfree(oids.list);\n+\tfree(ctx->commits.list);\n+\tfree(ctx->oids.list);\n+\tfree(ctx);\n \n \treturn res;\n }\n-- \ngitgitgadget\n\n"},{"id":"375236","messageId":"57366ffdaae5f646a26e52f9494a80b28761356d.1557411749.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 10/11] commit-graph: extract copy_oids_to_commits()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-09T14:22:39Z","receivedAt":"2019-05-09T14:22:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too complex, so we are\nextracting methods one by one.\n\nExtract copy_oids_to_commits(), which fills the commits list\nwith the distinct commits from the oids list. During this loop,\nit also counts the number of \"extra\" edges from octopus merges.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 57 ++++++++++++++++++++++++++++----------------------\n 1 file changed, 32 insertions(+), 25 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f7419c919b..16cdd7afb2 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -984,6 +984,37 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n \treturn count_distinct;\n }\n \n+static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n+{\n+\tuint32_t i;\n+\tstruct commit_list *parent;\n+\n+\tctx->num_extra_edges = 0;\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t_(\"Finding extra edges in commit graph\"),\n+\t\t\tctx->oids.nr);\n+\tfor (i = 0; i < ctx->oids.nr; i++) {\n+\t\tint num_parents = 0;\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tif (i > 0 && oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n+\t\t\tcontinue;\n+\n+\t\tctx->commits.list[ctx->commits.nr] = lookup_commit(ctx->r, &ctx->oids.list[i]);\n+\t\tparse_commit_no_graph(ctx->commits.list[ctx->commits.nr]);\n+\n+\t\tfor (parent = ctx->commits.list[ctx->commits.nr]->parents;\n+\t\t parent; parent = parent->next)\n+\t\t\tnum_parents++;\n+\n+\t\tif (num_parents > 2)\n+\t\t\tctx->num_extra_edges += num_parents - 1;\n+\n+\t\tctx->commits.nr++;\n+\t}\n+\tstop_progress(&ctx->progress);\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -997,7 +1028,6 @@ int write_commit_graph(const char *obj_dir,\n \tuint32_t chunk_ids[5];\n \tuint64_t chunk_offsets[5];\n \tint num_chunks;\n-\tstruct commit_list *parent;\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tint res = 0;\n@@ -1056,30 +1086,7 @@ int write_commit_graph(const char *obj_dir,\n \tctx->commits.alloc = count_distinct;\n \tALLOC_ARRAY(ctx->commits.list, ctx->commits.alloc);\n \n-\tctx->num_extra_edges = 0;\n-\tif (ctx->report_progress)\n-\t\tctx->progress = start_delayed_progress(\n-\t\t\t_(\"Finding extra edges in commit graph\"),\n-\t\t\tctx->oids.nr);\n-\tfor (i = 0; i < ctx->oids.nr; i++) {\n-\t\tint num_parents = 0;\n-\t\tdisplay_progress(ctx->progress, i + 1);\n-\t\tif (i > 0 && oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n-\t\t\tcontinue;\n-\n-\t\tctx->commits.list[ctx->commits.nr] = lookup_commit(ctx->r, &ctx->oids.list[i]);\n-\t\tparse_commit_no_graph(ctx->commits.list[ctx->commits.nr]);\n-\n-\t\tfor (parent = ctx->commits.list[ctx->commits.nr]->parents;\n-\t\t parent; parent = parent->next)\n-\t\t\tnum_parents++;\n-\n-\t\tif (num_parents > 2)\n-\t\t\tctx->num_extra_edges += num_parents - 1;\n-\n-\t\tctx->commits.nr++;\n-\t}\n-\tstop_progress(&ctx->progress);\n+\tcopy_oids_to_commits(ctx);\n \n \tif (ctx->commits.nr >= GRAPH_EDGE_LAST_MASK) {\n \t\terror(_(\"too many commits to write graph\"));\n-- \ngitgitgadget\n\n"},{"id":"375237","messageId":"fc81c8946dd53d926b03ac94e1d9e11b4f9b8fc3.1557411749.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 11/11] commit-graph: extract write_commit_graph_file()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-09T14:22:40Z","receivedAt":"2019-05-09T14:22:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too complex, so we are\nextracting methods one by one.\n\nExtract write_commit_graph_file() that takes all of the information\nin the context struct and writes the data to a commit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 155 +++++++++++++++++++++++++------------------------\n 1 file changed, 80 insertions(+), 75 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 16cdd7afb2..7723156964 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1015,21 +1015,91 @@ static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n-int write_commit_graph(const char *obj_dir,\n-\t\t struct string_list *pack_indexes,\n-\t\t struct string_list *commit_hex,\n-\t\t unsigned int flags)\n+static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n {\n-\tstruct write_commit_graph_context *ctx;\n+\tuint32_t i;\n \tstruct hashfile *f;\n-\tuint32_t i, count_distinct = 0;\n-\tchar *graph_name = NULL;\n \tstruct lock_file lk = LOCK_INIT;\n \tuint32_t chunk_ids[5];\n \tuint64_t chunk_offsets[5];\n-\tint num_chunks;\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n+\tint num_chunks = ctx->num_extra_edges ? 4 : 3;\n+\n+\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n+\tif (safe_create_leading_directories(ctx->graph_name)) {\n+\t\tUNLEAK(ctx->graph_name);\n+\t\terror(_(\"unable to create leading directories of %s\"),\n+\t\t\tctx->graph_name);\n+\t\treturn errno;\n+\t}\n+\n+\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n+\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n+\n+\thashwrite_be32(f, GRAPH_SIGNATURE);\n+\n+\thashwrite_u8(f, GRAPH_VERSION);\n+\thashwrite_u8(f, oid_version());\n+\thashwrite_u8(f, num_chunks);\n+\thashwrite_u8(f, 0); /* unused padding byte */\n+\n+\tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n+\tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n+\tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n+\tif (ctx->num_extra_edges)\n+\t\tchunk_ids[3] = GRAPH_CHUNKID_EXTRAEDGES;\n+\telse\n+\t\tchunk_ids[3] = 0;\n+\tchunk_ids[4] = 0;\n+\n+\tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n+\tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n+\tchunk_offsets[2] = chunk_offsets[1] + hashsz * ctx->commits.nr;\n+\tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * ctx->commits.nr;\n+\tchunk_offsets[4] = chunk_offsets[3] + 4 * ctx->num_extra_edges;\n+\n+\tfor (i = 0; i <= num_chunks; i++) {\n+\t\tuint32_t chunk_write[3];\n+\n+\t\tchunk_write[0] = htonl(chunk_ids[i]);\n+\t\tchunk_write[1] = htonl(chunk_offsets[i] >> 32);\n+\t\tchunk_write[2] = htonl(chunk_offsets[i] & 0xffffffff);\n+\t\thashwrite(f, chunk_write, 12);\n+\t}\n+\n+\tif (ctx->report_progress) {\n+\t\tstrbuf_addf(&progress_title,\n+\t\t\t Q_(\"Writing out commit graph in %d pass\",\n+\t\t\t \"Writing out commit graph in %d passes\",\n+\t\t\t num_chunks),\n+\t\t\t num_chunks);\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\tprogress_title.buf,\n+\t\t\tnum_chunks * ctx->commits.nr);\n+\t}\n+\twrite_graph_chunk_fanout(f, ctx);\n+\twrite_graph_chunk_oids(f, hashsz, ctx);\n+\twrite_graph_chunk_data(f, hashsz, ctx);\n+\tif (ctx->num_extra_edges)\n+\t\twrite_graph_chunk_extra_edges(f, ctx);\n+\tstop_progress(&ctx->progress);\n+\tstrbuf_release(&progress_title);\n+\n+\tclose_commit_graph(ctx->r);\n+\tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n+\tcommit_lock_file(&lk);\n+\n+\treturn 0;\n+}\n+\n+int write_commit_graph(const char *obj_dir,\n+\t\t struct string_list *pack_indexes,\n+\t\t struct string_list *commit_hex,\n+\t\t unsigned int flags)\n+{\n+\tstruct write_commit_graph_context *ctx;\n+\tuint32_t i, count_distinct = 0;\n \tint res = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n@@ -1096,75 +1166,10 @@ int write_commit_graph(const char *obj_dir,\n \n \tcompute_generation_numbers(ctx);\n \n-\tnum_chunks = ctx->num_extra_edges ? 4 : 3;\n-\n-\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n-\tif (safe_create_leading_directories(ctx->graph_name)) {\n-\t\tUNLEAK(ctx->graph_name);\n-\t\terror(_(\"unable to create leading directories of %s\"),\n-\t\t\tctx->graph_name);\n-\t\tres = errno;\n-\t\tgoto cleanup;\n-\t}\n-\n-\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n-\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n-\n-\thashwrite_be32(f, GRAPH_SIGNATURE);\n-\n-\thashwrite_u8(f, GRAPH_VERSION);\n-\thashwrite_u8(f, oid_version());\n-\thashwrite_u8(f, num_chunks);\n-\thashwrite_u8(f, 0); /* unused padding byte */\n-\n-\tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n-\tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n-\tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n-\tif (ctx->num_extra_edges)\n-\t\tchunk_ids[3] = GRAPH_CHUNKID_EXTRAEDGES;\n-\telse\n-\t\tchunk_ids[3] = 0;\n-\tchunk_ids[4] = 0;\n-\n-\tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n-\tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n-\tchunk_offsets[2] = chunk_offsets[1] + hashsz * ctx->commits.nr;\n-\tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * ctx->commits.nr;\n-\tchunk_offsets[4] = chunk_offsets[3] + 4 * ctx->num_extra_edges;\n-\n-\tfor (i = 0; i <= num_chunks; i++) {\n-\t\tuint32_t chunk_write[3];\n-\n-\t\tchunk_write[0] = htonl(chunk_ids[i]);\n-\t\tchunk_write[1] = htonl(chunk_offsets[i] >> 32);\n-\t\tchunk_write[2] = htonl(chunk_offsets[i] & 0xffffffff);\n-\t\thashwrite(f, chunk_write, 12);\n-\t}\n-\n-\tif (ctx->report_progress) {\n-\t\tstrbuf_addf(&progress_title,\n-\t\t\t Q_(\"Writing out commit graph in %d pass\",\n-\t\t\t \"Writing out commit graph in %d passes\",\n-\t\t\t num_chunks),\n-\t\t\t num_chunks);\n-\t\tctx->progress = start_delayed_progress(\n-\t\t\tprogress_title.buf,\n-\t\t\tnum_chunks * ctx->commits.nr);\n-\t}\n-\twrite_graph_chunk_fanout(f, ctx);\n-\twrite_graph_chunk_oids(f, hashsz, ctx);\n-\twrite_graph_chunk_data(f, hashsz, ctx);\n-\tif (ctx->num_extra_edges)\n-\t\twrite_graph_chunk_extra_edges(f, ctx);\n-\tstop_progress(&ctx->progress);\n-\tstrbuf_release(&progress_title);\n-\n-\tclose_commit_graph(ctx->r);\n-\tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n-\tcommit_lock_file(&lk);\n+\tres = write_commit_graph_file(ctx);\n \n cleanup:\n-\tfree(graph_name);\n+\tfree(ctx->graph_name);\n \tfree(ctx->commits.list);\n \tfree(ctx->oids.list);\n \tfree(ctx);\n-- \ngitgitgadget\n"},{"id":"375238","messageId":"a37548745b91cb275c0294aab15abdd2c4d3207a.1557411749.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 09/11] commit-graph: extract count_distinct_commits()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-09T14:22:38Z","receivedAt":"2019-05-09T14:22:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too complex, so we are\nextracting methods one by one.\n\nExtract count_distinct_commits(), which sorts the oids list, then\niterates through to find duplicates.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 35 ++++++++++++++++++++++-------------\n 1 file changed, 22 insertions(+), 13 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 730d529815..f7419c919b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -963,6 +963,27 @@ static void fill_oids_from_all_packs(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n+static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n+{\n+\tuint32_t i, count_distinct = 1;\n+\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t_(\"Counting distinct commits in commit graph\"),\n+\t\t\tctx->oids.nr);\n+\tdisplay_progress(ctx->progress, 0); /* TODO: Measure QSORT() progress */\n+\tQSORT(ctx->oids.list, ctx->oids.nr, commit_compare);\n+\n+\tfor (i = 1; i < ctx->oids.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n+\t\t\tcount_distinct++;\n+\t}\n+\tstop_progress(&ctx->progress);\n+\n+\treturn count_distinct;\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -1024,19 +1045,7 @@ int write_commit_graph(const char *obj_dir,\n \n \tclose_reachable(ctx);\n \n-\tif (ctx->report_progress)\n-\t\tctx->progress = start_delayed_progress(\n-\t\t\t_(\"Counting distinct commits in commit graph\"),\n-\t\t\tctx->oids.nr);\n-\tdisplay_progress(ctx->progress, 0); /* TODO: Measure QSORT() progress */\n-\tQSORT(ctx->oids.list, ctx->oids.nr, commit_compare);\n-\tcount_distinct = 1;\n-\tfor (i = 1; i < ctx->oids.nr; i++) {\n-\t\tdisplay_progress(ctx->progress, i + 1);\n-\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n-\t\t\tcount_distinct++;\n-\t}\n-\tstop_progress(&ctx->progress);\n+\tcount_distinct = count_distinct_commits(ctx);\n \n \tif (count_distinct >= GRAPH_EDGE_LAST_MASK) {\n \t\terror(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n-- \ngitgitgadget\n\n"},{"id":"375239","messageId":"8cb2613dfa5cad1f6872a1fe1cf8e317a1bc400e.1557411749.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 07/11] commit-graph: extract fill_oids_from_commit_hex()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-09T14:22:36Z","receivedAt":"2019-05-09T14:22:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too complex, so we are\nextracting methods one by one.\n\nExtract fill_oids_from_commit_hex() that reads the given commit\nid list and fille the oid list in the context.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 72 ++++++++++++++++++++++++++++----------------------\n 1 file changed, 40 insertions(+), 32 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 80c7069aaa..fb25280df1 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -912,6 +912,44 @@ static int fill_oids_from_packs(struct write_commit_graph_context *ctx,\n \treturn 0;\n }\n \n+static void fill_oids_from_commit_hex(struct write_commit_graph_context *ctx,\n+\t\t\t\t struct string_list *commit_hex)\n+{\n+\tuint32_t i;\n+\tstruct strbuf progress_title = STRBUF_INIT;\n+\n+\tif (ctx->report_progress) {\n+\t\tstrbuf_addf(&progress_title,\n+\t\t\t Q_(\"Finding commits for commit graph from %d ref\",\n+\t\t\t \"Finding commits for commit graph from %d refs\",\n+\t\t\t commit_hex->nr),\n+\t\t\t commit_hex->nr);\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\tprogress_title.buf,\n+\t\t\t\t\tcommit_hex->nr);\n+\t}\n+\tfor (i = 0; i < commit_hex->nr; i++) {\n+\t\tconst char *end;\n+\t\tstruct object_id oid;\n+\t\tstruct commit *result;\n+\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tif (commit_hex->items[i].string &&\n+\t\t parse_oid_hex(commit_hex->items[i].string, &oid, &end))\n+\t\t\tcontinue;\n+\n+\t\tresult = lookup_commit_reference_gently(ctx->r, &oid, 1);\n+\n+\t\tif (result) {\n+\t\t\tALLOC_GROW(ctx->oids.list, ctx->oids.nr + 1, ctx->oids.alloc);\n+\t\t\toidcpy(&ctx->oids.list[ctx->oids.nr], &(result->object.oid));\n+\t\t\tctx->oids.nr++;\n+\t\t}\n+\t}\n+\tstop_progress(&ctx->progress);\n+\tstrbuf_release(&progress_title);\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -965,38 +1003,8 @@ int write_commit_graph(const char *obj_dir,\n \t\t\tgoto cleanup;\n \t}\n \n-\tif (commit_hex) {\n-\t\tif (ctx->report_progress) {\n-\t\t\tstrbuf_addf(&progress_title,\n-\t\t\t\t Q_(\"Finding commits for commit graph from %d ref\",\n-\t\t\t\t \"Finding commits for commit graph from %d refs\",\n-\t\t\t\t commit_hex->nr),\n-\t\t\t\t commit_hex->nr);\n-\t\t\tctx->progress = start_delayed_progress(\n-\t\t\t\t\t\tprogress_title.buf,\n-\t\t\t\t\t\tcommit_hex->nr);\n-\t\t}\n-\t\tfor (i = 0; i < commit_hex->nr; i++) {\n-\t\t\tconst char *end;\n-\t\t\tstruct object_id oid;\n-\t\t\tstruct commit *result;\n-\n-\t\t\tdisplay_progress(ctx->progress, i + 1);\n-\t\t\tif (commit_hex->items[i].string &&\n-\t\t\t parse_oid_hex(commit_hex->items[i].string, &oid, &end))\n-\t\t\t\tcontinue;\n-\n-\t\t\tresult = lookup_commit_reference_gently(ctx->r, &oid, 1);\n-\n-\t\t\tif (result) {\n-\t\t\t\tALLOC_GROW(ctx->oids.list, ctx->oids.nr + 1, ctx->oids.alloc);\n-\t\t\t\toidcpy(&ctx->oids.list[ctx->oids.nr], &(result->object.oid));\n-\t\t\t\tctx->oids.nr++;\n-\t\t\t}\n-\t\t}\n-\t\tstop_progress(&ctx->progress);\n-\t\tstrbuf_reset(&progress_title);\n-\t}\n+\tif (commit_hex)\n+\t\tfill_oids_from_commit_hex(ctx, commit_hex);\n \n \tif (!pack_indexes && !commit_hex) {\n \t\tif (ctx->report_progress)\n-- \ngitgitgadget\n\n"},{"id":"375246","messageId":"20190509175848.GE26779@google.com","threadId":"50302","inReplyTo":"pull.112.v4.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 00/11] Commit-graph write refactor (was: Create commit-graph file format v2)","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2019-05-09T17:58:48Z","receivedAt":"2019-05-09T17:58:57Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2019.05.09 07:22, Derrick Stolee via GitGitGadget wrote:\n> This series replaces ds/commit-graph-file-v2, and I'm using the same\n> gitgitgadget PR to continue the version numbers and hopefully make that\n> clear. This is a slight modification on patches 1-11 from the incremental\n> file format RFC [0].\n> \n> The commit-graph feature is growing, thanks to all of the contributions by\n> several community members. This also means that the write_commit_graph()\n> method is a bit unwieldy now. This series refactors that method to use a\n> write_commit_graph_context struct that is passed between several smaller\n> methods. The final result should be a write_commit_graph() method that has a\n> clear set of steps. Future changes should then be easier to understand.\n\nThis series looks good to me.\n"},{"id":"375357","messageId":"xmqqbm07dptx.fsf@gitster-ct.c.googlers.com","threadId":"50302","inReplyTo":"0be7713a255d6f52c51ccb19299d0c9797e9a03f.1557411749.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 01/11] commit-graph: fix the_repository reference","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-05-13T02:56:58Z","receivedAt":"2019-05-13T02:57:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> The parse_commit_buffer() method takes a repository pointer, so it\n> should not refer to the_repository anymore.\n\nYup, makes sense. Thanks for spotting.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit.c | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/commit.c b/commit.c\n> index a5333c7ac6..e4d1233226 100644\n> --- a/commit.c\n> +++ b/commit.c\n> @@ -443,7 +443,7 @@ int parse_commit_buffer(struct repository *r, struct commit *item, const void *b\n> \titem->date = parse_commit_date(bufptr, tail);\n> \n> \tif (check_graph)\n> -\t\tload_commit_graph_info(the_repository, item);\n> +\t\tload_commit_graph_info(r, item);\n> \n> \treturn 0;\n> }\n"},{"id":"375358","messageId":"xmqq36ljdp2s.fsf@gitster-ct.c.googlers.com","threadId":"50302","inReplyTo":"a4082b827eb1999f6e46780b4147962d6dd87975.1557411749.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 02/11] commit-graph: return with errors during write","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-05-13T03:13:15Z","receivedAt":"2019-05-13T03:13:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> The write_commit_graph() method uses die() to report failure and\n> exit when confronted with an unexpected condition. This use of\n> die() in a library function is incorrect and is now replaced by\n> error() statements and an int return type.\n\nOK.\n\n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index 537fdfd0f0..2e86251f02 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -141,6 +141,7 @@ static int graph_write(int argc, const char **argv)\n> \tstruct string_list *pack_indexes = NULL;\n> \tstruct string_list *commit_hex = NULL;\n> \tstruct string_list lines;\n> +\tint result;\n> \n> \tstatic struct option builtin_commit_graph_write_options[] = {\n> \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n> @@ -168,10 +169,8 @@ static int graph_write(int argc, const char **argv)\n> \n> \tread_replace_refs = 0;\n> \n> -\tif (opts.reachable) {\n> -\t\twrite_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n> -\t\treturn 0;\n> -\t}\n> +\tif (opts.reachable)\n> +\t\treturn write_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n> \n> \tstring_list_init(&lines, 0);\n> \tif (opts.stdin_packs || opts.stdin_commits) {\n> @@ -188,14 +187,14 @@ static int graph_write(int argc, const char **argv)\n> \t\tUNLEAK(buf);\n> \t}\n> \n> -\twrite_commit_graph(opts.obj_dir,\n> -\t\t\t pack_indexes,\n> -\t\t\t commit_hex,\n> -\t\t\t opts.append,\n> -\t\t\t 1);\n> +\tresult = write_commit_graph(opts.obj_dir,\n> +\t\t\t\t pack_indexes,\n> +\t\t\t\t commit_hex,\n> +\t\t\t\t opts.append,\n> +\t\t\t\t 1);\n> \n> \tUNLEAK(lines);\n> -\treturn 0;\n> +\treturn result;\n> }\n\nWhat were the error values this function used to return? I am\nwondering if the callers of this function are prepraed to see the\nreturned values from write_commit_graph() this function stores in\n'result' (which presumably are small negative value like our usual\ninternal API convention)?\n\n\n> diff --git a/builtin/commit.c b/builtin/commit.c\n> index 2986553d5f..b9ea7222fa 100644\n> --- a/builtin/commit.c\n> +++ b/builtin/commit.c\n> @@ -1669,8 +1669,9 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n> \t\t \"new_index file. Check that disk is not full and quota is\\n\"\n> \t\t \"not exceeded, and then \\\"git reset HEAD\\\" to recover.\"));\n> \n> -\tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0))\n> -\t\twrite_commit_graph_reachable(get_object_directory(), 0, 0);\n> +\tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n> +\t write_commit_graph_reachable(get_object_directory(), 0, 0))\n> +\t\treturn 1;\n\nThis is good. An error signalled as a small negative integer would\nnot seep thru to the exit status but is explicitly turned into 1\nwith this change.\n\n> +\tif (gc_write_commit_graph &&\n> +\t write_commit_graph_reachable(get_object_directory(), 0,\n> +\t\t\t\t\t !quiet && !daemonized))\n> +\t\treturn 1;\n\nDitto.\n\n> +int write_commit_graph_reachable(const char *obj_dir, int append,\n> +\t\t\t\t int report_progress)\n> {\n> \tstruct string_list list = STRING_LIST_INIT_DUP;\n> +\tint result;\n> \n> \tfor_each_ref(add_ref_to_list, &list);\n> -\twrite_commit_graph(obj_dir, NULL, &list, append, report_progress);\n> +\tresult = write_commit_graph(obj_dir, NULL, &list,\n> +\t\t\t\t append, report_progress);\n> \n> \tstring_list_clear(&list, 0);\n> +\treturn result;\n> }\n\nOK. The callers of write_commit_graph_reachable() can be careful\nabout its return values to the same degree as the callers of\nwrite_commit_graph().\n\nThese functions perhaps deserves\n/*\n * returns X when ....\n */\nin front (or in *.h)?\n\n> +int write_commit_graph(const char *obj_dir,\n> +\t\t struct string_list *pack_indexes,\n> +\t\t struct string_list *commit_hex,\n> +\t\t int append, int report_progress)\n> {\n> \tstruct packed_oid_list oids;\n> \tstruct packed_commit_list commits;\n> \tstruct hashfile *f;\n> \tuint32_t i, count_distinct = 0;\n> -\tchar *graph_name;\n> +\tchar *graph_name = NULL;\n> \tstruct lock_file lk = LOCK_INIT;\n> \tuint32_t chunk_ids[5];\n> \tuint64_t chunk_offsets[5];\n> @@ -883,15 +886,17 @@ void write_commit_graph(const char *obj_dir,\n> \tuint64_t progress_cnt = 0;\n> \tstruct strbuf progress_title = STRBUF_INIT;\n> \tunsigned long approx_nr_objects;\n> +\tint res = 0;\n> \n> \tif (!commit_graph_compatible(the_repository))\n> -\t\treturn;\n> +\t\treturn 0;\n\nOK. I tend to find \"return 0\" easier to read/follow than \"return\nres\" here.\n\n> \toids.nr = 0;\n> \tapprox_nr_objects = approximate_object_count();\n> \toids.alloc = approx_nr_objects / 32;\n> \toids.progress = NULL;\n> \toids.progress_done = 0;\n> +\tcommits.list = NULL;\n> \n> \tif (append) {\n> \t\tprepare_commit_graph_one(the_repository, obj_dir);\n> @@ -932,10 +937,16 @@ void write_commit_graph(const char *obj_dir,\n> \t\t\tstrbuf_setlen(&packname, dirlen);\n> \t\t\tstrbuf_addstr(&packname, pack_indexes->items[i].string);\n> \t\t\tp = add_packed_git(packname.buf, packname.len, 1);\n> -\t\t\tif (!p)\n> -\t\t\t\tdie(_(\"error adding pack %s\"), packname.buf);\n> -\t\t\tif (open_pack_index(p))\n> -\t\t\t\tdie(_(\"error opening index for %s\"), packname.buf);\n> +\t\t\tif (!p) {\n> +\t\t\t\terror(_(\"error adding pack %s\"), packname.buf);\n> +\t\t\t\tres = 1;\n> +\t\t\t\tgoto cleanup;\n> +\t\t\t}\n> +\t\t\tif (open_pack_index(p)) {\n> +\t\t\t\terror(_(\"error opening index for %s\"), packname.buf);\n> +\t\t\t\tres = 1;\n> +\t\t\t\tgoto cleanup;\n> +\t\t\t}\n\nHmph, is this signal an error by returning a positive \"1\"? That's a\nbit unusual.\n\n> @@ -1006,8 +1017,11 @@ void write_commit_graph(const char *obj_dir,\n> \t}\n> \tstop_progress(&progress);\n> \n> -\tif (count_distinct >= GRAPH_EDGE_LAST_MASK)\n> -\t\tdie(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n> +\tif (count_distinct >= GRAPH_EDGE_LAST_MASK) {\n> +\t\terror(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n> +\t\tres = 1;\n> +\t\tgoto cleanup;\n> +\t}\n> \n> \tcommits.nr = 0;\n> \tcommits.alloc = count_distinct;\n> @@ -1039,16 +1053,21 @@ void write_commit_graph(const char *obj_dir,\n> \tnum_chunks = num_extra_edges ? 4 : 3;\n> \tstop_progress(&progress);\n> \n> -\tif (commits.nr >= GRAPH_EDGE_LAST_MASK)\n> -\t\tdie(_(\"too many commits to write graph\"));\n> +\tif (commits.nr >= GRAPH_EDGE_LAST_MASK) {\n> +\t\terror(_(\"too many commits to write graph\"));\n> +\t\tres = 1;\n> +\t\tgoto cleanup;\n> +\t}\n> \n> \tcompute_generation_numbers(&commits, report_progress);\n> \n> \tgraph_name = get_commit_graph_filename(obj_dir);\n> \tif (safe_create_leading_directories(graph_name)) {\n> \t\tUNLEAK(graph_name);\n> -\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n> -\t\t\t graph_name);\n> +\t\terror(_(\"unable to create leading directories of %s\"),\n> +\t\t\tgraph_name);\n> +\t\tres = errno;\n> +\t\tgoto cleanup;\n> \t}\n\nHmph. Do we know errno==0 means no error everywhere? Do we know\nerrno==1 is not used by anybody as a meaningful value?\n\nWhat I am getting at is if a hardcoded \"1\" we saw above as \"error\nexists but we are not telling the caller what kind of system-level\nerror led to it by returning errno\" (and a hardcoded \"0\" as \"there\nis no error\") are consistent with this use of \"res\" where \"the\ncallers are allowed to learn what system-level error led to this\nerror return from this function by sending the return value of this\nfunction to strerror() or comparing with EWHATEVER\". I do not think\nthis is a good design.\n\n"},{"id":"375359","messageId":"xmqqy33bc91q.fsf@gitster-ct.c.googlers.com","threadId":"50302","inReplyTo":"469d0c9a32ba5a3f797b5bb50f37d6ac7746385b.1557411749.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 03/11] commit-graph: collapse parameters into flags","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-05-13T03:44:49Z","receivedAt":"2019-05-13T03:44:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> The write_commit_graph() and write_commit_graph_reachable() methods\n> currently take two boolean parameters: 'append' and 'report_progress'.\n> We will soon expand the possible options to send to these methods, so\n> instead of complicating the parameter list, first simplify it.\n\nI think this change to introduce \"flags\" and pack these two into a\nsingle parameter, even if there is no plan to add code that starts\nusing third and subsequent bits immediately.\n\nWe are no longer adding anything beyond PROGRESS and APPEND in this\nseries, no?\n\n>\n> Collapse these parameters into a 'flags' parameter, and adjust the\n> callers to provide flags as necessary.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/commit-graph.c | 8 +++++---\n> builtin/commit.c | 2 +-\n> builtin/gc.c | 4 ++--\n> commit-graph.c | 9 +++++----\n> commit-graph.h | 8 +++++---\n> 5 files changed, 18 insertions(+), 13 deletions(-)\n>\n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index 2e86251f02..828b1a713f 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -142,6 +142,7 @@ static int graph_write(int argc, const char **argv)\n> \tstruct string_list *commit_hex = NULL;\n> \tstruct string_list lines;\n> \tint result;\n> +\tint flags = COMMIT_GRAPH_PROGRESS;\n\nMake it a habit to use \"unsigned\" not a signed type, when you pack a\ncollection of bits into a flag word, unless you are treating the MSB\nspecially, e.g. checking to see if it is negative is cheaper than\nmasking with MSB to see if it is set.\n\n> ...\n> \tresult = write_commit_graph(opts.obj_dir,\n> \t\t\t\t pack_indexes,\n> \t\t\t\t commit_hex,\n> -\t\t\t\t opts.append,\n> -\t\t\t\t 1);\n> +\t\t\t\t flags);\n> ...\n> -int write_commit_graph_reachable(const char *obj_dir, int append,\n> -\t\t\t\t int report_progress)\n> +int write_commit_graph_reachable(const char *obj_dir, unsigned int flags)\n> ...\n> int write_commit_graph(const char *obj_dir,\n> \t\t struct string_list *pack_indexes,\n> \t\t struct string_list *commit_hex,\n> -\t\t int append, int report_progress)\n> +\t\t unsigned int flags)\n\nOK, so the receivers of the flags word know the collection is\nunsigned; it's just the user of the API in graph_write() that gets\nthe signedness wrong. OK, easy enough to correct, I guess.\n\n"},{"id":"375361","messageId":"xmqqtvdzc5bc.fsf@gitster-ct.c.googlers.com","threadId":"50302","inReplyTo":"30c1b618b139e637122d9476f54d5eb8ce8e0d57.1557411749.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 06/11] commit-graph: extract fill_oids_from_packs()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-05-13T05:05:27Z","receivedAt":"2019-05-13T05:05:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> The write_commit_graph() method is too complex, so we are\n> extracting methods one by one.\n>\n> This extracts fill_oids_from_packs() that reads the given\n> pack-file list and fills the oid list in the context.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n\nQuite straight-forward. Looking good.\n\nThanks.\n"},{"id":"375362","messageId":"xmqqpnonc547.fsf@gitster-ct.c.googlers.com","threadId":"50302","inReplyTo":"fc81c8946dd53d926b03ac94e1d9e11b4f9b8fc3.1557411749.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 11/11] commit-graph: extract write_commit_graph_file()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-05-13T05:09:44Z","receivedAt":"2019-05-13T05:10:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> The write_commit_graph() method is too complex, so we are\n> extracting methods one by one.\n>\n> Extract write_commit_graph_file() that takes all of the information\n> in the context struct and writes the data to a commit-graph file.\n\nThe later parts of splitting pieces out of write_commit_graph() into\nseparate helper functions look all sensible. One big benefit of\ndoing this, even if each of these helper functions have a single\ncaller, is that each of these individual steps now has a descriptive\nname.\n\nModule a few nits (and possibly s/method/helper function/g), the\nseries look good to me.\n\nThanks.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 155 +++++++++++++++++++++++++------------------------\n> 1 file changed, 80 insertions(+), 75 deletions(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 16cdd7afb2..7723156964 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -1015,21 +1015,91 @@ static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n> \tstop_progress(&ctx->progress);\n> }\n> \n> -int write_commit_graph(const char *obj_dir,\n> -\t\t struct string_list *pack_indexes,\n> -\t\t struct string_list *commit_hex,\n> -\t\t unsigned int flags)\n> +static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n> {\n> -\tstruct write_commit_graph_context *ctx;\n> +\tuint32_t i;\n> \tstruct hashfile *f;\n> -\tuint32_t i, count_distinct = 0;\n> -\tchar *graph_name = NULL;\n> \tstruct lock_file lk = LOCK_INIT;\n> \tuint32_t chunk_ids[5];\n> \tuint64_t chunk_offsets[5];\n> -\tint num_chunks;\n> \tconst unsigned hashsz = the_hash_algo->rawsz;\n> \tstruct strbuf progress_title = STRBUF_INIT;\n> +\tint num_chunks = ctx->num_extra_edges ? 4 : 3;\n> +\n> +\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n> +\tif (safe_create_leading_directories(ctx->graph_name)) {\n> +\t\tUNLEAK(ctx->graph_name);\n> +\t\terror(_(\"unable to create leading directories of %s\"),\n> +\t\t\tctx->graph_name);\n> +\t\treturn errno;\n> +\t}\n> +\n> +\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n> +\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n> +\n> +\thashwrite_be32(f, GRAPH_SIGNATURE);\n> +\n> +\thashwrite_u8(f, GRAPH_VERSION);\n> +\thashwrite_u8(f, oid_version());\n> +\thashwrite_u8(f, num_chunks);\n> +\thashwrite_u8(f, 0); /* unused padding byte */\n> +\n> +\tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n> +\tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n> +\tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n> +\tif (ctx->num_extra_edges)\n> +\t\tchunk_ids[3] = GRAPH_CHUNKID_EXTRAEDGES;\n> +\telse\n> +\t\tchunk_ids[3] = 0;\n> +\tchunk_ids[4] = 0;\n> +\n> +\tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n> +\tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n> +\tchunk_offsets[2] = chunk_offsets[1] + hashsz * ctx->commits.nr;\n> +\tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * ctx->commits.nr;\n> +\tchunk_offsets[4] = chunk_offsets[3] + 4 * ctx->num_extra_edges;\n> +\n> +\tfor (i = 0; i <= num_chunks; i++) {\n> +\t\tuint32_t chunk_write[3];\n> +\n> +\t\tchunk_write[0] = htonl(chunk_ids[i]);\n> +\t\tchunk_write[1] = htonl(chunk_offsets[i] >> 32);\n> +\t\tchunk_write[2] = htonl(chunk_offsets[i] & 0xffffffff);\n> +\t\thashwrite(f, chunk_write, 12);\n> +\t}\n> +\n> +\tif (ctx->report_progress) {\n> +\t\tstrbuf_addf(&progress_title,\n> +\t\t\t Q_(\"Writing out commit graph in %d pass\",\n> +\t\t\t \"Writing out commit graph in %d passes\",\n> +\t\t\t num_chunks),\n> +\t\t\t num_chunks);\n> +\t\tctx->progress = start_delayed_progress(\n> +\t\t\tprogress_title.buf,\n> +\t\t\tnum_chunks * ctx->commits.nr);\n> +\t}\n> +\twrite_graph_chunk_fanout(f, ctx);\n> +\twrite_graph_chunk_oids(f, hashsz, ctx);\n> +\twrite_graph_chunk_data(f, hashsz, ctx);\n> +\tif (ctx->num_extra_edges)\n> +\t\twrite_graph_chunk_extra_edges(f, ctx);\n> +\tstop_progress(&ctx->progress);\n> +\tstrbuf_release(&progress_title);\n> +\n> +\tclose_commit_graph(ctx->r);\n> +\tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n> +\tcommit_lock_file(&lk);\n> +\n> +\treturn 0;\n> +}\n> +\n> +int write_commit_graph(const char *obj_dir,\n> +\t\t struct string_list *pack_indexes,\n> +\t\t struct string_list *commit_hex,\n> +\t\t unsigned int flags)\n> +{\n> +\tstruct write_commit_graph_context *ctx;\n> +\tuint32_t i, count_distinct = 0;\n> \tint res = 0;\n> \n> \tif (!commit_graph_compatible(the_repository))\n> @@ -1096,75 +1166,10 @@ int write_commit_graph(const char *obj_dir,\n> \n> \tcompute_generation_numbers(ctx);\n> \n> -\tnum_chunks = ctx->num_extra_edges ? 4 : 3;\n> -\n> -\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n> -\tif (safe_create_leading_directories(ctx->graph_name)) {\n> -\t\tUNLEAK(ctx->graph_name);\n> -\t\terror(_(\"unable to create leading directories of %s\"),\n> -\t\t\tctx->graph_name);\n> -\t\tres = errno;\n> -\t\tgoto cleanup;\n> -\t}\n> -\n> -\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n> -\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n> -\n> -\thashwrite_be32(f, GRAPH_SIGNATURE);\n> -\n> -\thashwrite_u8(f, GRAPH_VERSION);\n> -\thashwrite_u8(f, oid_version());\n> -\thashwrite_u8(f, num_chunks);\n> -\thashwrite_u8(f, 0); /* unused padding byte */\n> -\n> -\tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n> -\tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n> -\tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n> -\tif (ctx->num_extra_edges)\n> -\t\tchunk_ids[3] = GRAPH_CHUNKID_EXTRAEDGES;\n> -\telse\n> -\t\tchunk_ids[3] = 0;\n> -\tchunk_ids[4] = 0;\n> -\n> -\tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n> -\tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n> -\tchunk_offsets[2] = chunk_offsets[1] + hashsz * ctx->commits.nr;\n> -\tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * ctx->commits.nr;\n> -\tchunk_offsets[4] = chunk_offsets[3] + 4 * ctx->num_extra_edges;\n> -\n> -\tfor (i = 0; i <= num_chunks; i++) {\n> -\t\tuint32_t chunk_write[3];\n> -\n> -\t\tchunk_write[0] = htonl(chunk_ids[i]);\n> -\t\tchunk_write[1] = htonl(chunk_offsets[i] >> 32);\n> -\t\tchunk_write[2] = htonl(chunk_offsets[i] & 0xffffffff);\n> -\t\thashwrite(f, chunk_write, 12);\n> -\t}\n> -\n> -\tif (ctx->report_progress) {\n> -\t\tstrbuf_addf(&progress_title,\n> -\t\t\t Q_(\"Writing out commit graph in %d pass\",\n> -\t\t\t \"Writing out commit graph in %d passes\",\n> -\t\t\t num_chunks),\n> -\t\t\t num_chunks);\n> -\t\tctx->progress = start_delayed_progress(\n> -\t\t\tprogress_title.buf,\n> -\t\t\tnum_chunks * ctx->commits.nr);\n> -\t}\n> -\twrite_graph_chunk_fanout(f, ctx);\n> -\twrite_graph_chunk_oids(f, hashsz, ctx);\n> -\twrite_graph_chunk_data(f, hashsz, ctx);\n> -\tif (ctx->num_extra_edges)\n> -\t\twrite_graph_chunk_extra_edges(f, ctx);\n> -\tstop_progress(&ctx->progress);\n> -\tstrbuf_release(&progress_title);\n> -\n> -\tclose_commit_graph(ctx->r);\n> -\tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n> -\tcommit_lock_file(&lk);\n> +\tres = write_commit_graph_file(ctx);\n> \n> cleanup:\n> -\tfree(graph_name);\n> +\tfree(ctx->graph_name);\n> \tfree(ctx->commits.list);\n> \tfree(ctx->oids.list);\n> \tfree(ctx);\n"},{"id":"375376","messageId":"17829620-1084-74e5-54ad-aa95990f4dbd@gmail.com","threadId":"50302","inReplyTo":"xmqq36ljdp2s.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 02/11] commit-graph: return with errors during write","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-13T11:04:39Z","receivedAt":"2019-05-13T11:04:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/12/2019 11:13 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n>\n>> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n>> @@ -188,14 +187,14 @@ static int graph_write(int argc, const char **argv)\n>> \t\tUNLEAK(buf);\n>> \t}\n>> \n>> -\twrite_commit_graph(opts.obj_dir,\n>> -\t\t\t pack_indexes,\n>> -\t\t\t commit_hex,\n>> -\t\t\t opts.append,\n>> -\t\t\t 1);\n>> +\tresult = write_commit_graph(opts.obj_dir,\n>> +\t\t\t\t pack_indexes,\n>> +\t\t\t\t commit_hex,\n>> +\t\t\t\t opts.append,\n>> +\t\t\t\t 1);\n>> \n>> \tUNLEAK(lines);\n>> -\treturn 0;\n>> +\treturn result;\n>> }\n> \n> What were the error values this function used to return? I am\n> wondering if the callers of this function are prepraed to see the\n> returned values from write_commit_graph() this function stores in\n> 'result' (which presumably are small negative value like our usual\n> internal API convention)?\n\nThe only caller is cmd_commit_graph() and it is in this snippet:\n\n if (argc > 0) {\n if (!strcmp(argv[0], \"read\"))\n return graph_read(argc, argv);\n if (!strcmp(argv[0], \"verify\"))\n return graph_verify(argc, argv);\n if (!strcmp(argv[0], \"write\"))\n return graph_write(argc, argv);\n }\n\nSo these return values are passed directly to the result of the\nbuiltin. If that is against convention (passing an error code from\nthe library to the result of the builtin) then I can modify.\n\n> OK. The callers of write_commit_graph_reachable() can be careful\n> about its return values to the same degree as the callers of\n> write_commit_graph().\n> \n> These functions perhaps deserves\n> /*\n> * returns X when ....\n> */\n> in front (or in *.h)?\n\nCan do, in commit-graph.h.\n\n>> +int write_commit_graph(const char *obj_dir,\n>> +\t\t struct string_list *pack_indexes,\n>> +\t\t struct string_list *commit_hex,\n>> +\t\t int append, int report_progress)\n>> {\n>> \tstruct packed_oid_list oids;\n>> \tstruct packed_commit_list commits;\n>> \tstruct hashfile *f;\n>> \tuint32_t i, count_distinct = 0;\n>> -\tchar *graph_name;\n>> +\tchar *graph_name = NULL;\n>> \tstruct lock_file lk = LOCK_INIT;\n>> \tuint32_t chunk_ids[5];\n>> \tuint64_t chunk_offsets[5];\n>> @@ -883,15 +886,17 @@ void write_commit_graph(const char *obj_dir,\n>> \tuint64_t progress_cnt = 0;\n>> \tstruct strbuf progress_title = STRBUF_INIT;\n>> \tunsigned long approx_nr_objects;\n>> +\tint res = 0;\n>> \n>> \tif (!commit_graph_compatible(the_repository))\n>> -\t\treturn;\n>> +\t\treturn 0;\n> \n> OK. I tend to find \"return 0\" easier to read/follow than \"return\n> res\" here.\n\nYes, this choice was deliberate as there is no cleanup to do if we\nreturn this early. Also note that we don't \"fail\" because we did\nexactly as much work as we expect in this scenario. I'll be careful\nto point this out when I add a comment to the header file.\n\n>> \toids.nr = 0;\n>> \tapprox_nr_objects = approximate_object_count();\n>> \toids.alloc = approx_nr_objects / 32;\n>> \toids.progress = NULL;\n>> \toids.progress_done = 0;\n>> +\tcommits.list = NULL;\n>> \n>> \tif (append) {\n>> \t\tprepare_commit_graph_one(the_repository, obj_dir);\n>> @@ -932,10 +937,16 @@ void write_commit_graph(const char *obj_dir,\n>> \t\t\tstrbuf_setlen(&packname, dirlen);\n>> \t\t\tstrbuf_addstr(&packname, pack_indexes->items[i].string);\n>> \t\t\tp = add_packed_git(packname.buf, packname.len, 1);\n>> -\t\t\tif (!p)\n>> -\t\t\t\tdie(_(\"error adding pack %s\"), packname.buf);\n>> -\t\t\tif (open_pack_index(p))\n>> -\t\t\t\tdie(_(\"error opening index for %s\"), packname.buf);\n>> +\t\t\tif (!p) {\n>> +\t\t\t\terror(_(\"error adding pack %s\"), packname.buf);\n>> +\t\t\t\tres = 1;\n>> +\t\t\t\tgoto cleanup;\n>> +\t\t\t}\n>> +\t\t\tif (open_pack_index(p)) {\n>> +\t\t\t\terror(_(\"error opening index for %s\"), packname.buf);\n>> +\t\t\t\tres = 1;\n>> +\t\t\t\tgoto cleanup;\n>> +\t\t\t}\n> \n> Hmph, is this signal an error by returning a positive \"1\"? That's a\n> bit unusual.\n\nYour hint above of \"passing a negative value by convention\" did make me\nthink I must be doing something wrong.\n\n>> @@ -1006,8 +1017,11 @@ void write_commit_graph(const char *obj_dir,\n>> \t}\n>> \tstop_progress(&progress);\n>> \n>> -\tif (count_distinct >= GRAPH_EDGE_LAST_MASK)\n>> -\t\tdie(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n>> +\tif (count_distinct >= GRAPH_EDGE_LAST_MASK) {\n>> +\t\terror(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n>> +\t\tres = 1;\n>> +\t\tgoto cleanup;\n>> +\t}\n>> \n>> \tcommits.nr = 0;\n>> \tcommits.alloc = count_distinct;\n>> @@ -1039,16 +1053,21 @@ void write_commit_graph(const char *obj_dir,\n>> \tnum_chunks = num_extra_edges ? 4 : 3;\n>> \tstop_progress(&progress);\n>> \n>> -\tif (commits.nr >= GRAPH_EDGE_LAST_MASK)\n>> -\t\tdie(_(\"too many commits to write graph\"));\n>> +\tif (commits.nr >= GRAPH_EDGE_LAST_MASK) {\n>> +\t\terror(_(\"too many commits to write graph\"));\n>> +\t\tres = 1;\n>> +\t\tgoto cleanup;\n>> +\t}\n>> \n>> \tcompute_generation_numbers(&commits, report_progress);\n>> \n>> \tgraph_name = get_commit_graph_filename(obj_dir);\n>> \tif (safe_create_leading_directories(graph_name)) {\n>> \t\tUNLEAK(graph_name);\n>> -\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n>> -\t\t\t graph_name);\n>> +\t\terror(_(\"unable to create leading directories of %s\"),\n>> +\t\t\tgraph_name);\n>> +\t\tres = errno;\n>> +\t\tgoto cleanup;\n>> \t}\n> \n> Hmph. Do we know errno==0 means no error everywhere? Do we know\n> errno==1 is not used by anybody as a meaningful value?\n> \n> What I am getting at is if a hardcoded \"1\" we saw above as \"error\n> exists but we are not telling the caller what kind of system-level\n> error led to it by returning errno\" (and a hardcoded \"0\" as \"there\n> is no error\") are consistent with this use of \"res\" where \"the\n> callers are allowed to learn what system-level error led to this\n> error return from this function by sending the return value of this\n> function to strerror() or comparing with EWHATEVER\". I do not think\n> this is a good design.\n\nThat's a good point. In a new design, would you like me to (1) ignore\nerrno here and use a constant value for \"write_commit_graph() failed\nat some point\" or to (2) split the possible _reasons_ for the failure\ninto different constants? I believe the use of error() should prevent\nthe need for the second option. The first option would only change\nthis 'res = errno' into 'res = 1'.\n\nThanks,\n-Stolee\n"},{"id":"375377","messageId":"ad42ab76-76c9-c91c-55a2-a613080c2a01@gmail.com","threadId":"50302","inReplyTo":"xmqqy33bc91q.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 03/11] commit-graph: collapse parameters into flags","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-13T11:07:41Z","receivedAt":"2019-05-13T11:07:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/12/2019 11:44 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> From: Derrick Stolee \n>>\n>> The write_commit_graph() and write_commit_graph_reachable() methods\n>> currently take two boolean parameters: 'append' and 'report_progress'.\n>> We will soon expand the possible options to send to these methods, so\n>> instead of complicating the parameter list, first simplify it.\n> \n> I think this change to introduce \"flags\" and pack these two into a\n> single parameter, even if there is no plan to add code that starts\n> using third and subsequent bits immediately.\n> \n> We are no longer adding anything beyond PROGRESS and APPEND in this\n> series, no?\n\nIn this series, we are no longer expanding the options. I will add\na flag when I update the incremental file format series. I can modify\nthe message to no longer hint at an immediate addition.\n\n>>\n>> Collapse these parameters into a 'flags' parameter, and adjust the\n>> callers to provide flags as necessary.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> builtin/commit-graph.c | 8 +++++---\n>> builtin/commit.c | 2 +-\n>> builtin/gc.c | 4 ++--\n>> commit-graph.c | 9 +++++----\n>> commit-graph.h | 8 +++++---\n>> 5 files changed, 18 insertions(+), 13 deletions(-)\n>>\n>> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n>> index 2e86251f02..828b1a713f 100644\n>> --- a/builtin/commit-graph.c\n>> +++ b/builtin/commit-graph.c\n>> @@ -142,6 +142,7 @@ static int graph_write(int argc, const char **argv)\n>> \tstruct string_list *commit_hex = NULL;\n>> \tstruct string_list lines;\n>> \tint result;\n>> +\tint flags = COMMIT_GRAPH_PROGRESS;\n> \n> Make it a habit to use \"unsigned\" not a signed type, when you pack a\n> collection of bits into a flag word, unless you are treating the MSB\n> specially, e.g. checking to see if it is negative is cheaper than\n> masking with MSB to see if it is set.\n\nAh sorry. I missed this one after changing the parameter in your earlier\nfeedback.\n \n>> ...\n>> \tresult = write_commit_graph(opts.obj_dir,\n>> \t\t\t\t pack_indexes,\n>> \t\t\t\t commit_hex,\n>> -\t\t\t\t opts.append,\n>> -\t\t\t\t 1);\n>> +\t\t\t\t flags);\n>> ...\n>> -int write_commit_graph_reachable(const char *obj_dir, int append,\n>> -\t\t\t\t int report_progress)\n>> +int write_commit_graph_reachable(const char *obj_dir, unsigned int flags)\n>> ...\n>> int write_commit_graph(const char *obj_dir,\n>> \t\t struct string_list *pack_indexes,\n>> \t\t struct string_list *commit_hex,\n>> -\t\t int append, int report_progress)\n>> +\t\t unsigned int flags)\n> \n> OK, so the receivers of the flags word know the collection is\n> unsigned; it's just the user of the API in graph_write() that gets\n> the signedness wrong. OK, easy enough to correct, I guess.\n> \n\n"},{"id":"375379","messageId":"aa78b56d-bf88-3a04-63f5-b1cfa2c6da82@gmail.com","threadId":"50302","inReplyTo":"17829620-1084-74e5-54ad-aa95990f4dbd@gmail.com","subject":"Re: [PATCH v4 02/11] commit-graph: return with errors during write","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-13T11:22:49Z","receivedAt":"2019-05-13T11:22:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/13/2019 7:04 AM, Derrick Stolee wrote:\n> On 5/12/2019 11:13 PM, Junio C Hamano wrote:\n>> \"Derrick Stolee via GitGitGadget\" writes:\n>>\n>>> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n>>> @@ -188,14 +187,14 @@ static int graph_write(int argc, const char **argv)\n>>> \t\tUNLEAK(buf);\n>>> \t}\n>>> \n>>> -\twrite_commit_graph(opts.obj_dir,\n>>> -\t\t\t pack_indexes,\n>>> -\t\t\t commit_hex,\n>>> -\t\t\t opts.append,\n>>> -\t\t\t 1);\n>>> +\tresult = write_commit_graph(opts.obj_dir,\n>>> +\t\t\t\t pack_indexes,\n>>> +\t\t\t\t commit_hex,\n>>> +\t\t\t\t opts.append,\n>>> +\t\t\t\t 1);\n>>> \n>>> \tUNLEAK(lines);\n>>> -\treturn 0;\n>>> +\treturn result;\n>>> }\n>>\n>> What were the error values this function used to return? I am\n>> wondering if the callers of this function are prepraed to see the\n>> returned values from write_commit_graph() this function stores in\n>> 'result' (which presumably are small negative value like our usual\n>> internal API convention)?\n> \n> The only caller is cmd_commit_graph() and it is in this snippet:\n> \n> if (argc > 0) {\n> if (!strcmp(argv[0], \"read\"))\n> return graph_read(argc, argv);\n> if (!strcmp(argv[0], \"verify\"))\n> return graph_verify(argc, argv);\n> if (!strcmp(argv[0], \"write\"))\n> return graph_write(argc, argv);\n> }\n> \n> So these return values are passed directly to the result of the\n> builtin. If that is against convention (passing an error code from\n> the library to the result of the builtin) then I can modify.\n\nAnd I see from your other feedback (upon re-reading) that you prefer\ntranslating a negative error value from the library into a \"1\" here\nfor the builtin.\n\nAs I prepare my next version, I'll have write_commit_graph() return -1\nfor all errors and have graph_write() translate that to a 1. But I'll\nwait to see if you want more specific error codes from write_commit_graph().\n\n-Stolee\n\n"},{"id":"377063","messageId":"pull.112.v5.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v5 00/11] Commit-graph write refactor (was: Create commit-graph file format v2)","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-12T13:29:35Z","receivedAt":"2019-06-12T13:29:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This series replaces ds/commit-graph-file-v2, and I'm using the same\ngitgitgadget PR to continue the version numbers and hopefully make that\nclear. This is a slight modification on patches 1-11 from the incremental\nfile format RFC [0].\n\nThe commit-graph feature is growing, thanks to all of the contributions by\nseveral community members. This also means that the write_commit_graph()\nmethod is a bit unwieldy now. This series refactors that method to use a\nwrite_commit_graph_context struct that is passed between several smaller\nmethods. The final result should be a write_commit_graph() method that has a\nclear set of steps. Future changes should then be easier to understand.\n\n * Patches 1-4: these are small changes which either fix issues or just\n provide clean-up. These are mostly borrowed from\n ds/commit-graph-format-v2. \n \n \n * Patches 5-11: these provide a non-functional refactor of\n write_commit_graph() into several methods using a \"struct\n write_commit_graph_context\" to share across the methods.\n \n \n\nUpdates to commits previously in this thread:\n\n * \"commit-graph: remove Future Work section\" no longer says that 'verify'\n takes as long as 'write'. [1]\n \n \n * \"commit-graph: return with errors during write\" now has a test to check\n we don't die(). [2]\n \n \n\nÆvar: Looking at the old thread, I only saw two comments that still apply to\nthis series [1] [2]. Please point me to any comments I have missed.\n\nUpdates in V5:\n\n * API calls are updated to return 0 on success and a negative value on\n failure.\n \n \n * Stopped passing 'errno' through an API function, instead returns -1.\n \n \n * \"extracting methods\" -> \"extracting helper functions\" in commit messages.\n \n \n * flags are now unsigned ints.\n \n \n\nThanks, -Stolee\n\n[0] https://public-inbox.org/git/pull.184.git.gitgitgadget@gmail.com/\n\n[1] https://public-inbox.org/git/87o94mql0a.fsf@evledraar.gmail.com/\n\n[2] https://public-inbox.org/git/87pnp2qlkv.fsf@evledraar.gmail.com/\n\nDerrick Stolee (11):\n commit-graph: fix the_repository reference\n commit-graph: return with errors during write\n commit-graph: collapse parameters into flags\n commit-graph: remove Future Work section\n commit-graph: create write_commit_graph_context\n commit-graph: extract fill_oids_from_packs()\n commit-graph: extract fill_oids_from_commit_hex()\n commit-graph: extract fill_oids_from_all_packs()\n commit-graph: extract count_distinct_commits()\n commit-graph: extract copy_oids_to_commits()\n commit-graph: extract write_commit_graph_file()\n\n Documentation/technical/commit-graph.txt | 17 -\n builtin/commit-graph.c | 22 +-\n builtin/commit.c | 5 +-\n builtin/gc.c | 7 +-\n commit-graph.c | 607 +++++++++++++----------\n commit-graph.h | 20 +-\n commit.c | 2 +-\n t/t5318-commit-graph.sh | 8 +\n 8 files changed, 378 insertions(+), 310 deletions(-)\n\n\nbase-commit: 93b4405ffe4ad9308740e7c1c71383bfc369baaa\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-112%2Fderrickstolee%2Fgraph%2Fv2-head-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-112/derrickstolee/graph/v2-head-v5\nPull-Request: https://github.com/gitgitgadget/git/pull/112\n\nRange-diff vs v4:\n\n 1: 0be7713a25 = 1: 0be7713a25 commit-graph: fix the_repository reference\n 2: a4082b827e ! 2: 95f66e85b2 commit-graph: return with errors during write\n @@ -5,14 +5,17 @@\n The write_commit_graph() method uses die() to report failure and\n exit when confronted with an unexpected condition. This use of\n die() in a library function is incorrect and is now replaced by\n - error() statements and an int return type.\n + error() statements and an int return type. Return zero on success\n + and a negative value on failure.\n \n Now that we use 'goto cleanup' to jump to the terminal condition\n on an error, we have new paths that could lead to uninitialized\n values. New initializers are added to correct for this.\n \n The builtins 'commit-graph', 'gc', and 'commit' call these methods,\n - so update them to check the return value.\n + so update them to check the return value. Test that 'git commit-graph\n + write' returns a proper error code when hitting a failure condition\n + in write_commit_graph().\n \n Signed-off-by: Derrick Stolee \n \n @@ -23,7 +26,7 @@\n \tstruct string_list *pack_indexes = NULL;\n \tstruct string_list *commit_hex = NULL;\n \tstruct string_list lines;\n -+\tint result;\n ++\tint result = 0;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n @@ -49,11 +52,12 @@\n -\t\t\t commit_hex,\n -\t\t\t opts.append,\n -\t\t\t 1);\n -+\tresult = write_commit_graph(opts.obj_dir,\n -+\t\t\t\t pack_indexes,\n -+\t\t\t\t commit_hex,\n -+\t\t\t\t opts.append,\n -+\t\t\t\t 1);\n ++\tif (write_commit_graph(opts.obj_dir,\n ++\t\t\t pack_indexes,\n ++\t\t\t commit_hex,\n ++\t\t\t opts.append,\n ++\t\t\t 1))\n ++\t\tresult = 1;\n \n \tUNLEAK(lines);\n -\treturn 0;\n @@ -167,12 +171,12 @@\n -\t\t\t\tdie(_(\"error opening index for %s\"), packname.buf);\n +\t\t\tif (!p) {\n +\t\t\t\terror(_(\"error adding pack %s\"), packname.buf);\n -+\t\t\t\tres = 1;\n ++\t\t\t\tres = -1;\n +\t\t\t\tgoto cleanup;\n +\t\t\t}\n +\t\t\tif (open_pack_index(p)) {\n +\t\t\t\terror(_(\"error opening index for %s\"), packname.buf);\n -+\t\t\t\tres = 1;\n ++\t\t\t\tres = -1;\n +\t\t\t\tgoto cleanup;\n +\t\t\t}\n \t\t\tfor_each_object_in_pack(p, add_packed_commits, &oids,\n @@ -186,7 +190,7 @@\n -\t\tdie(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n +\tif (count_distinct >= GRAPH_EDGE_LAST_MASK) {\n +\t\terror(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n -+\t\tres = 1;\n ++\t\tres = -1;\n +\t\tgoto cleanup;\n +\t}\n \n @@ -200,7 +204,7 @@\n -\t\tdie(_(\"too many commits to write graph\"));\n +\tif (commits.nr >= GRAPH_EDGE_LAST_MASK) {\n +\t\terror(_(\"too many commits to write graph\"));\n -+\t\tres = 1;\n ++\t\tres = -1;\n +\t\tgoto cleanup;\n +\t}\n \n @@ -213,7 +217,7 @@\n -\t\t\t graph_name);\n +\t\terror(_(\"unable to create leading directories of %s\"),\n +\t\t\tgraph_name);\n -+\t\tres = errno;\n ++\t\tres = -1;\n +\t\tgoto cleanup;\n \t}\n \n @@ -240,6 +244,12 @@\n int generation_numbers_enabled(struct repository *r);\n \n -void write_commit_graph_reachable(const char *obj_dir, int append,\n ++/*\n ++ * The write_commit_graph* methods return zero on success\n ++ * and a negative value on failure. Note that if the repository\n ++ * is not compatible with the commit-graph feature, then the\n ++ * methods will return 0 without writing a commit-graph.\n ++ */\n +int write_commit_graph_reachable(const char *obj_dir, int append,\n \t\t\t\t int report_progress);\n -void write_commit_graph(const char *obj_dir,\n 3: 469d0c9a32 ! 3: b4e3ae579a commit-graph: collapse parameters into flags\n @@ -4,8 +4,8 @@\n \n The write_commit_graph() and write_commit_graph_reachable() methods\n currently take two boolean parameters: 'append' and 'report_progress'.\n - We will soon expand the possible options to send to these methods, so\n - instead of complicating the parameter list, first simplify it.\n + As we update these methods, adding more parameters this way becomes\n + cluttered and hard to maintain.\n \n Collapse these parameters into a 'flags' parameter, and adjust the\n callers to provide flags as necessary.\n @@ -18,8 +18,8 @@\n @@\n \tstruct string_list *commit_hex = NULL;\n \tstruct string_list lines;\n - \tint result;\n -+\tint flags = COMMIT_GRAPH_PROGRESS;\n + \tint result = 0;\n ++\tunsigned int flags = COMMIT_GRAPH_PROGRESS;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n @@ -39,15 +39,15 @@\n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n @@\n - \tresult = write_commit_graph(opts.obj_dir,\n - \t\t\t\t pack_indexes,\n - \t\t\t\t commit_hex,\n --\t\t\t\t opts.append,\n --\t\t\t\t 1);\n -+\t\t\t\t flags);\n + \tif (write_commit_graph(opts.obj_dir,\n + \t\t\t pack_indexes,\n + \t\t\t commit_hex,\n +-\t\t\t opts.append,\n +-\t\t\t 1))\n ++\t\t\t flags))\n + \t\tresult = 1;\n \n \tUNLEAK(lines);\n - \treturn result;\n \n diff --git a/builtin/commit.c b/builtin/commit.c\n --- a/builtin/commit.c\n @@ -124,11 +124,17 @@\n */\n int generation_numbers_enabled(struct repository *r);\n \n --int write_commit_graph_reachable(const char *obj_dir, int append,\n --\t\t\t\t int report_progress);\n +#define COMMIT_GRAPH_APPEND (1 << 0)\n +#define COMMIT_GRAPH_PROGRESS (1 << 1)\n +\n + /*\n + * The write_commit_graph* methods return zero on success\n + * and a negative value on failure. Note that if the repository\n + * is not compatible with the commit-graph feature, then the\n + * methods will return 0 without writing a commit-graph.\n + */\n +-int write_commit_graph_reachable(const char *obj_dir, int append,\n +-\t\t\t\t int report_progress);\n +int write_commit_graph_reachable(const char *obj_dir, unsigned int flags);\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n 4: 130007d0e1 = 4: a5223a37a9 commit-graph: remove Future Work section\n 5: 0ca4e18e98 ! 5: b5b8a87676 commit-graph: create write_commit_graph_context\n @@ -3,7 +3,7 @@\n commit-graph: create write_commit_graph_context\n \n The write_commit_graph() method is too large and complex. To simplify\n - it, we should extract several small methods. However, we will risk\n + it, we should extract several helper functions. However, we will risk\n repeating a lot of declarations related to progress incidators and\n object id or commit lists.\n \n @@ -11,7 +11,7 @@\n core data structures used in this process. Replace the other local\n variables with the values inside the context object. Following this\n change, we will start to lift code segments wholesale out of the\n - write_commit_graph() method and into their own methods.\n + write_commit_graph() method and into helper functions.\n \n Signed-off-by: Derrick Stolee \n \n @@ -454,7 +454,7 @@\n \t\tfor (i = 0; i < pack_indexes->nr; i++) {\n \t\t\tstruct packed_git *p;\n @@\n - \t\t\t\tres = 1;\n + \t\t\t\tres = -1;\n \t\t\t\tgoto cleanup;\n \t\t\t}\n -\t\t\tfor_each_object_in_pack(p, add_packed_commits, &oids,\n @@ -610,7 +610,7 @@\n -\tif (commits.nr >= GRAPH_EDGE_LAST_MASK) {\n +\tif (ctx->commits.nr >= GRAPH_EDGE_LAST_MASK) {\n \t\terror(_(\"too many commits to write graph\"));\n - \t\tres = 1;\n + \t\tres = -1;\n \t\tgoto cleanup;\n \t}\n \n @@ -628,7 +628,7 @@\n \t\terror(_(\"unable to create leading directories of %s\"),\n -\t\t\tgraph_name);\n +\t\t\tctx->graph_name);\n - \t\tres = errno;\n + \t\tres = -1;\n \t\tgoto cleanup;\n \t}\n \n 6: 30c1b618b1 ! 6: 98e243be67 commit-graph: extract fill_oids_from_packs()\n @@ -3,7 +3,7 @@\n commit-graph: extract fill_oids_from_packs()\n \n The write_commit_graph() method is too complex, so we are\n - extracting methods one by one.\n + extracting helper functions one by one.\n \n This extracts fill_oids_from_packs() that reads the given\n pack-file list and fills the oid list in the context.\n @@ -43,11 +43,11 @@\n +\t\tp = add_packed_git(packname.buf, packname.len, 1);\n +\t\tif (!p) {\n +\t\t\terror(_(\"error adding pack %s\"), packname.buf);\n -+\t\t\treturn 1;\n ++\t\t\treturn -1;\n +\t\t}\n +\t\tif (open_pack_index(p)) {\n +\t\t\terror(_(\"error opening index for %s\"), packname.buf);\n -+\t\t\treturn 1;\n ++\t\t\treturn -1;\n +\t\t}\n +\t\tfor_each_object_in_pack(p, add_packed_commits, ctx,\n +\t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n @@ -89,12 +89,12 @@\n -\t\t\tp = add_packed_git(packname.buf, packname.len, 1);\n -\t\t\tif (!p) {\n -\t\t\t\terror(_(\"error adding pack %s\"), packname.buf);\n --\t\t\t\tres = 1;\n +-\t\t\t\tres = -1;\n -\t\t\t\tgoto cleanup;\n -\t\t\t}\n -\t\t\tif (open_pack_index(p)) {\n -\t\t\t\terror(_(\"error opening index for %s\"), packname.buf);\n --\t\t\t\tres = 1;\n +-\t\t\t\tres = -1;\n -\t\t\t\tgoto cleanup;\n -\t\t\t}\n -\t\t\tfor_each_object_in_pack(p, add_packed_commits, ctx,\n 7: 8cb2613dfa ! 7: fe36c8ad28 commit-graph: extract fill_oids_from_commit_hex()\n @@ -3,7 +3,7 @@\n commit-graph: extract fill_oids_from_commit_hex()\n \n The write_commit_graph() method is too complex, so we are\n - extracting methods one by one.\n + extracting helper functions one by one.\n \n Extract fill_oids_from_commit_hex() that reads the given commit\n id list and fille the oid list in the context.\n 8: 8f7129672a ! 8: b8dfb663f3 commit-graph: extract fill_oids_from_all_packs()\n @@ -3,7 +3,7 @@\n commit-graph: extract fill_oids_from_all_packs()\n \n The write_commit_graph() method is too complex, so we are\n - extracting methods one by one.\n + extracting helper functions one by one.\n \n Extract fill_oids_from_all_packs() that reads all pack-files\n for commits and fills the oid list in the context.\n 9: a37548745b ! 9: 40acc6ec37 commit-graph: extract count_distinct_commits()\n @@ -3,7 +3,7 @@\n commit-graph: extract count_distinct_commits()\n \n The write_commit_graph() method is too complex, so we are\n - extracting methods one by one.\n + extracting helper functions one by one.\n \n Extract count_distinct_commits(), which sorts the oids list, then\n iterates through to find duplicates.\n 10: 57366ffdaa ! 10: b403c01ef5 commit-graph: extract copy_oids_to_commits()\n @@ -3,7 +3,7 @@\n commit-graph: extract copy_oids_to_commits()\n \n The write_commit_graph() method is too complex, so we are\n - extracting methods one by one.\n + extracting helper functions one by one.\n \n Extract copy_oids_to_commits(), which fills the commits list\n with the distinct commits from the oids list. During this loop,\n 11: fc81c8946d ! 11: 7ecf923040 commit-graph: extract write_commit_graph_file()\n @@ -3,7 +3,7 @@\n commit-graph: extract write_commit_graph_file()\n \n The write_commit_graph() method is too complex, so we are\n - extracting methods one by one.\n + extracting helper functions one by one.\n \n Extract write_commit_graph_file() that takes all of the information\n in the context struct and writes the data to a commit-graph file.\n @@ -41,7 +41,7 @@\n +\t\tUNLEAK(ctx->graph_name);\n +\t\terror(_(\"unable to create leading directories of %s\"),\n +\t\t\tctx->graph_name);\n -+\t\treturn errno;\n ++\t\treturn -1;\n +\t}\n +\n +\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n @@ -124,7 +124,7 @@\n -\t\tUNLEAK(ctx->graph_name);\n -\t\terror(_(\"unable to create leading directories of %s\"),\n -\t\t\tctx->graph_name);\n --\t\tres = errno;\n +-\t\tres = -1;\n -\t\tgoto cleanup;\n -\t}\n -\n\n-- \ngitgitgadget\n"},{"id":"377064","messageId":"0be7713a255d6f52c51ccb19299d0c9797e9a03f.1560346173.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 01/11] commit-graph: fix the_repository reference","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-12T13:29:36Z","receivedAt":"2019-06-12T13:29:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe parse_commit_buffer() method takes a repository pointer, so it\nshould not refer to the_repository anymore.\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/commit.c b/commit.c\nindex a5333c7ac6..e4d1233226 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -443,7 +443,7 @@ int parse_commit_buffer(struct repository *r, struct commit *item, const void *b\n \titem->date = parse_commit_date(bufptr, tail);\n \n \tif (check_graph)\n-\t\tload_commit_graph_info(the_repository, item);\n+\t\tload_commit_graph_info(r, item);\n \n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"377065","messageId":"b4e3ae579a04670956f7a06c2aae762fa7fea4fb.1560346173.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 03/11] commit-graph: collapse parameters into flags","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-12T13:29:38Z","receivedAt":"2019-06-12T13:29:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() and write_commit_graph_reachable() methods\ncurrently take two boolean parameters: 'append' and 'report_progress'.\nAs we update these methods, adding more parameters this way becomes\ncluttered and hard to maintain.\n\nCollapse these parameters into a 'flags' parameter, and adjust the\ncallers to provide flags as necessary.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 8 +++++---\n builtin/commit.c | 2 +-\n builtin/gc.c | 4 ++--\n commit-graph.c | 9 +++++----\n commit-graph.h | 8 +++++---\n 5 files changed, 18 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 2a1c4d701f..d8efa5bab2 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -142,6 +142,7 @@ static int graph_write(int argc, const char **argv)\n \tstruct string_list *commit_hex = NULL;\n \tstruct string_list lines;\n \tint result = 0;\n+\tunsigned int flags = COMMIT_GRAPH_PROGRESS;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n@@ -166,11 +167,13 @@ static int graph_write(int argc, const char **argv)\n \t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n+\tif (opts.append)\n+\t\tflags |= COMMIT_GRAPH_APPEND;\n \n \tread_replace_refs = 0;\n \n \tif (opts.reachable)\n-\t\treturn write_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n+\t\treturn write_commit_graph_reachable(opts.obj_dir, flags);\n \n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n@@ -190,8 +193,7 @@ static int graph_write(int argc, const char **argv)\n \tif (write_commit_graph(opts.obj_dir,\n \t\t\t pack_indexes,\n \t\t\t commit_hex,\n-\t\t\t opts.append,\n-\t\t\t 1))\n+\t\t\t flags))\n \t\tresult = 1;\n \n \tUNLEAK(lines);\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex b9ea7222fa..b001ef565d 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1670,7 +1670,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\t \"not exceeded, and then \\\"git reset HEAD\\\" to recover.\"));\n \n \tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n-\t write_commit_graph_reachable(get_object_directory(), 0, 0))\n+\t write_commit_graph_reachable(get_object_directory(), 0))\n \t\treturn 1;\n \n \trepo_rerere(the_repository, 0);\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 3984addf73..df2573f124 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -665,8 +665,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (gc_write_commit_graph &&\n-\t write_commit_graph_reachable(get_object_directory(), 0,\n-\t\t\t\t\t !quiet && !daemonized))\n+\t write_commit_graph_reachable(get_object_directory(),\n+\t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_PROGRESS : 0))\n \t\treturn 1;\n \n \tif (auto_gc && too_many_loose_objects())\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 1b58d1da14..fc40b531af 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -851,15 +851,14 @@ static int add_ref_to_list(const char *refname,\n \treturn 0;\n }\n \n-int write_commit_graph_reachable(const char *obj_dir, int append,\n-\t\t\t\t int report_progress)\n+int write_commit_graph_reachable(const char *obj_dir, unsigned int flags)\n {\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n \tint result;\n \n \tfor_each_ref(add_ref_to_list, &list);\n \tresult = write_commit_graph(obj_dir, NULL, &list,\n-\t\t\t\t append, report_progress);\n+\t\t\t\t flags);\n \n \tstring_list_clear(&list, 0);\n \treturn result;\n@@ -868,7 +867,7 @@ int write_commit_graph_reachable(const char *obj_dir, int append,\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t int append, int report_progress)\n+\t\t unsigned int flags)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n@@ -887,6 +886,8 @@ int write_commit_graph(const char *obj_dir,\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tunsigned long approx_nr_objects;\n \tint res = 0;\n+\tint append = flags & COMMIT_GRAPH_APPEND;\n+\tint report_progress = flags & COMMIT_GRAPH_PROGRESS;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn 0;\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 869717ca19..01538b5cf5 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -65,18 +65,20 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n */\n int generation_numbers_enabled(struct repository *r);\n \n+#define COMMIT_GRAPH_APPEND (1 << 0)\n+#define COMMIT_GRAPH_PROGRESS (1 << 1)\n+\n /*\n * The write_commit_graph* methods return zero on success\n * and a negative value on failure. Note that if the repository\n * is not compatible with the commit-graph feature, then the\n * methods will return 0 without writing a commit-graph.\n */\n-int write_commit_graph_reachable(const char *obj_dir, int append,\n-\t\t\t\t int report_progress);\n+int write_commit_graph_reachable(const char *obj_dir, unsigned int flags);\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t int append, int report_progress);\n+\t\t unsigned int flags);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \n-- \ngitgitgadget\n\n"},{"id":"377066","messageId":"95f66e85b2fe93a218dad4c04c16718d053fb002.1560346173.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 02/11] commit-graph: return with errors during write","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-12T13:29:37Z","receivedAt":"2019-06-12T13:29:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method uses die() to report failure and\nexit when confronted with an unexpected condition. This use of\ndie() in a library function is incorrect and is now replaced by\nerror() statements and an int return type. Return zero on success\nand a negative value on failure.\n\nNow that we use 'goto cleanup' to jump to the terminal condition\non an error, we have new paths that could lead to uninitialized\nvalues. New initializers are added to correct for this.\n\nThe builtins 'commit-graph', 'gc', and 'commit' call these methods,\nso update them to check the return value. Test that 'git commit-graph\nwrite' returns a proper error code when hitting a failure condition\nin write_commit_graph().\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 20 +++++++-------\n builtin/commit.c | 5 ++--\n builtin/gc.c | 7 ++---\n commit-graph.c | 60 ++++++++++++++++++++++++++++-------------\n commit-graph.h | 16 +++++++----\n t/t5318-commit-graph.sh | 8 ++++++\n 6 files changed, 77 insertions(+), 39 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 537fdfd0f0..2a1c4d701f 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -141,6 +141,7 @@ static int graph_write(int argc, const char **argv)\n \tstruct string_list *pack_indexes = NULL;\n \tstruct string_list *commit_hex = NULL;\n \tstruct string_list lines;\n+\tint result = 0;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n@@ -168,10 +169,8 @@ static int graph_write(int argc, const char **argv)\n \n \tread_replace_refs = 0;\n \n-\tif (opts.reachable) {\n-\t\twrite_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n-\t\treturn 0;\n-\t}\n+\tif (opts.reachable)\n+\t\treturn write_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n \n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n@@ -188,14 +187,15 @@ static int graph_write(int argc, const char **argv)\n \t\tUNLEAK(buf);\n \t}\n \n-\twrite_commit_graph(opts.obj_dir,\n-\t\t\t pack_indexes,\n-\t\t\t commit_hex,\n-\t\t\t opts.append,\n-\t\t\t 1);\n+\tif (write_commit_graph(opts.obj_dir,\n+\t\t\t pack_indexes,\n+\t\t\t commit_hex,\n+\t\t\t opts.append,\n+\t\t\t 1))\n+\t\tresult = 1;\n \n \tUNLEAK(lines);\n-\treturn 0;\n+\treturn result;\n }\n \n int cmd_commit_graph(int argc, const char **argv, const char *prefix)\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 2986553d5f..b9ea7222fa 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1669,8 +1669,9 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\t \"new_index file. Check that disk is not full and quota is\\n\"\n \t\t \"not exceeded, and then \\\"git reset HEAD\\\" to recover.\"));\n \n-\tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0))\n-\t\twrite_commit_graph_reachable(get_object_directory(), 0, 0);\n+\tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n+\t write_commit_graph_reachable(get_object_directory(), 0, 0))\n+\t\treturn 1;\n \n \trepo_rerere(the_repository, 0);\n \trun_command_v_opt(argv_gc_auto, RUN_GIT_CMD);\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 020f725acc..3984addf73 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -664,9 +664,10 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tclean_pack_garbage();\n \t}\n \n-\tif (gc_write_commit_graph)\n-\t\twrite_commit_graph_reachable(get_object_directory(), 0,\n-\t\t\t\t\t !quiet && !daemonized);\n+\tif (gc_write_commit_graph &&\n+\t write_commit_graph_reachable(get_object_directory(), 0,\n+\t\t\t\t\t !quiet && !daemonized))\n+\t\treturn 1;\n \n \tif (auto_gc && too_many_loose_objects())\n \t\twarning(_(\"There are too many unreachable loose objects; \"\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 66865acbd7..1b58d1da14 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -851,27 +851,30 @@ static int add_ref_to_list(const char *refname,\n \treturn 0;\n }\n \n-void write_commit_graph_reachable(const char *obj_dir, int append,\n-\t\t\t\t int report_progress)\n+int write_commit_graph_reachable(const char *obj_dir, int append,\n+\t\t\t\t int report_progress)\n {\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n+\tint result;\n \n \tfor_each_ref(add_ref_to_list, &list);\n-\twrite_commit_graph(obj_dir, NULL, &list, append, report_progress);\n+\tresult = write_commit_graph(obj_dir, NULL, &list,\n+\t\t\t\t append, report_progress);\n \n \tstring_list_clear(&list, 0);\n+\treturn result;\n }\n \n-void write_commit_graph(const char *obj_dir,\n-\t\t\tstruct string_list *pack_indexes,\n-\t\t\tstruct string_list *commit_hex,\n-\t\t\tint append, int report_progress)\n+int write_commit_graph(const char *obj_dir,\n+\t\t struct string_list *pack_indexes,\n+\t\t struct string_list *commit_hex,\n+\t\t int append, int report_progress)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n \tstruct hashfile *f;\n \tuint32_t i, count_distinct = 0;\n-\tchar *graph_name;\n+\tchar *graph_name = NULL;\n \tstruct lock_file lk = LOCK_INIT;\n \tuint32_t chunk_ids[5];\n \tuint64_t chunk_offsets[5];\n@@ -883,15 +886,17 @@ void write_commit_graph(const char *obj_dir,\n \tuint64_t progress_cnt = 0;\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tunsigned long approx_nr_objects;\n+\tint res = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n-\t\treturn;\n+\t\treturn 0;\n \n \toids.nr = 0;\n \tapprox_nr_objects = approximate_object_count();\n \toids.alloc = approx_nr_objects / 32;\n \toids.progress = NULL;\n \toids.progress_done = 0;\n+\tcommits.list = NULL;\n \n \tif (append) {\n \t\tprepare_commit_graph_one(the_repository, obj_dir);\n@@ -932,10 +937,16 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tstrbuf_setlen(&packname, dirlen);\n \t\t\tstrbuf_addstr(&packname, pack_indexes->items[i].string);\n \t\t\tp = add_packed_git(packname.buf, packname.len, 1);\n-\t\t\tif (!p)\n-\t\t\t\tdie(_(\"error adding pack %s\"), packname.buf);\n-\t\t\tif (open_pack_index(p))\n-\t\t\t\tdie(_(\"error opening index for %s\"), packname.buf);\n+\t\t\tif (!p) {\n+\t\t\t\terror(_(\"error adding pack %s\"), packname.buf);\n+\t\t\t\tres = -1;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\t\t\tif (open_pack_index(p)) {\n+\t\t\t\terror(_(\"error opening index for %s\"), packname.buf);\n+\t\t\t\tres = -1;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \t\t\tfor_each_object_in_pack(p, add_packed_commits, &oids,\n \t\t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n \t\t\tclose_pack(p);\n@@ -1006,8 +1017,11 @@ void write_commit_graph(const char *obj_dir,\n \t}\n \tstop_progress(&progress);\n \n-\tif (count_distinct >= GRAPH_EDGE_LAST_MASK)\n-\t\tdie(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n+\tif (count_distinct >= GRAPH_EDGE_LAST_MASK) {\n+\t\terror(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n+\t\tres = -1;\n+\t\tgoto cleanup;\n+\t}\n \n \tcommits.nr = 0;\n \tcommits.alloc = count_distinct;\n@@ -1039,16 +1053,21 @@ void write_commit_graph(const char *obj_dir,\n \tnum_chunks = num_extra_edges ? 4 : 3;\n \tstop_progress(&progress);\n \n-\tif (commits.nr >= GRAPH_EDGE_LAST_MASK)\n-\t\tdie(_(\"too many commits to write graph\"));\n+\tif (commits.nr >= GRAPH_EDGE_LAST_MASK) {\n+\t\terror(_(\"too many commits to write graph\"));\n+\t\tres = -1;\n+\t\tgoto cleanup;\n+\t}\n \n \tcompute_generation_numbers(&commits, report_progress);\n \n \tgraph_name = get_commit_graph_filename(obj_dir);\n \tif (safe_create_leading_directories(graph_name)) {\n \t\tUNLEAK(graph_name);\n-\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n-\t\t\t graph_name);\n+\t\terror(_(\"unable to create leading directories of %s\"),\n+\t\t\tgraph_name);\n+\t\tres = -1;\n+\t\tgoto cleanup;\n \t}\n \n \thold_lock_file_for_update(&lk, graph_name, LOCK_DIE_ON_ERROR);\n@@ -1107,9 +1126,12 @@ void write_commit_graph(const char *obj_dir,\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n \tcommit_lock_file(&lk);\n \n+cleanup:\n \tfree(graph_name);\n \tfree(commits.list);\n \tfree(oids.list);\n+\n+\treturn res;\n }\n \n #define VERIFY_COMMIT_GRAPH_ERROR_HASH 2\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 7dfb8c896f..869717ca19 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -65,12 +65,18 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n */\n int generation_numbers_enabled(struct repository *r);\n \n-void write_commit_graph_reachable(const char *obj_dir, int append,\n+/*\n+ * The write_commit_graph* methods return zero on success\n+ * and a negative value on failure. Note that if the repository\n+ * is not compatible with the commit-graph feature, then the\n+ * methods will return 0 without writing a commit-graph.\n+ */\n+int write_commit_graph_reachable(const char *obj_dir, int append,\n \t\t\t\t int report_progress);\n-void write_commit_graph(const char *obj_dir,\n-\t\t\tstruct string_list *pack_indexes,\n-\t\t\tstruct string_list *commit_hex,\n-\t\t\tint append, int report_progress);\n+int write_commit_graph(const char *obj_dir,\n+\t\t struct string_list *pack_indexes,\n+\t\t struct string_list *commit_hex,\n+\t\t int append, int report_progress);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex e80c1cac02..3b6fd0d728 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -23,6 +23,14 @@ test_expect_success 'write graph with no packs' '\n \ttest_path_is_file info/commit-graph\n '\n \n+test_expect_success 'close with correct error on bad input' '\n+\tcd \"$TRASH_DIRECTORY/full\" &&\n+\techo doesnotexist >in &&\n+\t{ git commit-graph write --stdin-packs stderr; ret=$?; } &&\n+\ttest \"$ret\" = 1 &&\n+\ttest_i18ngrep \"error adding pack\" stderr\n+'\n+\n test_expect_success 'create commits and repack' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tfor i in $(test_seq 3)\n-- \ngitgitgadget\n\n"},{"id":"377067","messageId":"a5223a37a9b6e705d3240b90d6a9226375f9b1a1.1560346174.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 04/11] commit-graph: remove Future Work section","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-12T13:29:39Z","receivedAt":"2019-06-12T13:29:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe commit-graph feature began with a long list of planned\nbenefits, most of which are now complete. The future work\nsection has only a few items left.\n\nAs for making more algorithms aware of generation numbers,\nsome are only waiting for generation number v2 to ensure the\nperformance matches the existing behavior using commit date.\n\nIt is unlikely that we will ever send a commit-graph file\nas part of the protocol, since we would need to verify the\ndata, and that is expensive. If we want to start trusting\nremote content, then that item can be investigated again.\n\nWhile there is more work to be done on the feature, having\na section of the docs devoted to a TODO list is wasteful and\nhard to keep up-to-date.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 17 -----------------\n 1 file changed, 17 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex 7805b0968c..fb53341d5e 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -127,23 +127,6 @@ Design Details\n helpful for these clones, anyway. The commit-graph will not be read or\n written when shallow commits are present.\n \n-Future Work\n------------\n-\n-- After computing and storing generation numbers, we must make graph\n- walks aware of generation numbers to gain the performance benefits they\n- enable. This will mostly be accomplished by swapping a commit-date-ordered\n- priority queue with one ordered by generation number. The following\n- operations are important candidates:\n-\n- - 'log --topo-order'\n- - 'tag --merged'\n-\n-- A server could provide a commit-graph file as part of the network protocol\n- to avoid extra calculations by clients. This feature is only of benefit if\n- the user is willing to trust the file, because verifying the file is correct\n- is as hard as computing it from scratch.\n-\n Related Links\n -------------\n [0] https://bugs.chromium.org/p/git/issues/detail?id=8\n-- \ngitgitgadget\n\n"},{"id":"377068","messageId":"b5b8a87676c5081f0afff2aba543e26822b97503.1560346174.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 05/11] commit-graph: create write_commit_graph_context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-12T13:29:40Z","receivedAt":"2019-06-12T13:29:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too large and complex. To simplify\nit, we should extract several helper functions. However, we will risk\nrepeating a lot of declarations related to progress incidators and\nobject id or commit lists.\n\nCreate a new write_commit_graph_context struct that contains the\ncore data structures used in this process. Replace the other local\nvariables with the values inside the context object. Following this\nchange, we will start to lift code segments wholesale out of the\nwrite_commit_graph() method and into helper functions.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 390 ++++++++++++++++++++++++-------------------------\n 1 file changed, 194 insertions(+), 196 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex fc40b531af..6d7e83cfe8 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -518,14 +518,38 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \treturn get_commit_tree_in_graph_one(r, r->objects->commit_graph, c);\n }\n \n+struct packed_commit_list {\n+\tstruct commit **list;\n+\tint nr;\n+\tint alloc;\n+};\n+\n+struct packed_oid_list {\n+\tstruct object_id *list;\n+\tint nr;\n+\tint alloc;\n+};\n+\n+struct write_commit_graph_context {\n+\tstruct repository *r;\n+\tconst char *obj_dir;\n+\tchar *graph_name;\n+\tstruct packed_oid_list oids;\n+\tstruct packed_commit_list commits;\n+\tint num_extra_edges;\n+\tunsigned long approx_nr_objects;\n+\tstruct progress *progress;\n+\tint progress_done;\n+\tuint64_t progress_cnt;\n+\tunsigned append:1,\n+\t\t report_progress:1;\n+};\n+\n static void write_graph_chunk_fanout(struct hashfile *f,\n-\t\t\t\t struct commit **commits,\n-\t\t\t\t int nr_commits,\n-\t\t\t\t struct progress *progress,\n-\t\t\t\t uint64_t *progress_cnt)\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tint i, count = 0;\n-\tstruct commit **list = commits;\n+\tstruct commit **list = ctx->commits.list;\n \n \t/*\n \t * Write the first-level table (the list is sorted,\n@@ -533,10 +557,10 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n \t * having to do eight extra binary search iterations).\n \t */\n \tfor (i = 0; i < 256; i++) {\n-\t\twhile (count < nr_commits) {\n+\t\twhile (count < ctx->commits.nr) {\n \t\t\tif ((*list)->object.oid.hash[0] != i)\n \t\t\t\tbreak;\n-\t\t\tdisplay_progress(progress, ++*progress_cnt);\n+\t\t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n \t\t\tcount++;\n \t\t\tlist++;\n \t\t}\n@@ -546,14 +570,12 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n }\n \n static void write_graph_chunk_oids(struct hashfile *f, int hash_len,\n-\t\t\t\t struct commit **commits, int nr_commits,\n-\t\t\t\t struct progress *progress,\n-\t\t\t\t uint64_t *progress_cnt)\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n-\tstruct commit **list = commits;\n+\tstruct commit **list = ctx->commits.list;\n \tint count;\n-\tfor (count = 0; count < nr_commits; count++, list++) {\n-\t\tdisplay_progress(progress, ++*progress_cnt);\n+\tfor (count = 0; count < ctx->commits.nr; count++, list++) {\n+\t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n \t\thashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n \t}\n }\n@@ -565,19 +587,17 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n }\n \n static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n-\t\t\t\t struct commit **commits, int nr_commits,\n-\t\t\t\t struct progress *progress,\n-\t\t\t\t uint64_t *progress_cnt)\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n-\tstruct commit **list = commits;\n-\tstruct commit **last = commits + nr_commits;\n+\tstruct commit **list = ctx->commits.list;\n+\tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tuint32_t num_extra_edges = 0;\n \n \twhile (list < last) {\n \t\tstruct commit_list *parent;\n \t\tint edge_value;\n \t\tuint32_t packedDate[2];\n-\t\tdisplay_progress(progress, ++*progress_cnt);\n+\t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n \n \t\tparse_commit_no_graph(*list);\n \t\thashwrite(f, get_commit_tree_oid(*list)->hash, hash_len);\n@@ -588,8 +608,8 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\tedge_value = GRAPH_PARENT_NONE;\n \t\telse {\n \t\t\tedge_value = sha1_pos(parent->item->object.oid.hash,\n-\t\t\t\t\t commits,\n-\t\t\t\t\t nr_commits,\n+\t\t\t\t\t ctx->commits.list,\n+\t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t commit_to_sha1);\n \n \t\t\tif (edge_value < 0)\n@@ -609,8 +629,8 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\tedge_value = GRAPH_EXTRA_EDGES_NEEDED | num_extra_edges;\n \t\telse {\n \t\t\tedge_value = sha1_pos(parent->item->object.oid.hash,\n-\t\t\t\t\t commits,\n-\t\t\t\t\t nr_commits,\n+\t\t\t\t\t ctx->commits.list,\n+\t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t commit_to_sha1);\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n@@ -642,19 +662,16 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n }\n \n static void write_graph_chunk_extra_edges(struct hashfile *f,\n-\t\t\t\t\t struct commit **commits,\n-\t\t\t\t\t int nr_commits,\n-\t\t\t\t\t struct progress *progress,\n-\t\t\t\t\t uint64_t *progress_cnt)\n+\t\t\t\t\t struct write_commit_graph_context *ctx)\n {\n-\tstruct commit **list = commits;\n-\tstruct commit **last = commits + nr_commits;\n+\tstruct commit **list = ctx->commits.list;\n+\tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tstruct commit_list *parent;\n \n \twhile (list < last) {\n \t\tint num_parents = 0;\n \n-\t\tdisplay_progress(progress, ++*progress_cnt);\n+\t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n \n \t\tfor (parent = (*list)->parents; num_parents < 3 && parent;\n \t\t parent = parent->next)\n@@ -668,8 +685,8 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n \t\t/* Since num_parents > 2, this initializer is safe. */\n \t\tfor (parent = (*list)->parents->next; parent; parent = parent->next) {\n \t\t\tint edge_value = sha1_pos(parent->item->object.oid.hash,\n-\t\t\t\t\t\t commits,\n-\t\t\t\t\t\t nr_commits,\n+\t\t\t\t\t\t ctx->commits.list,\n+\t\t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t\t commit_to_sha1);\n \n \t\t\tif (edge_value < 0)\n@@ -693,125 +710,111 @@ static int commit_compare(const void *_a, const void *_b)\n \treturn oidcmp(a, b);\n }\n \n-struct packed_commit_list {\n-\tstruct commit **list;\n-\tint nr;\n-\tint alloc;\n-};\n-\n-struct packed_oid_list {\n-\tstruct object_id *list;\n-\tint nr;\n-\tint alloc;\n-\tstruct progress *progress;\n-\tint progress_done;\n-};\n-\n static int add_packed_commits(const struct object_id *oid,\n \t\t\t struct packed_git *pack,\n \t\t\t uint32_t pos,\n \t\t\t void *data)\n {\n-\tstruct packed_oid_list *list = (struct packed_oid_list*)data;\n+\tstruct write_commit_graph_context *ctx = (struct write_commit_graph_context*)data;\n \tenum object_type type;\n \toff_t offset = nth_packed_object_offset(pack, pos);\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \n-\tif (list->progress)\n-\t\tdisplay_progress(list->progress, ++list->progress_done);\n+\tif (ctx->progress)\n+\t\tdisplay_progress(ctx->progress, ++ctx->progress_done);\n \n \toi.typep = &type;\n-\tif (packed_object_info(the_repository, pack, offset, &oi) < 0)\n+\tif (packed_object_info(ctx->r, pack, offset, &oi) < 0)\n \t\tdie(_(\"unable to get type of object %s\"), oid_to_hex(oid));\n \n \tif (type != OBJ_COMMIT)\n \t\treturn 0;\n \n-\tALLOC_GROW(list->list, list->nr + 1, list->alloc);\n-\toidcpy(&(list->list[list->nr]), oid);\n-\tlist->nr++;\n+\tALLOC_GROW(ctx->oids.list, ctx->oids.nr + 1, ctx->oids.alloc);\n+\toidcpy(&(ctx->oids.list[ctx->oids.nr]), oid);\n+\tctx->oids.nr++;\n \n \treturn 0;\n }\n \n-static void add_missing_parents(struct packed_oid_list *oids, struct commit *commit)\n+static void add_missing_parents(struct write_commit_graph_context *ctx, struct commit *commit)\n {\n \tstruct commit_list *parent;\n \tfor (parent = commit->parents; parent; parent = parent->next) {\n \t\tif (!(parent->item->object.flags & UNINTERESTING)) {\n-\t\t\tALLOC_GROW(oids->list, oids->nr + 1, oids->alloc);\n-\t\t\toidcpy(&oids->list[oids->nr], &(parent->item->object.oid));\n-\t\t\toids->nr++;\n+\t\t\tALLOC_GROW(ctx->oids.list, ctx->oids.nr + 1, ctx->oids.alloc);\n+\t\t\toidcpy(&ctx->oids.list[ctx->oids.nr], &(parent->item->object.oid));\n+\t\t\tctx->oids.nr++;\n \t\t\tparent->item->object.flags |= UNINTERESTING;\n \t\t}\n \t}\n }\n \n-static void close_reachable(struct packed_oid_list *oids, int report_progress)\n+static void close_reachable(struct write_commit_graph_context *ctx)\n {\n \tint i;\n \tstruct commit *commit;\n-\tstruct progress *progress = NULL;\n \n-\tif (report_progress)\n-\t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Loading known commits in commit graph\"), oids->nr);\n-\tfor (i = 0; i < oids->nr; i++) {\n-\t\tdisplay_progress(progress, i + 1);\n-\t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"Loading known commits in commit graph\"),\n+\t\t\t\t\tctx->oids.nr);\n+\tfor (i = 0; i < ctx->oids.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tcommit = lookup_commit(ctx->r, &ctx->oids.list[i]);\n \t\tif (commit)\n \t\t\tcommit->object.flags |= UNINTERESTING;\n \t}\n-\tstop_progress(&progress);\n+\tstop_progress(&ctx->progress);\n \n \t/*\n-\t * As this loop runs, oids->nr may grow, but not more\n+\t * As this loop runs, ctx->oids.nr may grow, but not more\n \t * than the number of missing commits in the reachable\n \t * closure.\n \t */\n-\tif (report_progress)\n-\t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Expanding reachable commits in commit graph\"), oids->nr);\n-\tfor (i = 0; i < oids->nr; i++) {\n-\t\tdisplay_progress(progress, i + 1);\n-\t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"Expanding reachable commits in commit graph\"),\n+\t\t\t\t\tctx->oids.nr);\n+\tfor (i = 0; i < ctx->oids.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tcommit = lookup_commit(ctx->r, &ctx->oids.list[i]);\n \n \t\tif (commit && !parse_commit_no_graph(commit))\n-\t\t\tadd_missing_parents(oids, commit);\n+\t\t\tadd_missing_parents(ctx, commit);\n \t}\n-\tstop_progress(&progress);\n+\tstop_progress(&ctx->progress);\n \n-\tif (report_progress)\n-\t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Clearing commit marks in commit graph\"), oids->nr);\n-\tfor (i = 0; i < oids->nr; i++) {\n-\t\tdisplay_progress(progress, i + 1);\n-\t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"Clearing commit marks in commit graph\"),\n+\t\t\t\t\tctx->oids.nr);\n+\tfor (i = 0; i < ctx->oids.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tcommit = lookup_commit(ctx->r, &ctx->oids.list[i]);\n \n \t\tif (commit)\n \t\t\tcommit->object.flags &= ~UNINTERESTING;\n \t}\n-\tstop_progress(&progress);\n+\tstop_progress(&ctx->progress);\n }\n \n-static void compute_generation_numbers(struct packed_commit_list* commits,\n-\t\t\t\t int report_progress)\n+static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n {\n \tint i;\n \tstruct commit_list *list = NULL;\n-\tstruct progress *progress = NULL;\n \n-\tif (report_progress)\n-\t\tprogress = start_progress(\n-\t\t\t_(\"Computing commit graph generation numbers\"),\n-\t\t\tcommits->nr);\n-\tfor (i = 0; i < commits->nr; i++) {\n-\t\tdisplay_progress(progress, i + 1);\n-\t\tif (commits->list[i]->generation != GENERATION_NUMBER_INFINITY &&\n-\t\t commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_progress(\n+\t\t\t\t\t_(\"Computing commit graph generation numbers\"),\n+\t\t\t\t\tctx->commits.nr);\n+\tfor (i = 0; i < ctx->commits.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tif (ctx->commits.list[i]->generation != GENERATION_NUMBER_INFINITY &&\n+\t\t ctx->commits.list[i]->generation != GENERATION_NUMBER_ZERO)\n \t\t\tcontinue;\n \n-\t\tcommit_list_insert(commits->list[i], &list);\n+\t\tcommit_list_insert(ctx->commits.list[i], &list);\n \t\twhile (list) {\n \t\t\tstruct commit *current = list->item;\n \t\t\tstruct commit_list *parent;\n@@ -838,7 +841,7 @@ static void compute_generation_numbers(struct packed_commit_list* commits,\n \t\t\t}\n \t\t}\n \t}\n-\tstop_progress(&progress);\n+\tstop_progress(&ctx->progress);\n }\n \n static int add_ref_to_list(const char *refname,\n@@ -869,8 +872,7 @@ int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *commit_hex,\n \t\t unsigned int flags)\n {\n-\tstruct packed_oid_list oids;\n-\tstruct packed_commit_list commits;\n+\tstruct write_commit_graph_context *ctx;\n \tstruct hashfile *f;\n \tuint32_t i, count_distinct = 0;\n \tchar *graph_name = NULL;\n@@ -878,44 +880,38 @@ int write_commit_graph(const char *obj_dir,\n \tuint32_t chunk_ids[5];\n \tuint64_t chunk_offsets[5];\n \tint num_chunks;\n-\tint num_extra_edges;\n \tstruct commit_list *parent;\n-\tstruct progress *progress = NULL;\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n-\tuint64_t progress_cnt = 0;\n \tstruct strbuf progress_title = STRBUF_INIT;\n-\tunsigned long approx_nr_objects;\n \tint res = 0;\n-\tint append = flags & COMMIT_GRAPH_APPEND;\n-\tint report_progress = flags & COMMIT_GRAPH_PROGRESS;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn 0;\n \n-\toids.nr = 0;\n-\tapprox_nr_objects = approximate_object_count();\n-\toids.alloc = approx_nr_objects / 32;\n-\toids.progress = NULL;\n-\toids.progress_done = 0;\n-\tcommits.list = NULL;\n-\n-\tif (append) {\n-\t\tprepare_commit_graph_one(the_repository, obj_dir);\n-\t\tif (the_repository->objects->commit_graph)\n-\t\t\toids.alloc += the_repository->objects->commit_graph->num_commits;\n+\tctx = xcalloc(1, sizeof(struct write_commit_graph_context));\n+\tctx->r = the_repository;\n+\tctx->obj_dir = obj_dir;\n+\tctx->append = flags & COMMIT_GRAPH_APPEND ? 1 : 0;\n+\tctx->report_progress = flags & COMMIT_GRAPH_PROGRESS ? 1 : 0;\n+\n+\tctx->approx_nr_objects = approximate_object_count();\n+\tctx->oids.alloc = ctx->approx_nr_objects / 32;\n+\n+\tif (ctx->append) {\n+\t\tprepare_commit_graph_one(ctx->r, ctx->obj_dir);\n+\t\tif (ctx->r->objects->commit_graph)\n+\t\t\tctx->oids.alloc += ctx->r->objects->commit_graph->num_commits;\n \t}\n \n-\tif (oids.alloc < 1024)\n-\t\toids.alloc = 1024;\n-\tALLOC_ARRAY(oids.list, oids.alloc);\n-\n-\tif (append && the_repository->objects->commit_graph) {\n-\t\tstruct commit_graph *commit_graph =\n-\t\t\tthe_repository->objects->commit_graph;\n-\t\tfor (i = 0; i < commit_graph->num_commits; i++) {\n-\t\t\tconst unsigned char *hash = commit_graph->chunk_oid_lookup +\n-\t\t\t\tcommit_graph->hash_len * i;\n-\t\t\thashcpy(oids.list[oids.nr++].hash, hash);\n+\tif (ctx->oids.alloc < 1024)\n+\t\tctx->oids.alloc = 1024;\n+\tALLOC_ARRAY(ctx->oids.list, ctx->oids.alloc);\n+\n+\tif (ctx->append && ctx->r->objects->commit_graph) {\n+\t\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\t\tfor (i = 0; i < g->num_commits; i++) {\n+\t\t\tconst unsigned char *hash = g->chunk_oid_lookup + g->hash_len * i;\n+\t\t\thashcpy(ctx->oids.list[ctx->oids.nr++].hash, hash);\n \t\t}\n \t}\n \n@@ -924,14 +920,14 @@ int write_commit_graph(const char *obj_dir,\n \t\tint dirlen;\n \t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n \t\tdirlen = packname.len;\n-\t\tif (report_progress) {\n+\t\tif (ctx->report_progress) {\n \t\t\tstrbuf_addf(&progress_title,\n \t\t\t\t Q_(\"Finding commits for commit graph in %d pack\",\n \t\t\t\t \"Finding commits for commit graph in %d packs\",\n \t\t\t\t pack_indexes->nr),\n \t\t\t\t pack_indexes->nr);\n-\t\t\toids.progress = start_delayed_progress(progress_title.buf, 0);\n-\t\t\toids.progress_done = 0;\n+\t\t\tctx->progress = start_delayed_progress(progress_title.buf, 0);\n+\t\t\tctx->progress_done = 0;\n \t\t}\n \t\tfor (i = 0; i < pack_indexes->nr; i++) {\n \t\t\tstruct packed_git *p;\n@@ -948,75 +944,76 @@ int write_commit_graph(const char *obj_dir,\n \t\t\t\tres = -1;\n \t\t\t\tgoto cleanup;\n \t\t\t}\n-\t\t\tfor_each_object_in_pack(p, add_packed_commits, &oids,\n+\t\t\tfor_each_object_in_pack(p, add_packed_commits, ctx,\n \t\t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n \t\t\tclose_pack(p);\n \t\t\tfree(p);\n \t\t}\n-\t\tstop_progress(&oids.progress);\n+\t\tstop_progress(&ctx->progress);\n \t\tstrbuf_reset(&progress_title);\n \t\tstrbuf_release(&packname);\n \t}\n \n \tif (commit_hex) {\n-\t\tif (report_progress) {\n+\t\tif (ctx->report_progress) {\n \t\t\tstrbuf_addf(&progress_title,\n \t\t\t\t Q_(\"Finding commits for commit graph from %d ref\",\n \t\t\t\t \"Finding commits for commit graph from %d refs\",\n \t\t\t\t commit_hex->nr),\n \t\t\t\t commit_hex->nr);\n-\t\t\tprogress = start_delayed_progress(progress_title.buf,\n-\t\t\t\t\t\t\t commit_hex->nr);\n+\t\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t\tprogress_title.buf,\n+\t\t\t\t\t\tcommit_hex->nr);\n \t\t}\n \t\tfor (i = 0; i < commit_hex->nr; i++) {\n \t\t\tconst char *end;\n \t\t\tstruct object_id oid;\n \t\t\tstruct commit *result;\n \n-\t\t\tdisplay_progress(progress, i + 1);\n+\t\t\tdisplay_progress(ctx->progress, i + 1);\n \t\t\tif (commit_hex->items[i].string &&\n \t\t\t parse_oid_hex(commit_hex->items[i].string, &oid, &end))\n \t\t\t\tcontinue;\n \n-\t\t\tresult = lookup_commit_reference_gently(the_repository, &oid, 1);\n+\t\t\tresult = lookup_commit_reference_gently(ctx->r, &oid, 1);\n \n \t\t\tif (result) {\n-\t\t\t\tALLOC_GROW(oids.list, oids.nr + 1, oids.alloc);\n-\t\t\t\toidcpy(&oids.list[oids.nr], &(result->object.oid));\n-\t\t\t\toids.nr++;\n+\t\t\t\tALLOC_GROW(ctx->oids.list, ctx->oids.nr + 1, ctx->oids.alloc);\n+\t\t\t\toidcpy(&ctx->oids.list[ctx->oids.nr], &(result->object.oid));\n+\t\t\t\tctx->oids.nr++;\n \t\t\t}\n \t\t}\n-\t\tstop_progress(&progress);\n+\t\tstop_progress(&ctx->progress);\n \t\tstrbuf_reset(&progress_title);\n \t}\n \n \tif (!pack_indexes && !commit_hex) {\n-\t\tif (report_progress)\n-\t\t\toids.progress = start_delayed_progress(\n+\t\tif (ctx->report_progress)\n+\t\t\tctx->progress = start_delayed_progress(\n \t\t\t\t_(\"Finding commits for commit graph among packed objects\"),\n-\t\t\t\tapprox_nr_objects);\n-\t\tfor_each_packed_object(add_packed_commits, &oids,\n+\t\t\t\tctx->approx_nr_objects);\n+\t\tfor_each_packed_object(add_packed_commits, ctx,\n \t\t\t\t FOR_EACH_OBJECT_PACK_ORDER);\n-\t\tif (oids.progress_done < approx_nr_objects)\n-\t\t\tdisplay_progress(oids.progress, approx_nr_objects);\n-\t\tstop_progress(&oids.progress);\n+\t\tif (ctx->progress_done < ctx->approx_nr_objects)\n+\t\t\tdisplay_progress(ctx->progress, ctx->approx_nr_objects);\n+\t\tstop_progress(&ctx->progress);\n \t}\n \n-\tclose_reachable(&oids, report_progress);\n+\tclose_reachable(ctx);\n \n-\tif (report_progress)\n-\t\tprogress = start_delayed_progress(\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n \t\t\t_(\"Counting distinct commits in commit graph\"),\n-\t\t\toids.nr);\n-\tdisplay_progress(progress, 0); /* TODO: Measure QSORT() progress */\n-\tQSORT(oids.list, oids.nr, commit_compare);\n+\t\t\tctx->oids.nr);\n+\tdisplay_progress(ctx->progress, 0); /* TODO: Measure QSORT() progress */\n+\tQSORT(ctx->oids.list, ctx->oids.nr, commit_compare);\n \tcount_distinct = 1;\n-\tfor (i = 1; i < oids.nr; i++) {\n-\t\tdisplay_progress(progress, i + 1);\n-\t\tif (!oideq(&oids.list[i - 1], &oids.list[i]))\n+\tfor (i = 1; i < ctx->oids.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n \t\t\tcount_distinct++;\n \t}\n-\tstop_progress(&progress);\n+\tstop_progress(&ctx->progress);\n \n \tif (count_distinct >= GRAPH_EDGE_LAST_MASK) {\n \t\terror(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n@@ -1024,54 +1021,54 @@ int write_commit_graph(const char *obj_dir,\n \t\tgoto cleanup;\n \t}\n \n-\tcommits.nr = 0;\n-\tcommits.alloc = count_distinct;\n-\tALLOC_ARRAY(commits.list, commits.alloc);\n+\tctx->commits.alloc = count_distinct;\n+\tALLOC_ARRAY(ctx->commits.list, ctx->commits.alloc);\n \n-\tnum_extra_edges = 0;\n-\tif (report_progress)\n-\t\tprogress = start_delayed_progress(\n+\tctx->num_extra_edges = 0;\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n \t\t\t_(\"Finding extra edges in commit graph\"),\n-\t\t\toids.nr);\n-\tfor (i = 0; i < oids.nr; i++) {\n+\t\t\tctx->oids.nr);\n+\tfor (i = 0; i < ctx->oids.nr; i++) {\n \t\tint num_parents = 0;\n-\t\tdisplay_progress(progress, i + 1);\n-\t\tif (i > 0 && oideq(&oids.list[i - 1], &oids.list[i]))\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tif (i > 0 && oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n \t\t\tcontinue;\n \n-\t\tcommits.list[commits.nr] = lookup_commit(the_repository, &oids.list[i]);\n-\t\tparse_commit_no_graph(commits.list[commits.nr]);\n+\t\tctx->commits.list[ctx->commits.nr] = lookup_commit(ctx->r, &ctx->oids.list[i]);\n+\t\tparse_commit_no_graph(ctx->commits.list[ctx->commits.nr]);\n \n-\t\tfor (parent = commits.list[commits.nr]->parents;\n+\t\tfor (parent = ctx->commits.list[ctx->commits.nr]->parents;\n \t\t parent; parent = parent->next)\n \t\t\tnum_parents++;\n \n \t\tif (num_parents > 2)\n-\t\t\tnum_extra_edges += num_parents - 1;\n+\t\t\tctx->num_extra_edges += num_parents - 1;\n \n-\t\tcommits.nr++;\n+\t\tctx->commits.nr++;\n \t}\n-\tnum_chunks = num_extra_edges ? 4 : 3;\n-\tstop_progress(&progress);\n+\tstop_progress(&ctx->progress);\n \n-\tif (commits.nr >= GRAPH_EDGE_LAST_MASK) {\n+\tif (ctx->commits.nr >= GRAPH_EDGE_LAST_MASK) {\n \t\terror(_(\"too many commits to write graph\"));\n \t\tres = -1;\n \t\tgoto cleanup;\n \t}\n \n-\tcompute_generation_numbers(&commits, report_progress);\n+\tcompute_generation_numbers(ctx);\n \n-\tgraph_name = get_commit_graph_filename(obj_dir);\n-\tif (safe_create_leading_directories(graph_name)) {\n-\t\tUNLEAK(graph_name);\n+\tnum_chunks = ctx->num_extra_edges ? 4 : 3;\n+\n+\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n+\tif (safe_create_leading_directories(ctx->graph_name)) {\n+\t\tUNLEAK(ctx->graph_name);\n \t\terror(_(\"unable to create leading directories of %s\"),\n-\t\t\tgraph_name);\n+\t\t\tctx->graph_name);\n \t\tres = -1;\n \t\tgoto cleanup;\n \t}\n \n-\thold_lock_file_for_update(&lk, graph_name, LOCK_DIE_ON_ERROR);\n+\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \n \thashwrite_be32(f, GRAPH_SIGNATURE);\n@@ -1084,7 +1081,7 @@ int write_commit_graph(const char *obj_dir,\n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n \tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n-\tif (num_extra_edges)\n+\tif (ctx->num_extra_edges)\n \t\tchunk_ids[3] = GRAPH_CHUNKID_EXTRAEDGES;\n \telse\n \t\tchunk_ids[3] = 0;\n@@ -1092,9 +1089,9 @@ int write_commit_graph(const char *obj_dir,\n \n \tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n-\tchunk_offsets[2] = chunk_offsets[1] + hashsz * commits.nr;\n-\tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * commits.nr;\n-\tchunk_offsets[4] = chunk_offsets[3] + 4 * num_extra_edges;\n+\tchunk_offsets[2] = chunk_offsets[1] + hashsz * ctx->commits.nr;\n+\tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * ctx->commits.nr;\n+\tchunk_offsets[4] = chunk_offsets[3] + 4 * ctx->num_extra_edges;\n \n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tuint32_t chunk_write[3];\n@@ -1105,32 +1102,33 @@ int write_commit_graph(const char *obj_dir,\n \t\thashwrite(f, chunk_write, 12);\n \t}\n \n-\tif (report_progress) {\n+\tif (ctx->report_progress) {\n \t\tstrbuf_addf(&progress_title,\n \t\t\t Q_(\"Writing out commit graph in %d pass\",\n \t\t\t \"Writing out commit graph in %d passes\",\n \t\t\t num_chunks),\n \t\t\t num_chunks);\n-\t\tprogress = start_delayed_progress(\n+\t\tctx->progress = start_delayed_progress(\n \t\t\tprogress_title.buf,\n-\t\t\tnum_chunks * commits.nr);\n+\t\t\tnum_chunks * ctx->commits.nr);\n \t}\n-\twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress, &progress_cnt);\n-\twrite_graph_chunk_oids(f, hashsz, commits.list, commits.nr, progress, &progress_cnt);\n-\twrite_graph_chunk_data(f, hashsz, commits.list, commits.nr, progress, &progress_cnt);\n-\tif (num_extra_edges)\n-\t\twrite_graph_chunk_extra_edges(f, commits.list, commits.nr, progress, &progress_cnt);\n-\tstop_progress(&progress);\n+\twrite_graph_chunk_fanout(f, ctx);\n+\twrite_graph_chunk_oids(f, hashsz, ctx);\n+\twrite_graph_chunk_data(f, hashsz, ctx);\n+\tif (ctx->num_extra_edges)\n+\t\twrite_graph_chunk_extra_edges(f, ctx);\n+\tstop_progress(&ctx->progress);\n \tstrbuf_release(&progress_title);\n \n-\tclose_commit_graph(the_repository);\n+\tclose_commit_graph(ctx->r);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n \tcommit_lock_file(&lk);\n \n cleanup:\n \tfree(graph_name);\n-\tfree(commits.list);\n-\tfree(oids.list);\n+\tfree(ctx->commits.list);\n+\tfree(ctx->oids.list);\n+\tfree(ctx);\n \n \treturn res;\n }\n-- \ngitgitgadget\n\n"},{"id":"377069","messageId":"b8dfb663f3675098bd1e8bef56478ea315dc9fb3.1560346174.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 08/11] commit-graph: extract fill_oids_from_all_packs()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-12T13:29:42Z","receivedAt":"2019-06-12T13:29:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too complex, so we are\nextracting helper functions one by one.\n\nExtract fill_oids_from_all_packs() that reads all pack-files\nfor commits and fills the oid list in the context.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 26 +++++++++++++++-----------\n 1 file changed, 15 insertions(+), 11 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 4fae1fcdb2..61cb43ddf8 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -950,6 +950,19 @@ static void fill_oids_from_commit_hex(struct write_commit_graph_context *ctx,\n \tstrbuf_release(&progress_title);\n }\n \n+static void fill_oids_from_all_packs(struct write_commit_graph_context *ctx)\n+{\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t_(\"Finding commits for commit graph among packed objects\"),\n+\t\t\tctx->approx_nr_objects);\n+\tfor_each_packed_object(add_packed_commits, ctx,\n+\t\t\t FOR_EACH_OBJECT_PACK_ORDER);\n+\tif (ctx->progress_done < ctx->approx_nr_objects)\n+\t\tdisplay_progress(ctx->progress, ctx->approx_nr_objects);\n+\tstop_progress(&ctx->progress);\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -1006,17 +1019,8 @@ int write_commit_graph(const char *obj_dir,\n \tif (commit_hex)\n \t\tfill_oids_from_commit_hex(ctx, commit_hex);\n \n-\tif (!pack_indexes && !commit_hex) {\n-\t\tif (ctx->report_progress)\n-\t\t\tctx->progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph among packed objects\"),\n-\t\t\t\tctx->approx_nr_objects);\n-\t\tfor_each_packed_object(add_packed_commits, ctx,\n-\t\t\t\t FOR_EACH_OBJECT_PACK_ORDER);\n-\t\tif (ctx->progress_done < ctx->approx_nr_objects)\n-\t\t\tdisplay_progress(ctx->progress, ctx->approx_nr_objects);\n-\t\tstop_progress(&ctx->progress);\n-\t}\n+\tif (!pack_indexes && !commit_hex)\n+\t\tfill_oids_from_all_packs(ctx);\n \n \tclose_reachable(ctx);\n \n-- \ngitgitgadget\n\n"},{"id":"377070","messageId":"98e243be673f9a29fb855d02845c7642e6375c8f.1560346174.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 06/11] commit-graph: extract fill_oids_from_packs()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-12T13:29:41Z","receivedAt":"2019-06-12T13:29:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too complex, so we are\nextracting helper functions one by one.\n\nThis extracts fill_oids_from_packs() that reads the given\npack-file list and fills the oid list in the context.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 83 ++++++++++++++++++++++++++++----------------------\n 1 file changed, 47 insertions(+), 36 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 6d7e83cfe8..02e5f8c651 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -867,6 +867,51 @@ int write_commit_graph_reachable(const char *obj_dir, unsigned int flags)\n \treturn result;\n }\n \n+static int fill_oids_from_packs(struct write_commit_graph_context *ctx,\n+\t\t\t\tstruct string_list *pack_indexes)\n+{\n+\tuint32_t i;\n+\tstruct strbuf progress_title = STRBUF_INIT;\n+\tstruct strbuf packname = STRBUF_INIT;\n+\tint dirlen;\n+\n+\tstrbuf_addf(&packname, \"%s/pack/\", ctx->obj_dir);\n+\tdirlen = packname.len;\n+\tif (ctx->report_progress) {\n+\t\tstrbuf_addf(&progress_title,\n+\t\t\t Q_(\"Finding commits for commit graph in %d pack\",\n+\t\t\t \"Finding commits for commit graph in %d packs\",\n+\t\t\t pack_indexes->nr),\n+\t\t\t pack_indexes->nr);\n+\t\tctx->progress = start_delayed_progress(progress_title.buf, 0);\n+\t\tctx->progress_done = 0;\n+\t}\n+\tfor (i = 0; i < pack_indexes->nr; i++) {\n+\t\tstruct packed_git *p;\n+\t\tstrbuf_setlen(&packname, dirlen);\n+\t\tstrbuf_addstr(&packname, pack_indexes->items[i].string);\n+\t\tp = add_packed_git(packname.buf, packname.len, 1);\n+\t\tif (!p) {\n+\t\t\terror(_(\"error adding pack %s\"), packname.buf);\n+\t\t\treturn -1;\n+\t\t}\n+\t\tif (open_pack_index(p)) {\n+\t\t\terror(_(\"error opening index for %s\"), packname.buf);\n+\t\t\treturn -1;\n+\t\t}\n+\t\tfor_each_object_in_pack(p, add_packed_commits, ctx,\n+\t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n+\t\tclose_pack(p);\n+\t\tfree(p);\n+\t}\n+\n+\tstop_progress(&ctx->progress);\n+\tstrbuf_reset(&progress_title);\n+\tstrbuf_release(&packname);\n+\n+\treturn 0;\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -916,42 +961,8 @@ int write_commit_graph(const char *obj_dir,\n \t}\n \n \tif (pack_indexes) {\n-\t\tstruct strbuf packname = STRBUF_INIT;\n-\t\tint dirlen;\n-\t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n-\t\tdirlen = packname.len;\n-\t\tif (ctx->report_progress) {\n-\t\t\tstrbuf_addf(&progress_title,\n-\t\t\t\t Q_(\"Finding commits for commit graph in %d pack\",\n-\t\t\t\t \"Finding commits for commit graph in %d packs\",\n-\t\t\t\t pack_indexes->nr),\n-\t\t\t\t pack_indexes->nr);\n-\t\t\tctx->progress = start_delayed_progress(progress_title.buf, 0);\n-\t\t\tctx->progress_done = 0;\n-\t\t}\n-\t\tfor (i = 0; i < pack_indexes->nr; i++) {\n-\t\t\tstruct packed_git *p;\n-\t\t\tstrbuf_setlen(&packname, dirlen);\n-\t\t\tstrbuf_addstr(&packname, pack_indexes->items[i].string);\n-\t\t\tp = add_packed_git(packname.buf, packname.len, 1);\n-\t\t\tif (!p) {\n-\t\t\t\terror(_(\"error adding pack %s\"), packname.buf);\n-\t\t\t\tres = -1;\n-\t\t\t\tgoto cleanup;\n-\t\t\t}\n-\t\t\tif (open_pack_index(p)) {\n-\t\t\t\terror(_(\"error opening index for %s\"), packname.buf);\n-\t\t\t\tres = -1;\n-\t\t\t\tgoto cleanup;\n-\t\t\t}\n-\t\t\tfor_each_object_in_pack(p, add_packed_commits, ctx,\n-\t\t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n-\t\t\tclose_pack(p);\n-\t\t\tfree(p);\n-\t\t}\n-\t\tstop_progress(&ctx->progress);\n-\t\tstrbuf_reset(&progress_title);\n-\t\tstrbuf_release(&packname);\n+\t\tif ((res = fill_oids_from_packs(ctx, pack_indexes)))\n+\t\t\tgoto cleanup;\n \t}\n \n \tif (commit_hex) {\n-- \ngitgitgadget\n\n"},{"id":"377071","messageId":"b403c01ef543360df3496f810a7e3988205bb18c.1560346174.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 10/11] commit-graph: extract copy_oids_to_commits()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-12T13:29:44Z","receivedAt":"2019-06-12T13:29:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too complex, so we are\nextracting helper functions one by one.\n\nExtract copy_oids_to_commits(), which fills the commits list\nwith the distinct commits from the oids list. During this loop,\nit also counts the number of \"extra\" edges from octopus merges.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 57 ++++++++++++++++++++++++++++----------------------\n 1 file changed, 32 insertions(+), 25 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 1a0a875a7b..72f9c5c7e2 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -984,6 +984,37 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n \treturn count_distinct;\n }\n \n+static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n+{\n+\tuint32_t i;\n+\tstruct commit_list *parent;\n+\n+\tctx->num_extra_edges = 0;\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t_(\"Finding extra edges in commit graph\"),\n+\t\t\tctx->oids.nr);\n+\tfor (i = 0; i < ctx->oids.nr; i++) {\n+\t\tint num_parents = 0;\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tif (i > 0 && oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n+\t\t\tcontinue;\n+\n+\t\tctx->commits.list[ctx->commits.nr] = lookup_commit(ctx->r, &ctx->oids.list[i]);\n+\t\tparse_commit_no_graph(ctx->commits.list[ctx->commits.nr]);\n+\n+\t\tfor (parent = ctx->commits.list[ctx->commits.nr]->parents;\n+\t\t parent; parent = parent->next)\n+\t\t\tnum_parents++;\n+\n+\t\tif (num_parents > 2)\n+\t\t\tctx->num_extra_edges += num_parents - 1;\n+\n+\t\tctx->commits.nr++;\n+\t}\n+\tstop_progress(&ctx->progress);\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -997,7 +1028,6 @@ int write_commit_graph(const char *obj_dir,\n \tuint32_t chunk_ids[5];\n \tuint64_t chunk_offsets[5];\n \tint num_chunks;\n-\tstruct commit_list *parent;\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tint res = 0;\n@@ -1056,30 +1086,7 @@ int write_commit_graph(const char *obj_dir,\n \tctx->commits.alloc = count_distinct;\n \tALLOC_ARRAY(ctx->commits.list, ctx->commits.alloc);\n \n-\tctx->num_extra_edges = 0;\n-\tif (ctx->report_progress)\n-\t\tctx->progress = start_delayed_progress(\n-\t\t\t_(\"Finding extra edges in commit graph\"),\n-\t\t\tctx->oids.nr);\n-\tfor (i = 0; i < ctx->oids.nr; i++) {\n-\t\tint num_parents = 0;\n-\t\tdisplay_progress(ctx->progress, i + 1);\n-\t\tif (i > 0 && oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n-\t\t\tcontinue;\n-\n-\t\tctx->commits.list[ctx->commits.nr] = lookup_commit(ctx->r, &ctx->oids.list[i]);\n-\t\tparse_commit_no_graph(ctx->commits.list[ctx->commits.nr]);\n-\n-\t\tfor (parent = ctx->commits.list[ctx->commits.nr]->parents;\n-\t\t parent; parent = parent->next)\n-\t\t\tnum_parents++;\n-\n-\t\tif (num_parents > 2)\n-\t\t\tctx->num_extra_edges += num_parents - 1;\n-\n-\t\tctx->commits.nr++;\n-\t}\n-\tstop_progress(&ctx->progress);\n+\tcopy_oids_to_commits(ctx);\n \n \tif (ctx->commits.nr >= GRAPH_EDGE_LAST_MASK) {\n \t\terror(_(\"too many commits to write graph\"));\n-- \ngitgitgadget\n\n"},{"id":"377072","messageId":"fe36c8ad28f4e608739f10d2bbef7fadf236a3db.1560346174.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 07/11] commit-graph: extract fill_oids_from_commit_hex()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-12T13:29:42Z","receivedAt":"2019-06-12T13:29:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too complex, so we are\nextracting helper functions one by one.\n\nExtract fill_oids_from_commit_hex() that reads the given commit\nid list and fille the oid list in the context.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 72 ++++++++++++++++++++++++++++----------------------\n 1 file changed, 40 insertions(+), 32 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 02e5f8c651..4fae1fcdb2 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -912,6 +912,44 @@ static int fill_oids_from_packs(struct write_commit_graph_context *ctx,\n \treturn 0;\n }\n \n+static void fill_oids_from_commit_hex(struct write_commit_graph_context *ctx,\n+\t\t\t\t struct string_list *commit_hex)\n+{\n+\tuint32_t i;\n+\tstruct strbuf progress_title = STRBUF_INIT;\n+\n+\tif (ctx->report_progress) {\n+\t\tstrbuf_addf(&progress_title,\n+\t\t\t Q_(\"Finding commits for commit graph from %d ref\",\n+\t\t\t \"Finding commits for commit graph from %d refs\",\n+\t\t\t commit_hex->nr),\n+\t\t\t commit_hex->nr);\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\tprogress_title.buf,\n+\t\t\t\t\tcommit_hex->nr);\n+\t}\n+\tfor (i = 0; i < commit_hex->nr; i++) {\n+\t\tconst char *end;\n+\t\tstruct object_id oid;\n+\t\tstruct commit *result;\n+\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tif (commit_hex->items[i].string &&\n+\t\t parse_oid_hex(commit_hex->items[i].string, &oid, &end))\n+\t\t\tcontinue;\n+\n+\t\tresult = lookup_commit_reference_gently(ctx->r, &oid, 1);\n+\n+\t\tif (result) {\n+\t\t\tALLOC_GROW(ctx->oids.list, ctx->oids.nr + 1, ctx->oids.alloc);\n+\t\t\toidcpy(&ctx->oids.list[ctx->oids.nr], &(result->object.oid));\n+\t\t\tctx->oids.nr++;\n+\t\t}\n+\t}\n+\tstop_progress(&ctx->progress);\n+\tstrbuf_release(&progress_title);\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -965,38 +1003,8 @@ int write_commit_graph(const char *obj_dir,\n \t\t\tgoto cleanup;\n \t}\n \n-\tif (commit_hex) {\n-\t\tif (ctx->report_progress) {\n-\t\t\tstrbuf_addf(&progress_title,\n-\t\t\t\t Q_(\"Finding commits for commit graph from %d ref\",\n-\t\t\t\t \"Finding commits for commit graph from %d refs\",\n-\t\t\t\t commit_hex->nr),\n-\t\t\t\t commit_hex->nr);\n-\t\t\tctx->progress = start_delayed_progress(\n-\t\t\t\t\t\tprogress_title.buf,\n-\t\t\t\t\t\tcommit_hex->nr);\n-\t\t}\n-\t\tfor (i = 0; i < commit_hex->nr; i++) {\n-\t\t\tconst char *end;\n-\t\t\tstruct object_id oid;\n-\t\t\tstruct commit *result;\n-\n-\t\t\tdisplay_progress(ctx->progress, i + 1);\n-\t\t\tif (commit_hex->items[i].string &&\n-\t\t\t parse_oid_hex(commit_hex->items[i].string, &oid, &end))\n-\t\t\t\tcontinue;\n-\n-\t\t\tresult = lookup_commit_reference_gently(ctx->r, &oid, 1);\n-\n-\t\t\tif (result) {\n-\t\t\t\tALLOC_GROW(ctx->oids.list, ctx->oids.nr + 1, ctx->oids.alloc);\n-\t\t\t\toidcpy(&ctx->oids.list[ctx->oids.nr], &(result->object.oid));\n-\t\t\t\tctx->oids.nr++;\n-\t\t\t}\n-\t\t}\n-\t\tstop_progress(&ctx->progress);\n-\t\tstrbuf_reset(&progress_title);\n-\t}\n+\tif (commit_hex)\n+\t\tfill_oids_from_commit_hex(ctx, commit_hex);\n \n \tif (!pack_indexes && !commit_hex) {\n \t\tif (ctx->report_progress)\n-- \ngitgitgadget\n\n"},{"id":"377073","messageId":"7ecf923040e30d120039d1d47eeae5e1d7ed271b.1560346174.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 11/11] commit-graph: extract write_commit_graph_file()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-12T13:29:45Z","receivedAt":"2019-06-12T13:29:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too complex, so we are\nextracting helper functions one by one.\n\nExtract write_commit_graph_file() that takes all of the information\nin the context struct and writes the data to a commit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 155 +++++++++++++++++++++++++------------------------\n 1 file changed, 80 insertions(+), 75 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 72f9c5c7e2..9d2c72f5b4 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1015,21 +1015,91 @@ static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n-int write_commit_graph(const char *obj_dir,\n-\t\t struct string_list *pack_indexes,\n-\t\t struct string_list *commit_hex,\n-\t\t unsigned int flags)\n+static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n {\n-\tstruct write_commit_graph_context *ctx;\n+\tuint32_t i;\n \tstruct hashfile *f;\n-\tuint32_t i, count_distinct = 0;\n-\tchar *graph_name = NULL;\n \tstruct lock_file lk = LOCK_INIT;\n \tuint32_t chunk_ids[5];\n \tuint64_t chunk_offsets[5];\n-\tint num_chunks;\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n+\tint num_chunks = ctx->num_extra_edges ? 4 : 3;\n+\n+\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n+\tif (safe_create_leading_directories(ctx->graph_name)) {\n+\t\tUNLEAK(ctx->graph_name);\n+\t\terror(_(\"unable to create leading directories of %s\"),\n+\t\t\tctx->graph_name);\n+\t\treturn -1;\n+\t}\n+\n+\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n+\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n+\n+\thashwrite_be32(f, GRAPH_SIGNATURE);\n+\n+\thashwrite_u8(f, GRAPH_VERSION);\n+\thashwrite_u8(f, oid_version());\n+\thashwrite_u8(f, num_chunks);\n+\thashwrite_u8(f, 0); /* unused padding byte */\n+\n+\tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n+\tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n+\tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n+\tif (ctx->num_extra_edges)\n+\t\tchunk_ids[3] = GRAPH_CHUNKID_EXTRAEDGES;\n+\telse\n+\t\tchunk_ids[3] = 0;\n+\tchunk_ids[4] = 0;\n+\n+\tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n+\tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n+\tchunk_offsets[2] = chunk_offsets[1] + hashsz * ctx->commits.nr;\n+\tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * ctx->commits.nr;\n+\tchunk_offsets[4] = chunk_offsets[3] + 4 * ctx->num_extra_edges;\n+\n+\tfor (i = 0; i <= num_chunks; i++) {\n+\t\tuint32_t chunk_write[3];\n+\n+\t\tchunk_write[0] = htonl(chunk_ids[i]);\n+\t\tchunk_write[1] = htonl(chunk_offsets[i] >> 32);\n+\t\tchunk_write[2] = htonl(chunk_offsets[i] & 0xffffffff);\n+\t\thashwrite(f, chunk_write, 12);\n+\t}\n+\n+\tif (ctx->report_progress) {\n+\t\tstrbuf_addf(&progress_title,\n+\t\t\t Q_(\"Writing out commit graph in %d pass\",\n+\t\t\t \"Writing out commit graph in %d passes\",\n+\t\t\t num_chunks),\n+\t\t\t num_chunks);\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\tprogress_title.buf,\n+\t\t\tnum_chunks * ctx->commits.nr);\n+\t}\n+\twrite_graph_chunk_fanout(f, ctx);\n+\twrite_graph_chunk_oids(f, hashsz, ctx);\n+\twrite_graph_chunk_data(f, hashsz, ctx);\n+\tif (ctx->num_extra_edges)\n+\t\twrite_graph_chunk_extra_edges(f, ctx);\n+\tstop_progress(&ctx->progress);\n+\tstrbuf_release(&progress_title);\n+\n+\tclose_commit_graph(ctx->r);\n+\tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n+\tcommit_lock_file(&lk);\n+\n+\treturn 0;\n+}\n+\n+int write_commit_graph(const char *obj_dir,\n+\t\t struct string_list *pack_indexes,\n+\t\t struct string_list *commit_hex,\n+\t\t unsigned int flags)\n+{\n+\tstruct write_commit_graph_context *ctx;\n+\tuint32_t i, count_distinct = 0;\n \tint res = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n@@ -1096,75 +1166,10 @@ int write_commit_graph(const char *obj_dir,\n \n \tcompute_generation_numbers(ctx);\n \n-\tnum_chunks = ctx->num_extra_edges ? 4 : 3;\n-\n-\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n-\tif (safe_create_leading_directories(ctx->graph_name)) {\n-\t\tUNLEAK(ctx->graph_name);\n-\t\terror(_(\"unable to create leading directories of %s\"),\n-\t\t\tctx->graph_name);\n-\t\tres = -1;\n-\t\tgoto cleanup;\n-\t}\n-\n-\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n-\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n-\n-\thashwrite_be32(f, GRAPH_SIGNATURE);\n-\n-\thashwrite_u8(f, GRAPH_VERSION);\n-\thashwrite_u8(f, oid_version());\n-\thashwrite_u8(f, num_chunks);\n-\thashwrite_u8(f, 0); /* unused padding byte */\n-\n-\tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n-\tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n-\tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n-\tif (ctx->num_extra_edges)\n-\t\tchunk_ids[3] = GRAPH_CHUNKID_EXTRAEDGES;\n-\telse\n-\t\tchunk_ids[3] = 0;\n-\tchunk_ids[4] = 0;\n-\n-\tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n-\tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n-\tchunk_offsets[2] = chunk_offsets[1] + hashsz * ctx->commits.nr;\n-\tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * ctx->commits.nr;\n-\tchunk_offsets[4] = chunk_offsets[3] + 4 * ctx->num_extra_edges;\n-\n-\tfor (i = 0; i <= num_chunks; i++) {\n-\t\tuint32_t chunk_write[3];\n-\n-\t\tchunk_write[0] = htonl(chunk_ids[i]);\n-\t\tchunk_write[1] = htonl(chunk_offsets[i] >> 32);\n-\t\tchunk_write[2] = htonl(chunk_offsets[i] & 0xffffffff);\n-\t\thashwrite(f, chunk_write, 12);\n-\t}\n-\n-\tif (ctx->report_progress) {\n-\t\tstrbuf_addf(&progress_title,\n-\t\t\t Q_(\"Writing out commit graph in %d pass\",\n-\t\t\t \"Writing out commit graph in %d passes\",\n-\t\t\t num_chunks),\n-\t\t\t num_chunks);\n-\t\tctx->progress = start_delayed_progress(\n-\t\t\tprogress_title.buf,\n-\t\t\tnum_chunks * ctx->commits.nr);\n-\t}\n-\twrite_graph_chunk_fanout(f, ctx);\n-\twrite_graph_chunk_oids(f, hashsz, ctx);\n-\twrite_graph_chunk_data(f, hashsz, ctx);\n-\tif (ctx->num_extra_edges)\n-\t\twrite_graph_chunk_extra_edges(f, ctx);\n-\tstop_progress(&ctx->progress);\n-\tstrbuf_release(&progress_title);\n-\n-\tclose_commit_graph(ctx->r);\n-\tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n-\tcommit_lock_file(&lk);\n+\tres = write_commit_graph_file(ctx);\n \n cleanup:\n-\tfree(graph_name);\n+\tfree(ctx->graph_name);\n \tfree(ctx->commits.list);\n \tfree(ctx->oids.list);\n \tfree(ctx);\n-- \ngitgitgadget\n"},{"id":"377074","messageId":"40acc6ec3716f2e8ce5e08b18b394bc07ed56f36.1560346174.git.gitgitgadget@gmail.com","threadId":"50302","inReplyTo":"pull.112.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 09/11] commit-graph: extract count_distinct_commits()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-12T13:29:43Z","receivedAt":"2019-06-12T13:29:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too complex, so we are\nextracting helper functions one by one.\n\nExtract count_distinct_commits(), which sorts the oids list, then\niterates through to find duplicates.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 35 ++++++++++++++++++++++-------------\n 1 file changed, 22 insertions(+), 13 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 61cb43ddf8..1a0a875a7b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -963,6 +963,27 @@ static void fill_oids_from_all_packs(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n+static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n+{\n+\tuint32_t i, count_distinct = 1;\n+\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t_(\"Counting distinct commits in commit graph\"),\n+\t\t\tctx->oids.nr);\n+\tdisplay_progress(ctx->progress, 0); /* TODO: Measure QSORT() progress */\n+\tQSORT(ctx->oids.list, ctx->oids.nr, commit_compare);\n+\n+\tfor (i = 1; i < ctx->oids.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n+\t\t\tcount_distinct++;\n+\t}\n+\tstop_progress(&ctx->progress);\n+\n+\treturn count_distinct;\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -1024,19 +1045,7 @@ int write_commit_graph(const char *obj_dir,\n \n \tclose_reachable(ctx);\n \n-\tif (ctx->report_progress)\n-\t\tctx->progress = start_delayed_progress(\n-\t\t\t_(\"Counting distinct commits in commit graph\"),\n-\t\t\tctx->oids.nr);\n-\tdisplay_progress(ctx->progress, 0); /* TODO: Measure QSORT() progress */\n-\tQSORT(ctx->oids.list, ctx->oids.nr, commit_compare);\n-\tcount_distinct = 1;\n-\tfor (i = 1; i < ctx->oids.nr; i++) {\n-\t\tdisplay_progress(ctx->progress, i + 1);\n-\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n-\t\t\tcount_distinct++;\n-\t}\n-\tstop_progress(&ctx->progress);\n+\tcount_distinct = count_distinct_commits(ctx);\n \n \tif (count_distinct >= GRAPH_EDGE_LAST_MASK) {\n \t\terror(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n-- \ngitgitgadget\n\n"},{"id":"378330","messageId":"20190629172335.GH21574@szeder.dev","threadId":"50302","inReplyTo":"95f66e85b2fe93a218dad4c04c16718d053fb002.1560346173.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 02/11] commit-graph: return with errors during write","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-06-29T17:23:35Z","receivedAt":"2019-06-29T17:23:41Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Jun 12, 2019 at 06:29:37AM -0700, Derrick Stolee via GitGitGadget wrote:\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index e80c1cac02..3b6fd0d728 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -23,6 +23,14 @@ test_expect_success 'write graph with no packs' '\n> \ttest_path_is_file info/commit-graph\n> '\n> \n> +test_expect_success 'close with correct error on bad input' '\n> +\tcd \"$TRASH_DIRECTORY/full\" &&\n> +\techo doesnotexist >in &&\n> +\t{ git commit-graph write --stdin-packs stderr; ret=$?; } &&\n> +\ttest \"$ret\" = 1 &&\n\nThis could be: \n\n test_expect_code 1 git commit-graph write --stdin-packs stderr\n\n\n> +\ttest_i18ngrep \"error adding pack\" stderr\n> +'\n"},{"id":"378369","messageId":"0e7f4844-69a1-b910-979b-64bd99c240ed@gmail.com","threadId":"50302","inReplyTo":"20190629172335.GH21574@szeder.dev","subject":"Re: [PATCH v5 02/11] commit-graph: return with errors during write","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-07-01T12:19:00Z","receivedAt":"2019-07-01T12:19:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/29/2019 1:23 PM, SZEDER Gábor wrote:\n> On Wed, Jun 12, 2019 at 06:29:37AM -0700, Derrick Stolee via GitGitGadget wrote:\n>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n>> index e80c1cac02..3b6fd0d728 100755\n>> --- a/t/t5318-commit-graph.sh\n>> +++ b/t/t5318-commit-graph.sh\n>> @@ -23,6 +23,14 @@ test_expect_success 'write graph with no packs' '\n>> \ttest_path_is_file info/commit-graph\n>> '\n>> \n>> +test_expect_success 'close with correct error on bad input' '\n>> +\tcd \"$TRASH_DIRECTORY/full\" &&\n>> +\techo doesnotexist >in &&\n>> +\t{ git commit-graph write --stdin-packs stderr; ret=$?; } &&\n>> +\ttest \"$ret\" = 1 &&\n> \n> This could be: \n> \n> test_expect_code 1 git commit-graph write --stdin-packs stderr\n> \n> \n>> +\ttest_i18ngrep \"error adding pack\" stderr\n>> +'\n\nThanks!, you are right! test_expect_code is what I should have used here\ninstead of finding the \"ret=$?\" trick in t0005-signals.sh, which needs to\ndo more interesting logic on the return code.\n\nHere is your suggestion as a diff. Junio: could you squash this in, or\nshould I submit a full patch?\n\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 22cb9d66430..4391007f4c1 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -26,8 +26,7 @@ test_expect_success 'write graph with no packs' '\n test_expect_success 'close with correct error on bad input' '\n cd \"$TRASH_DIRECTORY/full\" &&\n echo doesnotexist >in &&\n- { git commit-graph write --stdin-packs stderr; ret=$?; } &&\n- test \"$ret\" = 1 &&\n+ test_expect_code 1 git commit-graph write --stdin-packs stderr &&\n test_i18ngrep \"error adding pack\" stderr\n '\n\nI took inventory of when we are using \"=$?\" in the test scripts and saw\nthis was the only one that could easily be removed. Every other place is\ndoing something that can't be replaced by test_expect_code.\n\nThanks,\n-Stolee\n"}]}