{"thread":{"id":"51052","subject":"[PATCH 01/17] commit-graph: fix the_repository reference","startedAt":"2019-05-08T15:53:51Z","lastAt":"2019-06-18T18:14:48Z","messageCount":136,"participants":["Derrick Stolee via GitGitGadget","SZEDER Gábor","Derrick Stolee","Ævar Arnfjörð Bjarmason","Junio C Hamano","Philip Oakley","Ramsay Jones"],"isPatch":true,"patchVersion":1,"patchTotal":17},"messages":[{"id":"375148","messageId":"0be7713a255d6f52c51ccb19299d0c9797e9a03f.1557330826.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.git.gitgitgadget@gmail.com","subject":"[PATCH 01/17] commit-graph: fix the_repository reference","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-08T15:53:48Z","receivedAt":"2019-05-08T15:53:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe parse_commit_buffer() method takes a repository pointer, so it\nshould not refer to the_repository anymore.\n\nSigned-off-by: Derrick Stolee \n---\n commit.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/commit.c b/commit.c\nindex a5333c7ac6..e4d1233226 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -443,7 +443,7 @@ int parse_commit_buffer(struct repository *r, struct commit *item, const void *b\n \titem->date = parse_commit_date(bufptr, tail);\n \n \tif (check_graph)\n-\t\tload_commit_graph_info(the_repository, item);\n+\t\tload_commit_graph_info(r, item);\n \n \treturn 0;\n }\n-- \ngitgitgadget\n\n"},{"id":"375149","messageId":"pull.184.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":null,"subject":"[PATCH 00/17] [RFC] Commit-graph: Write incremental files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-08T15:53:47Z","receivedAt":"2019-05-08T15:53:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This patch series is marked as RFC quality because it is missing some key\nfeatures and tests, but hopefully starts a concrete discussion of how the\nincremental commit-graph writes can work. If this is a good direction, then\nit would replace ds/commit-graph-format-v2.\n\nThe commit-graph is a valuable performance feature for repos with large\ncommit histories, but suffers from the same problem as git repack: it\nrewrites the entire file every time. This can be slow when there are\nmillions of commits, especially after we stopped reading from the\ncommit-graph file during a write in 43d3561 (commit-graph write: don't die\nif the existing graph is corrupt).\n\nInstead, create a \"stack\" of commit-graphs where the existing commit-graph \nfile is \"level 0\" and other levels are in files \n$OBJDIR/info/commit-graphs/commit-graph-N for positive N. Each level is\nclosed under reachability with its lower levels, and the idea of \"graph\nposition\" now considers the concatenation of the commit orders from each\nlevel. See PATCH 12 for more details.\n\nWhen writing, we don't always want to add a new level to the stack. This\nwould eventually result in performance degradation, especially when\nsearching for a commit (before we know its graph position). We decide to\nmerge levels of the stack when the new commits we will write satisfy two\nconditions:\n\n 1. The expected size of the new file is more than half the size of the tip\n of the stack.\n 2. The new file contains more than 64,000 commits.\n\nThe first condition alone would prevent more than a logarithmic number of\nlevels. The second condition is a stop-gap to prevent performance issues\nwhen another process starts reading the commit-graph stack as we are merging\na large stack of commit-graph files. The reading process could be in a state\nwhere the new file is not ready, but the levels above the new file were\nalready deleted. Thus, the commits that were merged down must be parsed from\npack-files.\n\nThe performance is necessarily amortized across multiple writes, so I tested\nby writing commit-graphs from the (non-rc) tags in the Linux repo. My test\nincluded 72 tags, and wrote everything reachable from the tag using \n--stdin-commits. Here are the overall perf numbers:\n\ngit commit-graph write --stdin-commits: 8m 12s\ngit commit-graph write --stdin-commits --split: 45s\n\nThe test using --split included at least six full collapses to the full\ncommit-graph. I believe the commit-graph stack had at most three levels\nduring this test.\n\nThis series is long because I felt the need to refactor write_commit_graph()\nbefore making such a sweeping change to the format.\n\n * Patches 1-4: these are small changes which either fix issues or just\n provide clean-up. These are mostly borrowed from\n ds/commit-graph-format-v2.\n \n \n * Patches 5-11: these provide a non-functional refactor of\n write_commit_graph() into several methods using a \"struct\n write_commit_graph_context\" to share across the methods.\n \n \n * Patches 12-16: Implement the split commit-graph feature.\n \n \n * Patch 17: Demonstrate the value by writing a split commit-graph during \n git fetch when the new config setting fetch.writeCommitGraph is true.\n \n \n\nTODO: There are several things missing that need to be added before this\nseries is ready for full review and merging:\n\n 1. The documentation for git commit-graph needs updating for the --split \n option.\n \n \n 2. We likely want config settings for the merge strategy. This is mentioned\n in the design doc, and could be saved for later.\n \n \n 3. We want to update the git commit-graph verify subcommand to understand\n the commit-graph stack and optionally only verify the tip of the stack.\n This allows faster (amortized) verification if we are verifying\n immediately after writes and trusting the files at rest.\n \n \n 4. It would be helpful to add a new optional chunk that contains the\n trailing hash for the lower level of the commit-graph stack. This chunk\n would only be for the commit-graph-N files, and would provide a simple\n way to check that the stack is valid on read, in case we are still\n worried about other processes reading/writing in the wrong order.\n \n \n 5. Currently, --split essentially implies --append since we either (a)\n don't change the existing stack and only add commits, or (b) add all\n existing commits while merging files. However, if you would use --append \n with --split, the append logic will trigger a merge with the current tip\n (at minimum). Some care should be taken to make this more clear.\n \n \n\nThanks, -Stolee\n\n[1] \nhttps://github.com/git/git/commit/43d356180556180b4ef6ac232a14498a5bb2b446\ncommit-graph write: don't die if the existing graph is corrupt\n\nDerrick Stolee (17):\n commit-graph: fix the_repository reference\n commit-graph: return with errors during write\n commit-graph: collapse parameters into flags\n commit-graph: remove Future Work section\n commit-graph: create write_commit_graph_context\n commit-graph: extract fill_oids_from_packs()\n commit-graph: extract fill_oids_from_commit_hex()\n commit-graph: extract fill_oids_from_all_packs()\n commit-graph: extract count_distinct_commits()\n commit-graph: extract copy_oids_to_commits()\n commit-graph: extract write_commit_graph_file()\n Documentation: describe split commit-graphs\n commit-graph: lay groundwork for incremental files\n commit-graph: load split commit-graph files\n commit-graph: write split commit-graph files\n commit-graph: add --split option\n fetch: add fetch.writeCommitGraph config setting\n\n Documentation/technical/commit-graph.txt | 157 +++-\n builtin/commit-graph.c | 31 +-\n builtin/commit.c | 5 +-\n builtin/fetch.c | 17 +\n builtin/gc.c | 7 +-\n commit-graph.c | 946 ++++++++++++++++-------\n commit-graph.h | 19 +-\n commit.c | 2 +-\n t/t5318-commit-graph.sh | 28 +-\n 9 files changed, 887 insertions(+), 325 deletions(-)\n\n\nbase-commit: 93b4405ffe4ad9308740e7c1c71383bfc369baaa\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-184%2Fderrickstolee%2Fgraph%2Fincremental-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-184/derrickstolee/graph/incremental-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/184\n-- \ngitgitgadget\n"},{"id":"375151","messageId":"86b18f15ba1e497c6bcfc8cb60dc0ef03a58422c.1557330826.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.git.gitgitgadget@gmail.com","subject":"[PATCH 02/17] commit-graph: return with errors during write","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-08T15:53:49Z","receivedAt":"2019-05-08T15:53:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method uses die() to report failure and\nexit when confronted with an unexpected condition. This use of\ndie() in a library function is incorrect and is now replaced by\nerror() statements and an int return type.\n\nNow that we use 'goto cleanup' to jump to the terminal condition\non an error, we have new paths that could lead to uninitialized\nvalues. New initializers are added to correct for this.\n\nThe builtins 'commit-graph', 'gc', and 'commit' call these methods,\nso update them to check the return value.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 19 +++++++------\n builtin/commit.c | 5 ++--\n builtin/gc.c | 7 ++---\n commit-graph.c | 60 +++++++++++++++++++++++++++++-------------\n commit-graph.h | 10 +++----\n 5 files changed, 62 insertions(+), 39 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 537fdfd0f0..2e86251f02 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -141,6 +141,7 @@ static int graph_write(int argc, const char **argv)\n \tstruct string_list *pack_indexes = NULL;\n \tstruct string_list *commit_hex = NULL;\n \tstruct string_list lines;\n+\tint result;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n@@ -168,10 +169,8 @@ static int graph_write(int argc, const char **argv)\n \n \tread_replace_refs = 0;\n \n-\tif (opts.reachable) {\n-\t\twrite_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n-\t\treturn 0;\n-\t}\n+\tif (opts.reachable)\n+\t\treturn write_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n \n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n@@ -188,14 +187,14 @@ static int graph_write(int argc, const char **argv)\n \t\tUNLEAK(buf);\n \t}\n \n-\twrite_commit_graph(opts.obj_dir,\n-\t\t\t pack_indexes,\n-\t\t\t commit_hex,\n-\t\t\t opts.append,\n-\t\t\t 1);\n+\tresult = write_commit_graph(opts.obj_dir,\n+\t\t\t\t pack_indexes,\n+\t\t\t\t commit_hex,\n+\t\t\t\t opts.append,\n+\t\t\t\t 1);\n \n \tUNLEAK(lines);\n-\treturn 0;\n+\treturn result;\n }\n \n int cmd_commit_graph(int argc, const char **argv, const char *prefix)\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex 2986553d5f..b9ea7222fa 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1669,8 +1669,9 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\t \"new_index file. Check that disk is not full and quota is\\n\"\n \t\t \"not exceeded, and then \\\"git reset HEAD\\\" to recover.\"));\n \n-\tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0))\n-\t\twrite_commit_graph_reachable(get_object_directory(), 0, 0);\n+\tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n+\t write_commit_graph_reachable(get_object_directory(), 0, 0))\n+\t\treturn 1;\n \n \trepo_rerere(the_repository, 0);\n \trun_command_v_opt(argv_gc_auto, RUN_GIT_CMD);\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 020f725acc..3984addf73 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -664,9 +664,10 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tclean_pack_garbage();\n \t}\n \n-\tif (gc_write_commit_graph)\n-\t\twrite_commit_graph_reachable(get_object_directory(), 0,\n-\t\t\t\t\t !quiet && !daemonized);\n+\tif (gc_write_commit_graph &&\n+\t write_commit_graph_reachable(get_object_directory(), 0,\n+\t\t\t\t\t !quiet && !daemonized))\n+\t\treturn 1;\n \n \tif (auto_gc && too_many_loose_objects())\n \t\twarning(_(\"There are too many unreachable loose objects; \"\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 66865acbd7..ee487a364b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -851,27 +851,30 @@ static int add_ref_to_list(const char *refname,\n \treturn 0;\n }\n \n-void write_commit_graph_reachable(const char *obj_dir, int append,\n-\t\t\t\t int report_progress)\n+int write_commit_graph_reachable(const char *obj_dir, int append,\n+\t\t\t\t int report_progress)\n {\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n+\tint result;\n \n \tfor_each_ref(add_ref_to_list, &list);\n-\twrite_commit_graph(obj_dir, NULL, &list, append, report_progress);\n+\tresult = write_commit_graph(obj_dir, NULL, &list,\n+\t\t\t\t append, report_progress);\n \n \tstring_list_clear(&list, 0);\n+\treturn result;\n }\n \n-void write_commit_graph(const char *obj_dir,\n-\t\t\tstruct string_list *pack_indexes,\n-\t\t\tstruct string_list *commit_hex,\n-\t\t\tint append, int report_progress)\n+int write_commit_graph(const char *obj_dir,\n+\t\t struct string_list *pack_indexes,\n+\t\t struct string_list *commit_hex,\n+\t\t int append, int report_progress)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n \tstruct hashfile *f;\n \tuint32_t i, count_distinct = 0;\n-\tchar *graph_name;\n+\tchar *graph_name = NULL;\n \tstruct lock_file lk = LOCK_INIT;\n \tuint32_t chunk_ids[5];\n \tuint64_t chunk_offsets[5];\n@@ -883,15 +886,17 @@ void write_commit_graph(const char *obj_dir,\n \tuint64_t progress_cnt = 0;\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tunsigned long approx_nr_objects;\n+\tint res = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n-\t\treturn;\n+\t\treturn 0;\n \n \toids.nr = 0;\n \tapprox_nr_objects = approximate_object_count();\n \toids.alloc = approx_nr_objects / 32;\n \toids.progress = NULL;\n \toids.progress_done = 0;\n+\tcommits.list = NULL;\n \n \tif (append) {\n \t\tprepare_commit_graph_one(the_repository, obj_dir);\n@@ -932,10 +937,16 @@ void write_commit_graph(const char *obj_dir,\n \t\t\tstrbuf_setlen(&packname, dirlen);\n \t\t\tstrbuf_addstr(&packname, pack_indexes->items[i].string);\n \t\t\tp = add_packed_git(packname.buf, packname.len, 1);\n-\t\t\tif (!p)\n-\t\t\t\tdie(_(\"error adding pack %s\"), packname.buf);\n-\t\t\tif (open_pack_index(p))\n-\t\t\t\tdie(_(\"error opening index for %s\"), packname.buf);\n+\t\t\tif (!p) {\n+\t\t\t\terror(_(\"error adding pack %s\"), packname.buf);\n+\t\t\t\tres = 1;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n+\t\t\tif (open_pack_index(p)) {\n+\t\t\t\terror(_(\"error opening index for %s\"), packname.buf);\n+\t\t\t\tres = 1;\n+\t\t\t\tgoto cleanup;\n+\t\t\t}\n \t\t\tfor_each_object_in_pack(p, add_packed_commits, &oids,\n \t\t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n \t\t\tclose_pack(p);\n@@ -1006,8 +1017,11 @@ void write_commit_graph(const char *obj_dir,\n \t}\n \tstop_progress(&progress);\n \n-\tif (count_distinct >= GRAPH_EDGE_LAST_MASK)\n-\t\tdie(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n+\tif (count_distinct >= GRAPH_EDGE_LAST_MASK) {\n+\t\terror(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n+\t\tres = 1;\n+\t\tgoto cleanup;\n+\t}\n \n \tcommits.nr = 0;\n \tcommits.alloc = count_distinct;\n@@ -1039,16 +1053,21 @@ void write_commit_graph(const char *obj_dir,\n \tnum_chunks = num_extra_edges ? 4 : 3;\n \tstop_progress(&progress);\n \n-\tif (commits.nr >= GRAPH_EDGE_LAST_MASK)\n-\t\tdie(_(\"too many commits to write graph\"));\n+\tif (commits.nr >= GRAPH_EDGE_LAST_MASK) {\n+\t\terror(_(\"too many commits to write graph\"));\n+\t\tres = 1;\n+\t\tgoto cleanup;\n+\t}\n \n \tcompute_generation_numbers(&commits, report_progress);\n \n \tgraph_name = get_commit_graph_filename(obj_dir);\n \tif (safe_create_leading_directories(graph_name)) {\n \t\tUNLEAK(graph_name);\n-\t\tdie_errno(_(\"unable to create leading directories of %s\"),\n-\t\t\t graph_name);\n+\t\terror(_(\"unable to create leading directories of %s\"),\n+\t\t\tgraph_name);\n+\t\tres = errno;\n+\t\tgoto cleanup;\n \t}\n \n \thold_lock_file_for_update(&lk, graph_name, LOCK_DIE_ON_ERROR);\n@@ -1107,9 +1126,12 @@ void write_commit_graph(const char *obj_dir,\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n \tcommit_lock_file(&lk);\n \n+cleanup:\n \tfree(graph_name);\n \tfree(commits.list);\n \tfree(oids.list);\n+\n+\treturn res;\n }\n \n #define VERIFY_COMMIT_GRAPH_ERROR_HASH 2\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 7dfb8c896f..d15670bf46 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -65,12 +65,12 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n */\n int generation_numbers_enabled(struct repository *r);\n \n-void write_commit_graph_reachable(const char *obj_dir, int append,\n+int write_commit_graph_reachable(const char *obj_dir, int append,\n \t\t\t\t int report_progress);\n-void write_commit_graph(const char *obj_dir,\n-\t\t\tstruct string_list *pack_indexes,\n-\t\t\tstruct string_list *commit_hex,\n-\t\t\tint append, int report_progress);\n+int write_commit_graph(const char *obj_dir,\n+\t\t struct string_list *pack_indexes,\n+\t\t struct string_list *commit_hex,\n+\t\t int append, int report_progress);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \n-- \ngitgitgadget\n\n"},{"id":"375150","messageId":"9299a7fe255a80f3775747a4bc00b1a81d3356f3.1557330826.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.git.gitgitgadget@gmail.com","subject":"[PATCH 03/17] commit-graph: collapse parameters into flags","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-08T15:53:50Z","receivedAt":"2019-05-08T15:53:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() and write_commit_graph_reachable() methods\ncurrently take two boolean parameters: 'append' and 'report_progress'.\nWe will soon expand the possible options to send to these methods, so\ninstead of complicating the parameter list, first simplify it.\n\nCollapse these parameters into a 'flags' parameter, and adjust the\ncallers to provide flags as necessary.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 8 +++++---\n builtin/commit.c | 2 +-\n builtin/gc.c | 4 ++--\n commit-graph.c | 9 +++++----\n commit-graph.h | 8 +++++---\n 5 files changed, 18 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 2e86251f02..828b1a713f 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -142,6 +142,7 @@ static int graph_write(int argc, const char **argv)\n \tstruct string_list *commit_hex = NULL;\n \tstruct string_list lines;\n \tint result;\n+\tint flags = COMMIT_GRAPH_PROGRESS;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n@@ -166,11 +167,13 @@ static int graph_write(int argc, const char **argv)\n \t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n+\tif (opts.append)\n+\t\tflags |= COMMIT_GRAPH_APPEND;\n \n \tread_replace_refs = 0;\n \n \tif (opts.reachable)\n-\t\treturn write_commit_graph_reachable(opts.obj_dir, opts.append, 1);\n+\t\treturn write_commit_graph_reachable(opts.obj_dir, flags);\n \n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n@@ -190,8 +193,7 @@ static int graph_write(int argc, const char **argv)\n \tresult = write_commit_graph(opts.obj_dir,\n \t\t\t\t pack_indexes,\n \t\t\t\t commit_hex,\n-\t\t\t\t opts.append,\n-\t\t\t\t 1);\n+\t\t\t\t flags);\n \n \tUNLEAK(lines);\n \treturn result;\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex b9ea7222fa..b001ef565d 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1670,7 +1670,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\t \"not exceeded, and then \\\"git reset HEAD\\\" to recover.\"));\n \n \tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n-\t write_commit_graph_reachable(get_object_directory(), 0, 0))\n+\t write_commit_graph_reachable(get_object_directory(), 0))\n \t\treturn 1;\n \n \trepo_rerere(the_repository, 0);\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 3984addf73..df2573f124 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -665,8 +665,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (gc_write_commit_graph &&\n-\t write_commit_graph_reachable(get_object_directory(), 0,\n-\t\t\t\t\t !quiet && !daemonized))\n+\t write_commit_graph_reachable(get_object_directory(),\n+\t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_PROGRESS : 0))\n \t\treturn 1;\n \n \tif (auto_gc && too_many_loose_objects())\ndiff --git a/commit-graph.c b/commit-graph.c\nindex ee487a364b..8bbd50658c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -851,15 +851,14 @@ static int add_ref_to_list(const char *refname,\n \treturn 0;\n }\n \n-int write_commit_graph_reachable(const char *obj_dir, int append,\n-\t\t\t\t int report_progress)\n+int write_commit_graph_reachable(const char *obj_dir, unsigned int flags)\n {\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n \tint result;\n \n \tfor_each_ref(add_ref_to_list, &list);\n \tresult = write_commit_graph(obj_dir, NULL, &list,\n-\t\t\t\t append, report_progress);\n+\t\t\t\t flags);\n \n \tstring_list_clear(&list, 0);\n \treturn result;\n@@ -868,7 +867,7 @@ int write_commit_graph_reachable(const char *obj_dir, int append,\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t int append, int report_progress)\n+\t\t unsigned int flags)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n@@ -887,6 +886,8 @@ int write_commit_graph(const char *obj_dir,\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tunsigned long approx_nr_objects;\n \tint res = 0;\n+\tint append = flags & COMMIT_GRAPH_APPEND;\n+\tint report_progress = flags & COMMIT_GRAPH_PROGRESS;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn 0;\ndiff --git a/commit-graph.h b/commit-graph.h\nindex d15670bf46..70f4caf0c7 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -65,12 +65,14 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n */\n int generation_numbers_enabled(struct repository *r);\n \n-int write_commit_graph_reachable(const char *obj_dir, int append,\n-\t\t\t\t int report_progress);\n+#define COMMIT_GRAPH_APPEND (1 << 0)\n+#define COMMIT_GRAPH_PROGRESS (1 << 1)\n+\n+int write_commit_graph_reachable(const char *obj_dir, unsigned int flags);\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t int append, int report_progress);\n+\t\t unsigned int flags);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \n-- \ngitgitgadget\n\n"},{"id":"375152","messageId":"f9b719fc7a6bc3e025771481af2d671a4b9578f3.1557330826.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.git.gitgitgadget@gmail.com","subject":"[PATCH 04/17] commit-graph: remove Future Work section","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-08T15:53:50Z","receivedAt":"2019-05-08T15:53:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe commit-graph feature began with a long list of planned\nbenefits, most of which are now complete. The future work\nsection has only a few items left.\n\nAs for making more algorithms aware of generation numbers,\nsome are only waiting for generation number v2 to ensure the\nperformance matches the existing behavior using commit date.\n\nIt is unlikely that we will ever send a commit-graph file\nas part of the protocol, since we would need to verify the\ndata, and that is as expensive as writing a commit-graph from\nscratch. If we want to start trusting remote content, then\nthat item can be investigated again.\n\nWhile there is more work to be done on the feature, having\na section of the docs devoted to a TODO list is wasteful and\nhard to keep up-to-date.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 17 -----------------\n 1 file changed, 17 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex 7805b0968c..fb53341d5e 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -127,23 +127,6 @@ Design Details\n helpful for these clones, anyway. The commit-graph will not be read or\n written when shallow commits are present.\n \n-Future Work\n------------\n-\n-- After computing and storing generation numbers, we must make graph\n- walks aware of generation numbers to gain the performance benefits they\n- enable. This will mostly be accomplished by swapping a commit-date-ordered\n- priority queue with one ordered by generation number. The following\n- operations are important candidates:\n-\n- - 'log --topo-order'\n- - 'tag --merged'\n-\n-- A server could provide a commit-graph file as part of the network protocol\n- to avoid extra calculations by clients. This feature is only of benefit if\n- the user is willing to trust the file, because verifying the file is correct\n- is as hard as computing it from scratch.\n-\n Related Links\n -------------\n [0] https://bugs.chromium.org/p/git/issues/detail?id=8\n-- \ngitgitgadget\n\n"},{"id":"375153","messageId":"54817ef50b857257d826d1880714400ee4019980.1557330827.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.git.gitgitgadget@gmail.com","subject":"[PATCH 06/17] commit-graph: extract fill_oids_from_packs()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-08T15:53:52Z","receivedAt":"2019-05-08T15:53:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too complex, so we are\nextracting methods one by one.\n\nThis extracts fill_oids_from_packs() that reads the given\npack-file list and fills the oid list in the context.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 83 ++++++++++++++++++++++++++++----------------------\n 1 file changed, 47 insertions(+), 36 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 58f0f0ae34..80c7069aaa 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -867,6 +867,51 @@ int write_commit_graph_reachable(const char *obj_dir, unsigned int flags)\n \treturn result;\n }\n \n+static int fill_oids_from_packs(struct write_commit_graph_context *ctx,\n+\t\t\t\tstruct string_list *pack_indexes)\n+{\n+\tuint32_t i;\n+\tstruct strbuf progress_title = STRBUF_INIT;\n+\tstruct strbuf packname = STRBUF_INIT;\n+\tint dirlen;\n+\n+\tstrbuf_addf(&packname, \"%s/pack/\", ctx->obj_dir);\n+\tdirlen = packname.len;\n+\tif (ctx->report_progress) {\n+\t\tstrbuf_addf(&progress_title,\n+\t\t\t Q_(\"Finding commits for commit graph in %d pack\",\n+\t\t\t \"Finding commits for commit graph in %d packs\",\n+\t\t\t pack_indexes->nr),\n+\t\t\t pack_indexes->nr);\n+\t\tctx->progress = start_delayed_progress(progress_title.buf, 0);\n+\t\tctx->progress_done = 0;\n+\t}\n+\tfor (i = 0; i < pack_indexes->nr; i++) {\n+\t\tstruct packed_git *p;\n+\t\tstrbuf_setlen(&packname, dirlen);\n+\t\tstrbuf_addstr(&packname, pack_indexes->items[i].string);\n+\t\tp = add_packed_git(packname.buf, packname.len, 1);\n+\t\tif (!p) {\n+\t\t\terror(_(\"error adding pack %s\"), packname.buf);\n+\t\t\treturn 1;\n+\t\t}\n+\t\tif (open_pack_index(p)) {\n+\t\t\terror(_(\"error opening index for %s\"), packname.buf);\n+\t\t\treturn 1;\n+\t\t}\n+\t\tfor_each_object_in_pack(p, add_packed_commits, ctx,\n+\t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n+\t\tclose_pack(p);\n+\t\tfree(p);\n+\t}\n+\n+\tstop_progress(&ctx->progress);\n+\tstrbuf_reset(&progress_title);\n+\tstrbuf_release(&packname);\n+\n+\treturn 0;\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -916,42 +961,8 @@ int write_commit_graph(const char *obj_dir,\n \t}\n \n \tif (pack_indexes) {\n-\t\tstruct strbuf packname = STRBUF_INIT;\n-\t\tint dirlen;\n-\t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n-\t\tdirlen = packname.len;\n-\t\tif (ctx->report_progress) {\n-\t\t\tstrbuf_addf(&progress_title,\n-\t\t\t\t Q_(\"Finding commits for commit graph in %d pack\",\n-\t\t\t\t \"Finding commits for commit graph in %d packs\",\n-\t\t\t\t pack_indexes->nr),\n-\t\t\t\t pack_indexes->nr);\n-\t\t\tctx->progress = start_delayed_progress(progress_title.buf, 0);\n-\t\t\tctx->progress_done = 0;\n-\t\t}\n-\t\tfor (i = 0; i < pack_indexes->nr; i++) {\n-\t\t\tstruct packed_git *p;\n-\t\t\tstrbuf_setlen(&packname, dirlen);\n-\t\t\tstrbuf_addstr(&packname, pack_indexes->items[i].string);\n-\t\t\tp = add_packed_git(packname.buf, packname.len, 1);\n-\t\t\tif (!p) {\n-\t\t\t\terror(_(\"error adding pack %s\"), packname.buf);\n-\t\t\t\tres = 1;\n-\t\t\t\tgoto cleanup;\n-\t\t\t}\n-\t\t\tif (open_pack_index(p)) {\n-\t\t\t\terror(_(\"error opening index for %s\"), packname.buf);\n-\t\t\t\tres = 1;\n-\t\t\t\tgoto cleanup;\n-\t\t\t}\n-\t\t\tfor_each_object_in_pack(p, add_packed_commits, ctx,\n-\t\t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n-\t\t\tclose_pack(p);\n-\t\t\tfree(p);\n-\t\t}\n-\t\tstop_progress(&ctx->progress);\n-\t\tstrbuf_reset(&progress_title);\n-\t\tstrbuf_release(&packname);\n+\t\tif ((res = fill_oids_from_packs(ctx, pack_indexes)))\n+\t\t\tgoto cleanup;\n \t}\n \n \tif (commit_hex) {\n-- \ngitgitgadget\n\n"},{"id":"375155","messageId":"aaae85f1ec0b39a800078ce2aba68f9b1b6b71d1.1557330827.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.git.gitgitgadget@gmail.com","subject":"[PATCH 08/17] commit-graph: extract fill_oids_from_all_packs()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-08T15:53:54Z","receivedAt":"2019-05-08T15:53:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too complex, so we are\nextracting methods one by one.\n\nExtract fill_oids_from_all_packs() that reads all pack-files\nfor commits and fills the oid list in the context.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 26 +++++++++++++++-----------\n 1 file changed, 15 insertions(+), 11 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex fb25280df1..730d529815 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -950,6 +950,19 @@ static void fill_oids_from_commit_hex(struct write_commit_graph_context *ctx,\n \tstrbuf_release(&progress_title);\n }\n \n+static void fill_oids_from_all_packs(struct write_commit_graph_context *ctx)\n+{\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t_(\"Finding commits for commit graph among packed objects\"),\n+\t\t\tctx->approx_nr_objects);\n+\tfor_each_packed_object(add_packed_commits, ctx,\n+\t\t\t FOR_EACH_OBJECT_PACK_ORDER);\n+\tif (ctx->progress_done < ctx->approx_nr_objects)\n+\t\tdisplay_progress(ctx->progress, ctx->approx_nr_objects);\n+\tstop_progress(&ctx->progress);\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -1006,17 +1019,8 @@ int write_commit_graph(const char *obj_dir,\n \tif (commit_hex)\n \t\tfill_oids_from_commit_hex(ctx, commit_hex);\n \n-\tif (!pack_indexes && !commit_hex) {\n-\t\tif (ctx->report_progress)\n-\t\t\tctx->progress = start_delayed_progress(\n-\t\t\t\t_(\"Finding commits for commit graph among packed objects\"),\n-\t\t\t\tctx->approx_nr_objects);\n-\t\tfor_each_packed_object(add_packed_commits, ctx,\n-\t\t\t\t FOR_EACH_OBJECT_PACK_ORDER);\n-\t\tif (ctx->progress_done < ctx->approx_nr_objects)\n-\t\t\tdisplay_progress(ctx->progress, ctx->approx_nr_objects);\n-\t\tstop_progress(&ctx->progress);\n-\t}\n+\tif (!pack_indexes && !commit_hex)\n+\t\tfill_oids_from_all_packs(ctx);\n \n \tclose_reachable(ctx);\n \n-- \ngitgitgadget\n\n"},{"id":"375154","messageId":"ebd665468e1347752dabc0c963c890530ea0ea55.1557330827.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.git.gitgitgadget@gmail.com","subject":"[PATCH 10/17] commit-graph: extract copy_oids_to_commits()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-08T15:53:55Z","receivedAt":"2019-05-08T15:53:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too complex, so we are\nextracting methods one by one.\n\nExtract copy_oids_to_commits(), which fills the commits list\nwith the distinct commits from the oids list. During this loop,\nit also counts the number of \"extra\" edges from octopus merges.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 57 ++++++++++++++++++++++++++++----------------------\n 1 file changed, 32 insertions(+), 25 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f7419c919b..16cdd7afb2 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -984,6 +984,37 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n \treturn count_distinct;\n }\n \n+static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n+{\n+\tuint32_t i;\n+\tstruct commit_list *parent;\n+\n+\tctx->num_extra_edges = 0;\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t_(\"Finding extra edges in commit graph\"),\n+\t\t\tctx->oids.nr);\n+\tfor (i = 0; i < ctx->oids.nr; i++) {\n+\t\tint num_parents = 0;\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tif (i > 0 && oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n+\t\t\tcontinue;\n+\n+\t\tctx->commits.list[ctx->commits.nr] = lookup_commit(ctx->r, &ctx->oids.list[i]);\n+\t\tparse_commit_no_graph(ctx->commits.list[ctx->commits.nr]);\n+\n+\t\tfor (parent = ctx->commits.list[ctx->commits.nr]->parents;\n+\t\t parent; parent = parent->next)\n+\t\t\tnum_parents++;\n+\n+\t\tif (num_parents > 2)\n+\t\t\tctx->num_extra_edges += num_parents - 1;\n+\n+\t\tctx->commits.nr++;\n+\t}\n+\tstop_progress(&ctx->progress);\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -997,7 +1028,6 @@ int write_commit_graph(const char *obj_dir,\n \tuint32_t chunk_ids[5];\n \tuint64_t chunk_offsets[5];\n \tint num_chunks;\n-\tstruct commit_list *parent;\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tint res = 0;\n@@ -1056,30 +1086,7 @@ int write_commit_graph(const char *obj_dir,\n \tctx->commits.alloc = count_distinct;\n \tALLOC_ARRAY(ctx->commits.list, ctx->commits.alloc);\n \n-\tctx->num_extra_edges = 0;\n-\tif (ctx->report_progress)\n-\t\tctx->progress = start_delayed_progress(\n-\t\t\t_(\"Finding extra edges in commit graph\"),\n-\t\t\tctx->oids.nr);\n-\tfor (i = 0; i < ctx->oids.nr; i++) {\n-\t\tint num_parents = 0;\n-\t\tdisplay_progress(ctx->progress, i + 1);\n-\t\tif (i > 0 && oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n-\t\t\tcontinue;\n-\n-\t\tctx->commits.list[ctx->commits.nr] = lookup_commit(ctx->r, &ctx->oids.list[i]);\n-\t\tparse_commit_no_graph(ctx->commits.list[ctx->commits.nr]);\n-\n-\t\tfor (parent = ctx->commits.list[ctx->commits.nr]->parents;\n-\t\t parent; parent = parent->next)\n-\t\t\tnum_parents++;\n-\n-\t\tif (num_parents > 2)\n-\t\t\tctx->num_extra_edges += num_parents - 1;\n-\n-\t\tctx->commits.nr++;\n-\t}\n-\tstop_progress(&ctx->progress);\n+\tcopy_oids_to_commits(ctx);\n \n \tif (ctx->commits.nr >= GRAPH_EDGE_LAST_MASK) {\n \t\terror(_(\"too many commits to write graph\"));\n-- \ngitgitgadget\n\n"},{"id":"375156","messageId":"9d434dc38c7f3762de94dee57185c0504cf2667e.1557330827.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.git.gitgitgadget@gmail.com","subject":"[PATCH 09/17] commit-graph: extract count_distinct_commits()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-08T15:53:55Z","receivedAt":"2019-05-08T15:53:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too complex, so we are\nextracting methods one by one.\n\nExtract count_distinct_commits(), which sorts the oids list, then\niterates through to find duplicates.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 35 ++++++++++++++++++++++-------------\n 1 file changed, 22 insertions(+), 13 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 730d529815..f7419c919b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -963,6 +963,27 @@ static void fill_oids_from_all_packs(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n+static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n+{\n+\tuint32_t i, count_distinct = 1;\n+\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t_(\"Counting distinct commits in commit graph\"),\n+\t\t\tctx->oids.nr);\n+\tdisplay_progress(ctx->progress, 0); /* TODO: Measure QSORT() progress */\n+\tQSORT(ctx->oids.list, ctx->oids.nr, commit_compare);\n+\n+\tfor (i = 1; i < ctx->oids.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n+\t\t\tcount_distinct++;\n+\t}\n+\tstop_progress(&ctx->progress);\n+\n+\treturn count_distinct;\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -1024,19 +1045,7 @@ int write_commit_graph(const char *obj_dir,\n \n \tclose_reachable(ctx);\n \n-\tif (ctx->report_progress)\n-\t\tctx->progress = start_delayed_progress(\n-\t\t\t_(\"Counting distinct commits in commit graph\"),\n-\t\t\tctx->oids.nr);\n-\tdisplay_progress(ctx->progress, 0); /* TODO: Measure QSORT() progress */\n-\tQSORT(ctx->oids.list, ctx->oids.nr, commit_compare);\n-\tcount_distinct = 1;\n-\tfor (i = 1; i < ctx->oids.nr; i++) {\n-\t\tdisplay_progress(ctx->progress, i + 1);\n-\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n-\t\t\tcount_distinct++;\n-\t}\n-\tstop_progress(&ctx->progress);\n+\tcount_distinct = count_distinct_commits(ctx);\n \n \tif (count_distinct >= GRAPH_EDGE_LAST_MASK) {\n \t\terror(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n-- \ngitgitgadget\n\n"},{"id":"375157","messageId":"cf792d38ed8c6315b209f5a3c8b9dd010ff572a5.1557330827.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.git.gitgitgadget@gmail.com","subject":"[PATCH 07/17] commit-graph: extract fill_oids_from_commit_hex()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-08T15:53:53Z","receivedAt":"2019-05-08T15:54:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too complex, so we are\nextracting methods one by one.\n\nExtract fill_oids_from_commit_hex() that reads the given commit\nid list and fille the oid list in the context.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 72 ++++++++++++++++++++++++++++----------------------\n 1 file changed, 40 insertions(+), 32 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 80c7069aaa..fb25280df1 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -912,6 +912,44 @@ static int fill_oids_from_packs(struct write_commit_graph_context *ctx,\n \treturn 0;\n }\n \n+static void fill_oids_from_commit_hex(struct write_commit_graph_context *ctx,\n+\t\t\t\t struct string_list *commit_hex)\n+{\n+\tuint32_t i;\n+\tstruct strbuf progress_title = STRBUF_INIT;\n+\n+\tif (ctx->report_progress) {\n+\t\tstrbuf_addf(&progress_title,\n+\t\t\t Q_(\"Finding commits for commit graph from %d ref\",\n+\t\t\t \"Finding commits for commit graph from %d refs\",\n+\t\t\t commit_hex->nr),\n+\t\t\t commit_hex->nr);\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\tprogress_title.buf,\n+\t\t\t\t\tcommit_hex->nr);\n+\t}\n+\tfor (i = 0; i < commit_hex->nr; i++) {\n+\t\tconst char *end;\n+\t\tstruct object_id oid;\n+\t\tstruct commit *result;\n+\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tif (commit_hex->items[i].string &&\n+\t\t parse_oid_hex(commit_hex->items[i].string, &oid, &end))\n+\t\t\tcontinue;\n+\n+\t\tresult = lookup_commit_reference_gently(ctx->r, &oid, 1);\n+\n+\t\tif (result) {\n+\t\t\tALLOC_GROW(ctx->oids.list, ctx->oids.nr + 1, ctx->oids.alloc);\n+\t\t\toidcpy(&ctx->oids.list[ctx->oids.nr], &(result->object.oid));\n+\t\t\tctx->oids.nr++;\n+\t\t}\n+\t}\n+\tstop_progress(&ctx->progress);\n+\tstrbuf_release(&progress_title);\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -965,38 +1003,8 @@ int write_commit_graph(const char *obj_dir,\n \t\t\tgoto cleanup;\n \t}\n \n-\tif (commit_hex) {\n-\t\tif (ctx->report_progress) {\n-\t\t\tstrbuf_addf(&progress_title,\n-\t\t\t\t Q_(\"Finding commits for commit graph from %d ref\",\n-\t\t\t\t \"Finding commits for commit graph from %d refs\",\n-\t\t\t\t commit_hex->nr),\n-\t\t\t\t commit_hex->nr);\n-\t\t\tctx->progress = start_delayed_progress(\n-\t\t\t\t\t\tprogress_title.buf,\n-\t\t\t\t\t\tcommit_hex->nr);\n-\t\t}\n-\t\tfor (i = 0; i < commit_hex->nr; i++) {\n-\t\t\tconst char *end;\n-\t\t\tstruct object_id oid;\n-\t\t\tstruct commit *result;\n-\n-\t\t\tdisplay_progress(ctx->progress, i + 1);\n-\t\t\tif (commit_hex->items[i].string &&\n-\t\t\t parse_oid_hex(commit_hex->items[i].string, &oid, &end))\n-\t\t\t\tcontinue;\n-\n-\t\t\tresult = lookup_commit_reference_gently(ctx->r, &oid, 1);\n-\n-\t\t\tif (result) {\n-\t\t\t\tALLOC_GROW(ctx->oids.list, ctx->oids.nr + 1, ctx->oids.alloc);\n-\t\t\t\toidcpy(&ctx->oids.list[ctx->oids.nr], &(result->object.oid));\n-\t\t\t\tctx->oids.nr++;\n-\t\t\t}\n-\t\t}\n-\t\tstop_progress(&ctx->progress);\n-\t\tstrbuf_reset(&progress_title);\n-\t}\n+\tif (commit_hex)\n+\t\tfill_oids_from_commit_hex(ctx, commit_hex);\n \n \tif (!pack_indexes && !commit_hex) {\n \t\tif (ctx->report_progress)\n-- \ngitgitgadget\n\n"},{"id":"375158","messageId":"74e40970e0dfe00404a8ef52f0d38389a6e4a987.1557330826.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.git.gitgitgadget@gmail.com","subject":"[PATCH 05/17] commit-graph: create write_commit_graph_context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-08T15:53:51Z","receivedAt":"2019-05-08T15:54:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too large and complex. To simplify\nit, we should extract several small methods. However, we will risk\nrepeating a lot of declarations related to progress incidators and\nobject id or commit lists.\n\nCreate a new write_commit_graph_context struct that contains the\ncore data structures used in this process. Replace the other local\nvariables with the values inside the context object. Following this\nchange, we will start to lift code segments wholesale out of the\nwrite_commit_graph() method and into their own methods.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 390 ++++++++++++++++++++++++-------------------------\n 1 file changed, 194 insertions(+), 196 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8bbd50658c..58f0f0ae34 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -518,14 +518,38 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \treturn get_commit_tree_in_graph_one(r, r->objects->commit_graph, c);\n }\n \n+struct packed_commit_list {\n+\tstruct commit **list;\n+\tint nr;\n+\tint alloc;\n+};\n+\n+struct packed_oid_list {\n+\tstruct object_id *list;\n+\tint nr;\n+\tint alloc;\n+};\n+\n+struct write_commit_graph_context {\n+\tstruct repository *r;\n+\tconst char *obj_dir;\n+\tchar *graph_name;\n+\tstruct packed_oid_list oids;\n+\tstruct packed_commit_list commits;\n+\tint num_extra_edges;\n+\tunsigned long approx_nr_objects;\n+\tstruct progress *progress;\n+\tint progress_done;\n+\tuint64_t progress_cnt;\n+\tunsigned append:1,\n+\t\t report_progress:1;\n+};\n+\n static void write_graph_chunk_fanout(struct hashfile *f,\n-\t\t\t\t struct commit **commits,\n-\t\t\t\t int nr_commits,\n-\t\t\t\t struct progress *progress,\n-\t\t\t\t uint64_t *progress_cnt)\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tint i, count = 0;\n-\tstruct commit **list = commits;\n+\tstruct commit **list = ctx->commits.list;\n \n \t/*\n \t * Write the first-level table (the list is sorted,\n@@ -533,10 +557,10 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n \t * having to do eight extra binary search iterations).\n \t */\n \tfor (i = 0; i < 256; i++) {\n-\t\twhile (count < nr_commits) {\n+\t\twhile (count < ctx->commits.nr) {\n \t\t\tif ((*list)->object.oid.hash[0] != i)\n \t\t\t\tbreak;\n-\t\t\tdisplay_progress(progress, ++*progress_cnt);\n+\t\t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n \t\t\tcount++;\n \t\t\tlist++;\n \t\t}\n@@ -546,14 +570,12 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n }\n \n static void write_graph_chunk_oids(struct hashfile *f, int hash_len,\n-\t\t\t\t struct commit **commits, int nr_commits,\n-\t\t\t\t struct progress *progress,\n-\t\t\t\t uint64_t *progress_cnt)\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n-\tstruct commit **list = commits;\n+\tstruct commit **list = ctx->commits.list;\n \tint count;\n-\tfor (count = 0; count < nr_commits; count++, list++) {\n-\t\tdisplay_progress(progress, ++*progress_cnt);\n+\tfor (count = 0; count < ctx->commits.nr; count++, list++) {\n+\t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n \t\thashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n \t}\n }\n@@ -565,19 +587,17 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n }\n \n static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n-\t\t\t\t struct commit **commits, int nr_commits,\n-\t\t\t\t struct progress *progress,\n-\t\t\t\t uint64_t *progress_cnt)\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n-\tstruct commit **list = commits;\n-\tstruct commit **last = commits + nr_commits;\n+\tstruct commit **list = ctx->commits.list;\n+\tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tuint32_t num_extra_edges = 0;\n \n \twhile (list < last) {\n \t\tstruct commit_list *parent;\n \t\tint edge_value;\n \t\tuint32_t packedDate[2];\n-\t\tdisplay_progress(progress, ++*progress_cnt);\n+\t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n \n \t\tparse_commit_no_graph(*list);\n \t\thashwrite(f, get_commit_tree_oid(*list)->hash, hash_len);\n@@ -588,8 +608,8 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\tedge_value = GRAPH_PARENT_NONE;\n \t\telse {\n \t\t\tedge_value = sha1_pos(parent->item->object.oid.hash,\n-\t\t\t\t\t commits,\n-\t\t\t\t\t nr_commits,\n+\t\t\t\t\t ctx->commits.list,\n+\t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t commit_to_sha1);\n \n \t\t\tif (edge_value < 0)\n@@ -609,8 +629,8 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\tedge_value = GRAPH_EXTRA_EDGES_NEEDED | num_extra_edges;\n \t\telse {\n \t\t\tedge_value = sha1_pos(parent->item->object.oid.hash,\n-\t\t\t\t\t commits,\n-\t\t\t\t\t nr_commits,\n+\t\t\t\t\t ctx->commits.list,\n+\t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t commit_to_sha1);\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n@@ -642,19 +662,16 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n }\n \n static void write_graph_chunk_extra_edges(struct hashfile *f,\n-\t\t\t\t\t struct commit **commits,\n-\t\t\t\t\t int nr_commits,\n-\t\t\t\t\t struct progress *progress,\n-\t\t\t\t\t uint64_t *progress_cnt)\n+\t\t\t\t\t struct write_commit_graph_context *ctx)\n {\n-\tstruct commit **list = commits;\n-\tstruct commit **last = commits + nr_commits;\n+\tstruct commit **list = ctx->commits.list;\n+\tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n \tstruct commit_list *parent;\n \n \twhile (list < last) {\n \t\tint num_parents = 0;\n \n-\t\tdisplay_progress(progress, ++*progress_cnt);\n+\t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n \n \t\tfor (parent = (*list)->parents; num_parents < 3 && parent;\n \t\t parent = parent->next)\n@@ -668,8 +685,8 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n \t\t/* Since num_parents > 2, this initializer is safe. */\n \t\tfor (parent = (*list)->parents->next; parent; parent = parent->next) {\n \t\t\tint edge_value = sha1_pos(parent->item->object.oid.hash,\n-\t\t\t\t\t\t commits,\n-\t\t\t\t\t\t nr_commits,\n+\t\t\t\t\t\t ctx->commits.list,\n+\t\t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t\t commit_to_sha1);\n \n \t\t\tif (edge_value < 0)\n@@ -693,125 +710,111 @@ static int commit_compare(const void *_a, const void *_b)\n \treturn oidcmp(a, b);\n }\n \n-struct packed_commit_list {\n-\tstruct commit **list;\n-\tint nr;\n-\tint alloc;\n-};\n-\n-struct packed_oid_list {\n-\tstruct object_id *list;\n-\tint nr;\n-\tint alloc;\n-\tstruct progress *progress;\n-\tint progress_done;\n-};\n-\n static int add_packed_commits(const struct object_id *oid,\n \t\t\t struct packed_git *pack,\n \t\t\t uint32_t pos,\n \t\t\t void *data)\n {\n-\tstruct packed_oid_list *list = (struct packed_oid_list*)data;\n+\tstruct write_commit_graph_context *ctx = (struct write_commit_graph_context*)data;\n \tenum object_type type;\n \toff_t offset = nth_packed_object_offset(pack, pos);\n \tstruct object_info oi = OBJECT_INFO_INIT;\n \n-\tif (list->progress)\n-\t\tdisplay_progress(list->progress, ++list->progress_done);\n+\tif (ctx->progress)\n+\t\tdisplay_progress(ctx->progress, ++ctx->progress_done);\n \n \toi.typep = &type;\n-\tif (packed_object_info(the_repository, pack, offset, &oi) < 0)\n+\tif (packed_object_info(ctx->r, pack, offset, &oi) < 0)\n \t\tdie(_(\"unable to get type of object %s\"), oid_to_hex(oid));\n \n \tif (type != OBJ_COMMIT)\n \t\treturn 0;\n \n-\tALLOC_GROW(list->list, list->nr + 1, list->alloc);\n-\toidcpy(&(list->list[list->nr]), oid);\n-\tlist->nr++;\n+\tALLOC_GROW(ctx->oids.list, ctx->oids.nr + 1, ctx->oids.alloc);\n+\toidcpy(&(ctx->oids.list[ctx->oids.nr]), oid);\n+\tctx->oids.nr++;\n \n \treturn 0;\n }\n \n-static void add_missing_parents(struct packed_oid_list *oids, struct commit *commit)\n+static void add_missing_parents(struct write_commit_graph_context *ctx, struct commit *commit)\n {\n \tstruct commit_list *parent;\n \tfor (parent = commit->parents; parent; parent = parent->next) {\n \t\tif (!(parent->item->object.flags & UNINTERESTING)) {\n-\t\t\tALLOC_GROW(oids->list, oids->nr + 1, oids->alloc);\n-\t\t\toidcpy(&oids->list[oids->nr], &(parent->item->object.oid));\n-\t\t\toids->nr++;\n+\t\t\tALLOC_GROW(ctx->oids.list, ctx->oids.nr + 1, ctx->oids.alloc);\n+\t\t\toidcpy(&ctx->oids.list[ctx->oids.nr], &(parent->item->object.oid));\n+\t\t\tctx->oids.nr++;\n \t\t\tparent->item->object.flags |= UNINTERESTING;\n \t\t}\n \t}\n }\n \n-static void close_reachable(struct packed_oid_list *oids, int report_progress)\n+static void close_reachable(struct write_commit_graph_context *ctx)\n {\n \tint i;\n \tstruct commit *commit;\n-\tstruct progress *progress = NULL;\n \n-\tif (report_progress)\n-\t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Loading known commits in commit graph\"), oids->nr);\n-\tfor (i = 0; i < oids->nr; i++) {\n-\t\tdisplay_progress(progress, i + 1);\n-\t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"Loading known commits in commit graph\"),\n+\t\t\t\t\tctx->oids.nr);\n+\tfor (i = 0; i < ctx->oids.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tcommit = lookup_commit(ctx->r, &ctx->oids.list[i]);\n \t\tif (commit)\n \t\t\tcommit->object.flags |= UNINTERESTING;\n \t}\n-\tstop_progress(&progress);\n+\tstop_progress(&ctx->progress);\n \n \t/*\n-\t * As this loop runs, oids->nr may grow, but not more\n+\t * As this loop runs, ctx->oids.nr may grow, but not more\n \t * than the number of missing commits in the reachable\n \t * closure.\n \t */\n-\tif (report_progress)\n-\t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Expanding reachable commits in commit graph\"), oids->nr);\n-\tfor (i = 0; i < oids->nr; i++) {\n-\t\tdisplay_progress(progress, i + 1);\n-\t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"Expanding reachable commits in commit graph\"),\n+\t\t\t\t\tctx->oids.nr);\n+\tfor (i = 0; i < ctx->oids.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tcommit = lookup_commit(ctx->r, &ctx->oids.list[i]);\n \n \t\tif (commit && !parse_commit_no_graph(commit))\n-\t\t\tadd_missing_parents(oids, commit);\n+\t\t\tadd_missing_parents(ctx, commit);\n \t}\n-\tstop_progress(&progress);\n+\tstop_progress(&ctx->progress);\n \n-\tif (report_progress)\n-\t\tprogress = start_delayed_progress(\n-\t\t\t_(\"Clearing commit marks in commit graph\"), oids->nr);\n-\tfor (i = 0; i < oids->nr; i++) {\n-\t\tdisplay_progress(progress, i + 1);\n-\t\tcommit = lookup_commit(the_repository, &oids->list[i]);\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"Clearing commit marks in commit graph\"),\n+\t\t\t\t\tctx->oids.nr);\n+\tfor (i = 0; i < ctx->oids.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tcommit = lookup_commit(ctx->r, &ctx->oids.list[i]);\n \n \t\tif (commit)\n \t\t\tcommit->object.flags &= ~UNINTERESTING;\n \t}\n-\tstop_progress(&progress);\n+\tstop_progress(&ctx->progress);\n }\n \n-static void compute_generation_numbers(struct packed_commit_list* commits,\n-\t\t\t\t int report_progress)\n+static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n {\n \tint i;\n \tstruct commit_list *list = NULL;\n-\tstruct progress *progress = NULL;\n \n-\tif (report_progress)\n-\t\tprogress = start_progress(\n-\t\t\t_(\"Computing commit graph generation numbers\"),\n-\t\t\tcommits->nr);\n-\tfor (i = 0; i < commits->nr; i++) {\n-\t\tdisplay_progress(progress, i + 1);\n-\t\tif (commits->list[i]->generation != GENERATION_NUMBER_INFINITY &&\n-\t\t commits->list[i]->generation != GENERATION_NUMBER_ZERO)\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_progress(\n+\t\t\t\t\t_(\"Computing commit graph generation numbers\"),\n+\t\t\t\t\tctx->commits.nr);\n+\tfor (i = 0; i < ctx->commits.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tif (ctx->commits.list[i]->generation != GENERATION_NUMBER_INFINITY &&\n+\t\t ctx->commits.list[i]->generation != GENERATION_NUMBER_ZERO)\n \t\t\tcontinue;\n \n-\t\tcommit_list_insert(commits->list[i], &list);\n+\t\tcommit_list_insert(ctx->commits.list[i], &list);\n \t\twhile (list) {\n \t\t\tstruct commit *current = list->item;\n \t\t\tstruct commit_list *parent;\n@@ -838,7 +841,7 @@ static void compute_generation_numbers(struct packed_commit_list* commits,\n \t\t\t}\n \t\t}\n \t}\n-\tstop_progress(&progress);\n+\tstop_progress(&ctx->progress);\n }\n \n static int add_ref_to_list(const char *refname,\n@@ -869,8 +872,7 @@ int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *commit_hex,\n \t\t unsigned int flags)\n {\n-\tstruct packed_oid_list oids;\n-\tstruct packed_commit_list commits;\n+\tstruct write_commit_graph_context *ctx;\n \tstruct hashfile *f;\n \tuint32_t i, count_distinct = 0;\n \tchar *graph_name = NULL;\n@@ -878,44 +880,38 @@ int write_commit_graph(const char *obj_dir,\n \tuint32_t chunk_ids[5];\n \tuint64_t chunk_offsets[5];\n \tint num_chunks;\n-\tint num_extra_edges;\n \tstruct commit_list *parent;\n-\tstruct progress *progress = NULL;\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n-\tuint64_t progress_cnt = 0;\n \tstruct strbuf progress_title = STRBUF_INIT;\n-\tunsigned long approx_nr_objects;\n \tint res = 0;\n-\tint append = flags & COMMIT_GRAPH_APPEND;\n-\tint report_progress = flags & COMMIT_GRAPH_PROGRESS;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn 0;\n \n-\toids.nr = 0;\n-\tapprox_nr_objects = approximate_object_count();\n-\toids.alloc = approx_nr_objects / 32;\n-\toids.progress = NULL;\n-\toids.progress_done = 0;\n-\tcommits.list = NULL;\n-\n-\tif (append) {\n-\t\tprepare_commit_graph_one(the_repository, obj_dir);\n-\t\tif (the_repository->objects->commit_graph)\n-\t\t\toids.alloc += the_repository->objects->commit_graph->num_commits;\n+\tctx = xcalloc(1, sizeof(struct write_commit_graph_context));\n+\tctx->r = the_repository;\n+\tctx->obj_dir = obj_dir;\n+\tctx->append = flags & COMMIT_GRAPH_APPEND ? 1 : 0;\n+\tctx->report_progress = flags & COMMIT_GRAPH_PROGRESS ? 1 : 0;\n+\n+\tctx->approx_nr_objects = approximate_object_count();\n+\tctx->oids.alloc = ctx->approx_nr_objects / 32;\n+\n+\tif (ctx->append) {\n+\t\tprepare_commit_graph_one(ctx->r, ctx->obj_dir);\n+\t\tif (ctx->r->objects->commit_graph)\n+\t\t\tctx->oids.alloc += ctx->r->objects->commit_graph->num_commits;\n \t}\n \n-\tif (oids.alloc < 1024)\n-\t\toids.alloc = 1024;\n-\tALLOC_ARRAY(oids.list, oids.alloc);\n-\n-\tif (append && the_repository->objects->commit_graph) {\n-\t\tstruct commit_graph *commit_graph =\n-\t\t\tthe_repository->objects->commit_graph;\n-\t\tfor (i = 0; i < commit_graph->num_commits; i++) {\n-\t\t\tconst unsigned char *hash = commit_graph->chunk_oid_lookup +\n-\t\t\t\tcommit_graph->hash_len * i;\n-\t\t\thashcpy(oids.list[oids.nr++].hash, hash);\n+\tif (ctx->oids.alloc < 1024)\n+\t\tctx->oids.alloc = 1024;\n+\tALLOC_ARRAY(ctx->oids.list, ctx->oids.alloc);\n+\n+\tif (ctx->append && ctx->r->objects->commit_graph) {\n+\t\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\t\tfor (i = 0; i < g->num_commits; i++) {\n+\t\t\tconst unsigned char *hash = g->chunk_oid_lookup + g->hash_len * i;\n+\t\t\thashcpy(ctx->oids.list[ctx->oids.nr++].hash, hash);\n \t\t}\n \t}\n \n@@ -924,14 +920,14 @@ int write_commit_graph(const char *obj_dir,\n \t\tint dirlen;\n \t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n \t\tdirlen = packname.len;\n-\t\tif (report_progress) {\n+\t\tif (ctx->report_progress) {\n \t\t\tstrbuf_addf(&progress_title,\n \t\t\t\t Q_(\"Finding commits for commit graph in %d pack\",\n \t\t\t\t \"Finding commits for commit graph in %d packs\",\n \t\t\t\t pack_indexes->nr),\n \t\t\t\t pack_indexes->nr);\n-\t\t\toids.progress = start_delayed_progress(progress_title.buf, 0);\n-\t\t\toids.progress_done = 0;\n+\t\t\tctx->progress = start_delayed_progress(progress_title.buf, 0);\n+\t\t\tctx->progress_done = 0;\n \t\t}\n \t\tfor (i = 0; i < pack_indexes->nr; i++) {\n \t\t\tstruct packed_git *p;\n@@ -948,75 +944,76 @@ int write_commit_graph(const char *obj_dir,\n \t\t\t\tres = 1;\n \t\t\t\tgoto cleanup;\n \t\t\t}\n-\t\t\tfor_each_object_in_pack(p, add_packed_commits, &oids,\n+\t\t\tfor_each_object_in_pack(p, add_packed_commits, ctx,\n \t\t\t\t\t\tFOR_EACH_OBJECT_PACK_ORDER);\n \t\t\tclose_pack(p);\n \t\t\tfree(p);\n \t\t}\n-\t\tstop_progress(&oids.progress);\n+\t\tstop_progress(&ctx->progress);\n \t\tstrbuf_reset(&progress_title);\n \t\tstrbuf_release(&packname);\n \t}\n \n \tif (commit_hex) {\n-\t\tif (report_progress) {\n+\t\tif (ctx->report_progress) {\n \t\t\tstrbuf_addf(&progress_title,\n \t\t\t\t Q_(\"Finding commits for commit graph from %d ref\",\n \t\t\t\t \"Finding commits for commit graph from %d refs\",\n \t\t\t\t commit_hex->nr),\n \t\t\t\t commit_hex->nr);\n-\t\t\tprogress = start_delayed_progress(progress_title.buf,\n-\t\t\t\t\t\t\t commit_hex->nr);\n+\t\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t\tprogress_title.buf,\n+\t\t\t\t\t\tcommit_hex->nr);\n \t\t}\n \t\tfor (i = 0; i < commit_hex->nr; i++) {\n \t\t\tconst char *end;\n \t\t\tstruct object_id oid;\n \t\t\tstruct commit *result;\n \n-\t\t\tdisplay_progress(progress, i + 1);\n+\t\t\tdisplay_progress(ctx->progress, i + 1);\n \t\t\tif (commit_hex->items[i].string &&\n \t\t\t parse_oid_hex(commit_hex->items[i].string, &oid, &end))\n \t\t\t\tcontinue;\n \n-\t\t\tresult = lookup_commit_reference_gently(the_repository, &oid, 1);\n+\t\t\tresult = lookup_commit_reference_gently(ctx->r, &oid, 1);\n \n \t\t\tif (result) {\n-\t\t\t\tALLOC_GROW(oids.list, oids.nr + 1, oids.alloc);\n-\t\t\t\toidcpy(&oids.list[oids.nr], &(result->object.oid));\n-\t\t\t\toids.nr++;\n+\t\t\t\tALLOC_GROW(ctx->oids.list, ctx->oids.nr + 1, ctx->oids.alloc);\n+\t\t\t\toidcpy(&ctx->oids.list[ctx->oids.nr], &(result->object.oid));\n+\t\t\t\tctx->oids.nr++;\n \t\t\t}\n \t\t}\n-\t\tstop_progress(&progress);\n+\t\tstop_progress(&ctx->progress);\n \t\tstrbuf_reset(&progress_title);\n \t}\n \n \tif (!pack_indexes && !commit_hex) {\n-\t\tif (report_progress)\n-\t\t\toids.progress = start_delayed_progress(\n+\t\tif (ctx->report_progress)\n+\t\t\tctx->progress = start_delayed_progress(\n \t\t\t\t_(\"Finding commits for commit graph among packed objects\"),\n-\t\t\t\tapprox_nr_objects);\n-\t\tfor_each_packed_object(add_packed_commits, &oids,\n+\t\t\t\tctx->approx_nr_objects);\n+\t\tfor_each_packed_object(add_packed_commits, ctx,\n \t\t\t\t FOR_EACH_OBJECT_PACK_ORDER);\n-\t\tif (oids.progress_done < approx_nr_objects)\n-\t\t\tdisplay_progress(oids.progress, approx_nr_objects);\n-\t\tstop_progress(&oids.progress);\n+\t\tif (ctx->progress_done < ctx->approx_nr_objects)\n+\t\t\tdisplay_progress(ctx->progress, ctx->approx_nr_objects);\n+\t\tstop_progress(&ctx->progress);\n \t}\n \n-\tclose_reachable(&oids, report_progress);\n+\tclose_reachable(ctx);\n \n-\tif (report_progress)\n-\t\tprogress = start_delayed_progress(\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n \t\t\t_(\"Counting distinct commits in commit graph\"),\n-\t\t\toids.nr);\n-\tdisplay_progress(progress, 0); /* TODO: Measure QSORT() progress */\n-\tQSORT(oids.list, oids.nr, commit_compare);\n+\t\t\tctx->oids.nr);\n+\tdisplay_progress(ctx->progress, 0); /* TODO: Measure QSORT() progress */\n+\tQSORT(ctx->oids.list, ctx->oids.nr, commit_compare);\n \tcount_distinct = 1;\n-\tfor (i = 1; i < oids.nr; i++) {\n-\t\tdisplay_progress(progress, i + 1);\n-\t\tif (!oideq(&oids.list[i - 1], &oids.list[i]))\n+\tfor (i = 1; i < ctx->oids.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n \t\t\tcount_distinct++;\n \t}\n-\tstop_progress(&progress);\n+\tstop_progress(&ctx->progress);\n \n \tif (count_distinct >= GRAPH_EDGE_LAST_MASK) {\n \t\terror(_(\"the commit graph format cannot write %d commits\"), count_distinct);\n@@ -1024,54 +1021,54 @@ int write_commit_graph(const char *obj_dir,\n \t\tgoto cleanup;\n \t}\n \n-\tcommits.nr = 0;\n-\tcommits.alloc = count_distinct;\n-\tALLOC_ARRAY(commits.list, commits.alloc);\n+\tctx->commits.alloc = count_distinct;\n+\tALLOC_ARRAY(ctx->commits.list, ctx->commits.alloc);\n \n-\tnum_extra_edges = 0;\n-\tif (report_progress)\n-\t\tprogress = start_delayed_progress(\n+\tctx->num_extra_edges = 0;\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n \t\t\t_(\"Finding extra edges in commit graph\"),\n-\t\t\toids.nr);\n-\tfor (i = 0; i < oids.nr; i++) {\n+\t\t\tctx->oids.nr);\n+\tfor (i = 0; i < ctx->oids.nr; i++) {\n \t\tint num_parents = 0;\n-\t\tdisplay_progress(progress, i + 1);\n-\t\tif (i > 0 && oideq(&oids.list[i - 1], &oids.list[i]))\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\t\tif (i > 0 && oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n \t\t\tcontinue;\n \n-\t\tcommits.list[commits.nr] = lookup_commit(the_repository, &oids.list[i]);\n-\t\tparse_commit_no_graph(commits.list[commits.nr]);\n+\t\tctx->commits.list[ctx->commits.nr] = lookup_commit(ctx->r, &ctx->oids.list[i]);\n+\t\tparse_commit_no_graph(ctx->commits.list[ctx->commits.nr]);\n \n-\t\tfor (parent = commits.list[commits.nr]->parents;\n+\t\tfor (parent = ctx->commits.list[ctx->commits.nr]->parents;\n \t\t parent; parent = parent->next)\n \t\t\tnum_parents++;\n \n \t\tif (num_parents > 2)\n-\t\t\tnum_extra_edges += num_parents - 1;\n+\t\t\tctx->num_extra_edges += num_parents - 1;\n \n-\t\tcommits.nr++;\n+\t\tctx->commits.nr++;\n \t}\n-\tnum_chunks = num_extra_edges ? 4 : 3;\n-\tstop_progress(&progress);\n+\tstop_progress(&ctx->progress);\n \n-\tif (commits.nr >= GRAPH_EDGE_LAST_MASK) {\n+\tif (ctx->commits.nr >= GRAPH_EDGE_LAST_MASK) {\n \t\terror(_(\"too many commits to write graph\"));\n \t\tres = 1;\n \t\tgoto cleanup;\n \t}\n \n-\tcompute_generation_numbers(&commits, report_progress);\n+\tcompute_generation_numbers(ctx);\n \n-\tgraph_name = get_commit_graph_filename(obj_dir);\n-\tif (safe_create_leading_directories(graph_name)) {\n-\t\tUNLEAK(graph_name);\n+\tnum_chunks = ctx->num_extra_edges ? 4 : 3;\n+\n+\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n+\tif (safe_create_leading_directories(ctx->graph_name)) {\n+\t\tUNLEAK(ctx->graph_name);\n \t\terror(_(\"unable to create leading directories of %s\"),\n-\t\t\tgraph_name);\n+\t\t\tctx->graph_name);\n \t\tres = errno;\n \t\tgoto cleanup;\n \t}\n \n-\thold_lock_file_for_update(&lk, graph_name, LOCK_DIE_ON_ERROR);\n+\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \n \thashwrite_be32(f, GRAPH_SIGNATURE);\n@@ -1084,7 +1081,7 @@ int write_commit_graph(const char *obj_dir,\n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n \tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n-\tif (num_extra_edges)\n+\tif (ctx->num_extra_edges)\n \t\tchunk_ids[3] = GRAPH_CHUNKID_EXTRAEDGES;\n \telse\n \t\tchunk_ids[3] = 0;\n@@ -1092,9 +1089,9 @@ int write_commit_graph(const char *obj_dir,\n \n \tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n-\tchunk_offsets[2] = chunk_offsets[1] + hashsz * commits.nr;\n-\tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * commits.nr;\n-\tchunk_offsets[4] = chunk_offsets[3] + 4 * num_extra_edges;\n+\tchunk_offsets[2] = chunk_offsets[1] + hashsz * ctx->commits.nr;\n+\tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * ctx->commits.nr;\n+\tchunk_offsets[4] = chunk_offsets[3] + 4 * ctx->num_extra_edges;\n \n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tuint32_t chunk_write[3];\n@@ -1105,32 +1102,33 @@ int write_commit_graph(const char *obj_dir,\n \t\thashwrite(f, chunk_write, 12);\n \t}\n \n-\tif (report_progress) {\n+\tif (ctx->report_progress) {\n \t\tstrbuf_addf(&progress_title,\n \t\t\t Q_(\"Writing out commit graph in %d pass\",\n \t\t\t \"Writing out commit graph in %d passes\",\n \t\t\t num_chunks),\n \t\t\t num_chunks);\n-\t\tprogress = start_delayed_progress(\n+\t\tctx->progress = start_delayed_progress(\n \t\t\tprogress_title.buf,\n-\t\t\tnum_chunks * commits.nr);\n+\t\t\tnum_chunks * ctx->commits.nr);\n \t}\n-\twrite_graph_chunk_fanout(f, commits.list, commits.nr, progress, &progress_cnt);\n-\twrite_graph_chunk_oids(f, hashsz, commits.list, commits.nr, progress, &progress_cnt);\n-\twrite_graph_chunk_data(f, hashsz, commits.list, commits.nr, progress, &progress_cnt);\n-\tif (num_extra_edges)\n-\t\twrite_graph_chunk_extra_edges(f, commits.list, commits.nr, progress, &progress_cnt);\n-\tstop_progress(&progress);\n+\twrite_graph_chunk_fanout(f, ctx);\n+\twrite_graph_chunk_oids(f, hashsz, ctx);\n+\twrite_graph_chunk_data(f, hashsz, ctx);\n+\tif (ctx->num_extra_edges)\n+\t\twrite_graph_chunk_extra_edges(f, ctx);\n+\tstop_progress(&ctx->progress);\n \tstrbuf_release(&progress_title);\n \n-\tclose_commit_graph(the_repository);\n+\tclose_commit_graph(ctx->r);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n \tcommit_lock_file(&lk);\n \n cleanup:\n \tfree(graph_name);\n-\tfree(commits.list);\n-\tfree(oids.list);\n+\tfree(ctx->commits.list);\n+\tfree(ctx->oids.list);\n+\tfree(ctx);\n \n \treturn res;\n }\n-- \ngitgitgadget\n\n"},{"id":"375159","messageId":"4436c8f4f1ebd97b0709a030a85a46221e111cad.1557330827.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.git.gitgitgadget@gmail.com","subject":"[PATCH 14/17] commit-graph: load split commit-graph files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-08T15:53:59Z","receivedAt":"2019-05-08T15:54:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nStarting with commit-graph, load commit-graph files in a\nsequence as follows:\n\n commit-graph\n commit-graph-1\n commit-graph-2\n ...\n commit-graph-N\n\nThis creates N + 1 files in order.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 39 ++++++++++++++++++++++++++++++++++-----\n 1 file changed, 34 insertions(+), 5 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f790f44a9c..5f6193277a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -45,6 +45,12 @@ char *get_commit_graph_filename(const char *obj_dir)\n \treturn xstrfmt(\"%s/info/commit-graph\", obj_dir);\n }\n \n+static char *get_split_graph_filename(const char *obj_dir,\n+\t\t\t\t uint32_t split_count)\n+{\n+\treturn xstrfmt(\"%s/info/commit-graphs/commit-graph-%d\", obj_dir, split_count);\n+}\n+\n static uint8_t oid_version(void)\n {\n \treturn 1;\n@@ -289,15 +295,31 @@ static struct commit_graph *load_commit_graph_one(const char *graph_file)\n static void prepare_commit_graph_one(struct repository *r, const char *obj_dir)\n {\n \tchar *graph_name;\n+\tuint32_t split_count = 1;\n+\tstruct commit_graph *g;\n \n \tif (r->objects->commit_graph)\n \t\treturn;\n \n \tgraph_name = get_commit_graph_filename(obj_dir);\n-\tr->objects->commit_graph =\n-\t\tload_commit_graph_one(graph_name);\n-\n+\tg = load_commit_graph_one(graph_name);\n \tFREE_AND_NULL(graph_name);\n+\n+\twhile (g) {\n+\t\tg->base_graph = r->objects->commit_graph;\n+\n+\t\tif (g->base_graph)\n+\t\t\tg->num_commits_in_base = g->base_graph->num_commits +\n+\t\t\t\t\t\t g->base_graph->num_commits_in_base;;\n+\n+\t\tr->objects->commit_graph = g;\n+\n+\t\tgraph_name = get_split_graph_filename(obj_dir, split_count);\n+\t\tg = load_commit_graph_one(graph_name);\n+\t\tFREE_AND_NULL(graph_name);\n+\n+\t\tsplit_count++;\n+\t}\n }\n \n /*\n@@ -411,8 +433,15 @@ static struct commit_list **insert_parent_or_die(struct repository *r,\n \n static void fill_commit_graph_info(struct commit *item, struct commit_graph *g, uint32_t pos)\n {\n-\tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n-\titem->graph_pos = pos + g->num_commits_in_base;\n+\tconst unsigned char *commit_data;\n+\n+\tif (pos < g->num_commits_in_base) {\n+\t\tfill_commit_graph_info(item, g->base_graph, pos);\n+\t\treturn;\n+\t}\n+\n+\tcommit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * (pos - g->num_commits_in_base);\n+\titem->graph_pos = pos;\n \titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"375160","messageId":"3eee3667cf32c1049fa1f39264d36a33aa31d0f0.1557330827.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.git.gitgitgadget@gmail.com","subject":"[PATCH 11/17] commit-graph: extract write_commit_graph_file()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-08T15:53:56Z","receivedAt":"2019-05-08T15:54:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe write_commit_graph() method is too complex, so we are\nextracting methods one by one.\n\nExtract write_commit_graph_file() that takes all of the information\nin the context struct and writes the data to a commit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 155 +++++++++++++++++++++++++------------------------\n 1 file changed, 80 insertions(+), 75 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 16cdd7afb2..7723156964 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1015,21 +1015,91 @@ static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n-int write_commit_graph(const char *obj_dir,\n-\t\t struct string_list *pack_indexes,\n-\t\t struct string_list *commit_hex,\n-\t\t unsigned int flags)\n+static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n {\n-\tstruct write_commit_graph_context *ctx;\n+\tuint32_t i;\n \tstruct hashfile *f;\n-\tuint32_t i, count_distinct = 0;\n-\tchar *graph_name = NULL;\n \tstruct lock_file lk = LOCK_INIT;\n \tuint32_t chunk_ids[5];\n \tuint64_t chunk_offsets[5];\n-\tint num_chunks;\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n+\tint num_chunks = ctx->num_extra_edges ? 4 : 3;\n+\n+\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n+\tif (safe_create_leading_directories(ctx->graph_name)) {\n+\t\tUNLEAK(ctx->graph_name);\n+\t\terror(_(\"unable to create leading directories of %s\"),\n+\t\t\tctx->graph_name);\n+\t\treturn errno;\n+\t}\n+\n+\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n+\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n+\n+\thashwrite_be32(f, GRAPH_SIGNATURE);\n+\n+\thashwrite_u8(f, GRAPH_VERSION);\n+\thashwrite_u8(f, oid_version());\n+\thashwrite_u8(f, num_chunks);\n+\thashwrite_u8(f, 0); /* unused padding byte */\n+\n+\tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n+\tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n+\tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n+\tif (ctx->num_extra_edges)\n+\t\tchunk_ids[3] = GRAPH_CHUNKID_EXTRAEDGES;\n+\telse\n+\t\tchunk_ids[3] = 0;\n+\tchunk_ids[4] = 0;\n+\n+\tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n+\tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n+\tchunk_offsets[2] = chunk_offsets[1] + hashsz * ctx->commits.nr;\n+\tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * ctx->commits.nr;\n+\tchunk_offsets[4] = chunk_offsets[3] + 4 * ctx->num_extra_edges;\n+\n+\tfor (i = 0; i <= num_chunks; i++) {\n+\t\tuint32_t chunk_write[3];\n+\n+\t\tchunk_write[0] = htonl(chunk_ids[i]);\n+\t\tchunk_write[1] = htonl(chunk_offsets[i] >> 32);\n+\t\tchunk_write[2] = htonl(chunk_offsets[i] & 0xffffffff);\n+\t\thashwrite(f, chunk_write, 12);\n+\t}\n+\n+\tif (ctx->report_progress) {\n+\t\tstrbuf_addf(&progress_title,\n+\t\t\t Q_(\"Writing out commit graph in %d pass\",\n+\t\t\t \"Writing out commit graph in %d passes\",\n+\t\t\t num_chunks),\n+\t\t\t num_chunks);\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\tprogress_title.buf,\n+\t\t\tnum_chunks * ctx->commits.nr);\n+\t}\n+\twrite_graph_chunk_fanout(f, ctx);\n+\twrite_graph_chunk_oids(f, hashsz, ctx);\n+\twrite_graph_chunk_data(f, hashsz, ctx);\n+\tif (ctx->num_extra_edges)\n+\t\twrite_graph_chunk_extra_edges(f, ctx);\n+\tstop_progress(&ctx->progress);\n+\tstrbuf_release(&progress_title);\n+\n+\tclose_commit_graph(ctx->r);\n+\tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n+\tcommit_lock_file(&lk);\n+\n+\treturn 0;\n+}\n+\n+int write_commit_graph(const char *obj_dir,\n+\t\t struct string_list *pack_indexes,\n+\t\t struct string_list *commit_hex,\n+\t\t unsigned int flags)\n+{\n+\tstruct write_commit_graph_context *ctx;\n+\tuint32_t i, count_distinct = 0;\n \tint res = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n@@ -1096,75 +1166,10 @@ int write_commit_graph(const char *obj_dir,\n \n \tcompute_generation_numbers(ctx);\n \n-\tnum_chunks = ctx->num_extra_edges ? 4 : 3;\n-\n-\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n-\tif (safe_create_leading_directories(ctx->graph_name)) {\n-\t\tUNLEAK(ctx->graph_name);\n-\t\terror(_(\"unable to create leading directories of %s\"),\n-\t\t\tctx->graph_name);\n-\t\tres = errno;\n-\t\tgoto cleanup;\n-\t}\n-\n-\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n-\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n-\n-\thashwrite_be32(f, GRAPH_SIGNATURE);\n-\n-\thashwrite_u8(f, GRAPH_VERSION);\n-\thashwrite_u8(f, oid_version());\n-\thashwrite_u8(f, num_chunks);\n-\thashwrite_u8(f, 0); /* unused padding byte */\n-\n-\tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n-\tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n-\tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n-\tif (ctx->num_extra_edges)\n-\t\tchunk_ids[3] = GRAPH_CHUNKID_EXTRAEDGES;\n-\telse\n-\t\tchunk_ids[3] = 0;\n-\tchunk_ids[4] = 0;\n-\n-\tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n-\tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n-\tchunk_offsets[2] = chunk_offsets[1] + hashsz * ctx->commits.nr;\n-\tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * ctx->commits.nr;\n-\tchunk_offsets[4] = chunk_offsets[3] + 4 * ctx->num_extra_edges;\n-\n-\tfor (i = 0; i <= num_chunks; i++) {\n-\t\tuint32_t chunk_write[3];\n-\n-\t\tchunk_write[0] = htonl(chunk_ids[i]);\n-\t\tchunk_write[1] = htonl(chunk_offsets[i] >> 32);\n-\t\tchunk_write[2] = htonl(chunk_offsets[i] & 0xffffffff);\n-\t\thashwrite(f, chunk_write, 12);\n-\t}\n-\n-\tif (ctx->report_progress) {\n-\t\tstrbuf_addf(&progress_title,\n-\t\t\t Q_(\"Writing out commit graph in %d pass\",\n-\t\t\t \"Writing out commit graph in %d passes\",\n-\t\t\t num_chunks),\n-\t\t\t num_chunks);\n-\t\tctx->progress = start_delayed_progress(\n-\t\t\tprogress_title.buf,\n-\t\t\tnum_chunks * ctx->commits.nr);\n-\t}\n-\twrite_graph_chunk_fanout(f, ctx);\n-\twrite_graph_chunk_oids(f, hashsz, ctx);\n-\twrite_graph_chunk_data(f, hashsz, ctx);\n-\tif (ctx->num_extra_edges)\n-\t\twrite_graph_chunk_extra_edges(f, ctx);\n-\tstop_progress(&ctx->progress);\n-\tstrbuf_release(&progress_title);\n-\n-\tclose_commit_graph(ctx->r);\n-\tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n-\tcommit_lock_file(&lk);\n+\tres = write_commit_graph_file(ctx);\n \n cleanup:\n-\tfree(graph_name);\n+\tfree(ctx->graph_name);\n \tfree(ctx->commits.list);\n \tfree(ctx->oids.list);\n \tfree(ctx);\n-- \ngitgitgadget\n\n"},{"id":"375161","messageId":"9d0e966a3d47a0dff2cb5ec24f8225f5f63c6e8f.1557330827.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.git.gitgitgadget@gmail.com","subject":"[PATCH 13/17] commit-graph: lay groundwork for incremental files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-08T15:53:58Z","receivedAt":"2019-05-08T15:54:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 66 ++++++++++++++++++++++++++++++++++++++++++++------\n commit-graph.h | 4 +++\n 2 files changed, 62 insertions(+), 8 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 7723156964..f790f44a9c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -371,6 +371,25 @@ static int bsearch_graph(struct commit_graph *g, struct object_id *oid, uint32_t\n \t\t\t g->chunk_oid_lookup, g->hash_len, pos);\n }\n \n+static void load_oid_from_graph(struct commit_graph *g, int pos, struct object_id *oid)\n+{\n+\tif (!g)\n+\t\tBUG(\"NULL commit-graph\");\n+\n+\tif (pos < g->num_commits_in_base) {\n+\t\tload_oid_from_graph(g->base_graph, pos, oid);\n+\t\treturn;\n+\t}\n+\n+\tif (pos >= g->num_commits + g->num_commits_in_base)\n+\t\tBUG(\"position %d is beyond the scope of this commit-graph (%d local + %d base commits)\",\n+\t\t pos, g->num_commits, g->num_commits_in_base);\n+\n+\tpos -= g->num_commits_in_base;\n+\n+\thashcpy(oid->hash, g->chunk_oid_lookup + g->hash_len * pos);\n+}\n+\n static struct commit_list **insert_parent_or_die(struct repository *r,\n \t\t\t\t\t\t struct commit_graph *g,\n \t\t\t\t\t\t uint64_t pos,\n@@ -379,10 +398,10 @@ static struct commit_list **insert_parent_or_die(struct repository *r,\n \tstruct commit *c;\n \tstruct object_id oid;\n \n-\tif (pos >= g->num_commits)\n+\tif (pos >= g->num_commits + g->num_commits_in_base)\n \t\tdie(\"invalid parent position %\"PRIu64, pos);\n \n-\thashcpy(oid.hash, g->chunk_oid_lookup + g->hash_len * pos);\n+\tload_oid_from_graph(g, pos, &oid);\n \tc = lookup_commit(r, &oid);\n \tif (!c)\n \t\tdie(_(\"could not find commit %s\"), oid_to_hex(&oid));\n@@ -393,7 +412,7 @@ static struct commit_list **insert_parent_or_die(struct repository *r,\n static void fill_commit_graph_info(struct commit *item, struct commit_graph *g, uint32_t pos)\n {\n \tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n-\titem->graph_pos = pos;\n+\titem->graph_pos = pos + g->num_commits_in_base;\n \titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n }\n \n@@ -405,10 +424,25 @@ static int fill_commit_in_graph(struct repository *r,\n \tuint32_t *parent_data_ptr;\n \tuint64_t date_low, date_high;\n \tstruct commit_list **pptr;\n-\tconst unsigned char *commit_data = g->chunk_commit_data + (g->hash_len + 16) * pos;\n+\tconst unsigned char *commit_data;\n \n-\titem->object.parsed = 1;\n+\tif (pos < g->num_commits_in_base)\n+\t\treturn fill_commit_in_graph(r, item, g->base_graph, pos);\n+\n+\tif (pos >= g->num_commits + g->num_commits_in_base)\n+\t\tBUG(\"position %d is beyond the scope of this commit-graph (%d local + %d base commits)\",\n+\t\t pos, g->num_commits, g->num_commits_in_base);\n+\n+\t/*\n+\t * Store the \"full\" position, but then use the\n+\t * \"local\" position for the rest of the calculation.\n+\t */\n \titem->graph_pos = pos;\n+\tpos -= g->num_commits_in_base;\n+\n+\tcommit_data = g->chunk_commit_data + (g->hash_len + 16) * pos;\n+\n+\titem->object.parsed = 1;\n \n \titem->maybe_tree = NULL;\n \n@@ -452,7 +486,18 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \t\t*pos = item->graph_pos;\n \t\treturn 1;\n \t} else {\n-\t\treturn bsearch_graph(g, &(item->object.oid), pos);\n+\t\tstruct commit_graph *cur_g = g;\n+\t\tuint32_t pos_in_g;\n+\n+\t\twhile (cur_g && !bsearch_graph(cur_g, &(item->object.oid), &pos_in_g))\n+\t\t\tcur_g = cur_g->base_graph;\n+\n+\t\tif (cur_g) {\n+\t\t\t*pos = pos_in_g + cur_g->num_commits_in_base;\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\treturn 0;\n \t}\n }\n \n@@ -492,8 +537,13 @@ static struct tree *load_tree_for_commit(struct repository *r,\n \t\t\t\t\t struct commit *c)\n {\n \tstruct object_id oid;\n-\tconst unsigned char *commit_data = g->chunk_commit_data +\n-\t\t\t\t\t GRAPH_DATA_WIDTH * (c->graph_pos);\n+\tconst unsigned char *commit_data;\n+\n+\tif (c->graph_pos < g->num_commits_in_base)\n+\t\treturn load_tree_for_commit(r, g->base_graph, c);\n+\n+\tcommit_data = g->chunk_commit_data +\n+\t\t\tGRAPH_DATA_WIDTH * (c->graph_pos - g->num_commits_in_base);\n \n \thashcpy(oid.hash, commit_data);\n \tc->maybe_tree = lookup_tree(r, &oid);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 70f4caf0c7..170920720d 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -48,10 +48,14 @@ struct commit_graph {\n \tuint32_t num_commits;\n \tstruct object_id oid;\n \n+\tuint32_t num_commits_in_base;\n+\tstruct commit_graph *base_graph;\n+\n \tconst uint32_t *chunk_oid_fanout;\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_commit_data;\n \tconst unsigned char *chunk_extra_edges;\n+\tconst unsigned char *chunk_base_graph;\n };\n \n struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st);\n-- \ngitgitgadget\n\n"},{"id":"375162","messageId":"7bbe8d9150a623ea684c94d129eda1607dd32a79.1557330827.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.git.gitgitgadget@gmail.com","subject":"[PATCH 12/17] Documentation: describe split commit-graphs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-08T15:53:57Z","receivedAt":"2019-05-08T15:54:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe design for the split commit-graphs uses file names to force\na \"stack\" of commit-graph files. This allows incremental writes\nwithout updating the file format.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 142 +++++++++++++++++++++++\n 1 file changed, 142 insertions(+)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex fb53341d5e..ca1661d2d8 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -127,6 +127,148 @@ Design Details\n helpful for these clones, anyway. The commit-graph will not be read or\n written when shallow commits are present.\n \n+Split Commit Graphs\n+-------------------\n+\n+Typically, repos grow with near-constant velocity (commits per day). Over\n+time, the number of commits added by a fetch operation is much smaller than\n+the number of commits in the full history. The split commit-graph feature\n+allows for fast writes of new commit data without rewriting the entire\n+commit history -- at least, most of the time.\n+\n+## File Layout\n+\n+A split commit-graph uses multiple files, and we use a fixed naming\n+convention to organize these files. The base commit-graph file is the\n+same: `$OBJDIR/info/commit-graph`. The rest of the commit-graph files have\n+the format `$OBJDIR/info/commit-graphs/commit-graph-` where N is a\n+positive integer. The integers must start at 1 and grow sequentially\n+to form a stack of files.\n+\n+Each `commit-graph-` file has the same format as the `commit-graph`\n+file, including a lexicographic list of commit ids. The only difference\n+is that this list is considered to be concatenated to the list from\n+the lower commit-graphs. As an example, consider this diagram of three\n+files:\n+\n+ +-----------------------+\n+ | commit-graph-2 |\n+ +-----------------------+\n+\t |\n+ +-----------------------+\n+ | |\n+ | commit-graph-1 |\n+ | |\n+ +-----------------------+\n+\t |\n+ +-----------------------+\n+ | |\n+ | |\n+ | |\n+ | commit-graph |\n+ | |\n+ | |\n+ | |\n+ +-----------------------+\n+\n+Let X0 be the number of commits in `commit-graph`, X1 be the number\n+of commits in commit-graph-1, and X2 be the number of commits in\n+commit-graph-2. If a commit appears in position i in `commit-graph-2`,\n+then we interpret this as being the commit in position (X0 + X1 + i),\n+and that will be used as its \"graph position\". The commits in\n+commit-graph-2 use these positions to refer to their parents, which\n+may be in commit-graph-1 or commit-graph. We can navigate to an\n+arbitrary commit in position j by checking its containment in the\n+intervals [0, X0), [X0, X0 + X1), [X0 + X1, X0 + X1 + X2).\n+\n+When Git reads from these files, it starts by acquiring a read handle\n+on the `commit-graph` file. On success, it continues acquiring read\n+handles on the `commit-graph-` files in increasing order. This\n+order is important for how we replace the files.\n+\n+## Merging commit-graph files\n+\n+If we only added a `commit-graph-` file on every write, we would\n+run into a linear search problem through many commit-graph files.\n+Instead, we use a merge strategy to decide when the stack should\n+collapse some number of levels.\n+\n+The diagram below shows such a collapse. As a set of new commits\n+are added, it is determined by the merge strategy that the files\n+should collapse to `commit-graph-1`. Thus, the new commits, the\n+commits in `commit-graph-2` and the commits in `commit-graph-1`\n+should be combined into a new `commit-graph-1` file.\n+\n+\t\t\t +---------------------+\n+\t\t\t | |\n+\t\t\t | (new commits) |\n+\t\t\t | |\n+\t\t\t +---------------------+\n+\t\t\t | |\n+ +-----------------------+ +---------------------+\n+ | commit-graph-2 |->| |\n+ +-----------------------+ +---------------------+\n+\t | | |\n+ +-----------------------+ +---------------------+\n+ | | | |\n+ | commit-graph-1 |->| |\n+ | | | |\n+ +-----------------------+ +---------------------+\n+\t | commit-graph-1.lock\n+ +-----------------------+\n+ | |\n+ | |\n+ | |\n+ | commit-graph |\n+ | |\n+ | |\n+ | |\n+ +-----------------------+\n+\n+During this process, the commits to write are combined, sorted\n+and we write the contents to the `commit-graph-1.lock` file.\n+When the file is flushed and ready to swap to `commit-graph-1`,\n+we first unlink the files above our target file. This unlinking\n+is done from the top of the stack, the reverse direction that\n+another process would use to read the stack.\n+\n+During this time window, another process trying to read the\n+commit-graph stack could read `commit-graph-1` before the swap\n+but try to read `commit-graph-2` after it is unlinked. That\n+process would then believe that this stack is complete, but\n+will miss out on the performance benefits of the commits in\n+`commit-graph-2`. For this reason, the stack above the\n+`commit-graph` file should be small.\n+\n+## Merge Strategy\n+\n+When writing a set of commits that do not exist in the\n+commit-graph stack of height N, we default to creating\n+a new file at level N + 1. We then decide to merge\n+with the Nth level if one of two conditions hold:\n+\n+ 1. The expected file size for level N + 1 is at\n+ least half the file size for level N.\n+\n+ 2. Level N + 1 contains more than MAX_SPLIT_COMMITS\n+ commits (64,0000 commits).\n+\n+This decision cascades down the levels: when we\n+merge a level we create a new set of commits that\n+then compares to the next level.\n+\n+The first condition bounds the number of levels\n+to be logarithmic in the total number of commits.\n+The second condition bounds the total number of\n+commits in a `commit-graph-N` file and not in\n+the `commit-graph` file, preventing significant\n+performance issues when the stack merges and another\n+process only partially reads the previous stack.\n+\n+The merge strategy values (2 for the size multiple,\n+64,000 for the maximum number of commits) could be\n+extracted into config settings for full flexibility.\n+\n Related Links\n -------------\n [0] https://bugs.chromium.org/p/git/issues/detail?id=8\n-- \ngitgitgadget\n\n"},{"id":"375163","messageId":"7c5bc06d14b6c500c3e08f3366da4e1d8ca6133f.1557330827.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.git.gitgitgadget@gmail.com","subject":"[PATCH 16/17] commit-graph: add --split option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-08T15:54:01Z","receivedAt":"2019-05-08T15:54:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 10 +++++++---\n t/t5318-commit-graph.sh | 26 ++++++++++++++++++++++++++\n 2 files changed, 33 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 828b1a713f..c2c07d3917 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -10,7 +10,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph verify [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -25,7 +25,7 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -35,9 +35,9 @@ static struct opts_commit_graph {\n \tint stdin_packs;\n \tint stdin_commits;\n \tint append;\n+\tint split;\n } opts;\n \n-\n static int graph_verify(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n@@ -156,6 +156,8 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"start walk at commits listed by stdin\")),\n \t\tOPT_BOOL(0, \"append\", &opts.append,\n \t\t\tN_(\"include all commits already in the commit-graph file\")),\n+\t\tOPT_BOOL(0, \"split\", &opts.split,\n+\t\t\tN_(\"allow writing an incremental commit-graph file\")),\n \t\tOPT_END(),\n \t};\n \n@@ -169,6 +171,8 @@ static int graph_write(int argc, const char **argv)\n \t\topts.obj_dir = get_object_directory();\n \tif (opts.append)\n \t\tflags |= COMMIT_GRAPH_APPEND;\n+\tif (opts.split)\n+\t\tflags |= COMMIT_GRAPH_SPLIT;\n \n \tread_replace_refs = 0;\n \ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex d621608500..9dfd4cc9b1 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -260,6 +260,32 @@ test_expect_success 'check that gc computes commit-graph' '\n \ttest_cmp_bin commit-graph-after-gc $objdir/info/commit-graph\n '\n \n+test_expect_success 'write split commit-graph' '\n+\tcd \"$TRASH_DIRECTORY\" &&\n+\tgit clone full split &&\n+\tcd split &&\n+\tgit config core.commitGraph true &&\n+\tfor i in $(test_seq 1 20); do\n+\t\ttest_commit padding-$i\n+\tdone &&\n+\tgit commit-graph write --reachable &&\n+\ttest_commit split-commit &&\n+\tgit branch -f split-commit &&\n+\tgit commit-graph write --reachable --split &&\n+\ttest_path_is_file .git/objects/info/commit-graphs/commit-graph-1\n+'\n+\n+graph_git_behavior 'split graph, split-commit vs merge 1' bare split-commit merge/1\n+\n+test_expect_success 'collapse split commit-graph' '\n+\tcd \"$TRASH_DIRECTORY/split\" &&\n+\tgit commit-graph write --reachable &&\n+\ttest_path_is_missing .git/objects/info/commit-graphs/commit-graph-1 &&\n+\ttest_path_is_file .git/objects/info/commit-graph\n+'\n+\n+graph_git_behavior 'collapsed graph, split-commit vs merge 1' bare split-commit merge/1\n+\n test_expect_success 'replace-objects invalidates commit-graph' '\n \tcd \"$TRASH_DIRECTORY\" &&\n \ttest_when_finished rm -rf replace &&\n-- \ngitgitgadget\n\n"},{"id":"375164","messageId":"aa4a096813aaa3796cef391f866cc4a6ae99a6bf.1557330827.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.git.gitgitgadget@gmail.com","subject":"[PATCH 15/17] commit-graph: write split commit-graph files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-08T15:54:00Z","receivedAt":"2019-05-08T15:54:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 248 +++++++++++++++++++++++++++++++++++++++-\n commit-graph.h | 1 +\n t/t5318-commit-graph.sh | 2 +-\n 3 files changed, 244 insertions(+), 7 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 5f6193277a..44448aabe4 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -40,6 +40,8 @@\n #define GRAPH_MIN_SIZE (GRAPH_HEADER_SIZE + 4 * GRAPH_CHUNKLOOKUP_WIDTH \\\n \t\t\t+ GRAPH_FANOUT_SIZE + the_hash_algo->rawsz)\n \n+#define MAX_SPLIT_COMMITS 64000\n+\n char *get_commit_graph_filename(const char *obj_dir)\n {\n \treturn xstrfmt(\"%s/info/commit-graph\", obj_dir);\n@@ -619,9 +621,14 @@ struct write_commit_graph_context {\n \tunsigned long approx_nr_objects;\n \tstruct progress *progress;\n \tint progress_done;\n+\tint num_commit_graphs_before;\n+\tint num_commit_graphs_after;\n+\tuint32_t new_num_commits_in_base;\n+\tstruct commit_graph *new_base_graph;\n \tuint64_t progress_cnt;\n \tunsigned append:1,\n-\t\t report_progress:1;\n+\t\t report_progress:1,\n+\t\t split:1;\n };\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n@@ -691,6 +698,16 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t commit_to_sha1);\n \n+\t\t\tif (edge_value >= 0)\n+\t\t\t\tedge_value += ctx->new_num_commits_in_base;\n+\t\t\telse {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (find_commit_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n+\t\t\t\t\tedge_value = pos;\n+\t\t\t}\n+\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n \t\t\t\t oid_to_hex(&parent->item->object.oid),\n@@ -711,6 +728,17 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\t\t\t ctx->commits.list,\n \t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t commit_to_sha1);\n+\n+\t\t\tif (edge_value >= 0)\n+\t\t\t\tedge_value += ctx->new_num_commits_in_base;\n+\t\t\telse {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (find_commit_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n+\t\t\t\t\tedge_value = pos;\n+\t\t\t}\n+\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n \t\t\t\t oid_to_hex(&parent->item->object.oid),\n@@ -768,6 +796,16 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n \t\t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t\t commit_to_sha1);\n \n+\t\t\tif (edge_value >= 0)\n+\t\t\t\tedge_value += ctx->new_num_commits_in_base;\n+\t\t\telse {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (find_commit_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n+\t\t\t\t\tedge_value = pos;\n+\t\t\t}\n+\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n \t\t\t\t oid_to_hex(&parent->item->object.oid),\n@@ -782,7 +820,7 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n \t}\n }\n \n-static int commit_compare(const void *_a, const void *_b)\n+static int oid_compare(const void *_a, const void *_b)\n {\n \tconst struct object_id *a = (const struct object_id *)_a;\n \tconst struct object_id *b = (const struct object_id *)_b;\n@@ -859,7 +897,13 @@ static void close_reachable(struct write_commit_graph_context *ctx)\n \t\tdisplay_progress(ctx->progress, i + 1);\n \t\tcommit = lookup_commit(ctx->r, &ctx->oids.list[i]);\n \n-\t\tif (commit && !parse_commit_no_graph(commit))\n+\t\tif (!commit)\n+\t\t\tcontinue;\n+\t\tif (ctx->split) {\n+\t\t\tif (!parse_commit(commit) &&\n+\t\t\t commit->graph_pos == COMMIT_NOT_FROM_GRAPH)\n+\t\t\t\tadd_missing_parents(ctx, commit);\n+\t\t} else if (!parse_commit_no_graph(commit))\n \t\t\tadd_missing_parents(ctx, commit);\n \t}\n \tstop_progress(&ctx->progress);\n@@ -1051,12 +1095,20 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n \t\t\t_(\"Counting distinct commits in commit graph\"),\n \t\t\tctx->oids.nr);\n \tdisplay_progress(ctx->progress, 0); /* TODO: Measure QSORT() progress */\n-\tQSORT(ctx->oids.list, ctx->oids.nr, commit_compare);\n+\tQSORT(ctx->oids.list, ctx->oids.nr, oid_compare);\n \n \tfor (i = 1; i < ctx->oids.nr; i++) {\n \t\tdisplay_progress(ctx->progress, i + 1);\n-\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n+\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i])) {\n+\t\t\tif (ctx->split) {\n+\t\t\t\tstruct commit *c = lookup_commit(ctx->r, &ctx->oids.list[i]);\n+\n+\t\t\t\tif (!c || c->graph_pos != COMMIT_NOT_FROM_GRAPH)\n+\t\t\t\t\tcontinue;\n+\t\t\t}\n+\n \t\t\tcount_distinct++;\n+\t\t}\n \t}\n \tstop_progress(&ctx->progress);\n \n@@ -1079,7 +1131,13 @@ static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n \t\tif (i > 0 && oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n \t\t\tcontinue;\n \n+\t\tALLOC_GROW(ctx->commits.list, ctx->commits.nr + 1, ctx->commits.alloc);\n \t\tctx->commits.list[ctx->commits.nr] = lookup_commit(ctx->r, &ctx->oids.list[i]);\n+\n+\t\tif (ctx->split &&\n+\t\t ctx->commits.list[ctx->commits.nr]->graph_pos != COMMIT_NOT_FROM_GRAPH)\n+\t\t\tcontinue;\n+\n \t\tparse_commit_no_graph(ctx->commits.list[ctx->commits.nr]);\n \n \t\tfor (parent = ctx->commits.list[ctx->commits.nr]->parents;\n@@ -1105,7 +1163,13 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tint num_chunks = ctx->num_extra_edges ? 4 : 3;\n \n-\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n+\tif (ctx->num_commit_graphs_after > 1)\n+\t\tctx->graph_name = get_split_graph_filename(\n+\t\t\t\t\tctx->obj_dir,\n+\t\t\t\t\tctx->num_commit_graphs_after - 1);\n+\telse\n+\t\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n+\n \tif (safe_create_leading_directories(ctx->graph_name)) {\n \t\tUNLEAK(ctx->graph_name);\n \t\terror(_(\"unable to create leading directories of %s\"),\n@@ -1167,11 +1231,166 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \n \tclose_commit_graph(ctx->r);\n \tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n+\n+\twhile (ctx->num_commit_graphs_before > ctx->num_commit_graphs_after) {\n+\t\tchar *graph_name = get_split_graph_filename(\n+\t\t\t\t\tctx->obj_dir,\n+\t\t\t\t\t--ctx->num_commit_graphs_before);\n+\t\tunlink(graph_name);\n+\t\tfree(graph_name);\n+\t}\n+\n \tcommit_lock_file(&lk);\n \n \treturn 0;\n }\n \n+static size_t expected_commit_graph_size(size_t num_commits)\n+{\n+\treturn GRAPH_HEADER_SIZE + GRAPH_FANOUT_SIZE + 6 * GRAPH_CHUNKLOOKUP_WIDTH +\n+\t\t(num_commits + 1) * (GRAPH_DATA_WIDTH + the_hash_algo->rawsz);\n+}\n+\n+static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n+{\n+\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\tuint32_t num_commits = ctx->commits.nr;\n+\tsize_t expected_size = expected_commit_graph_size(num_commits);\n+\n+\tctx->num_commit_graphs_before = 0;\n+\twhile (g) {\n+\t\tctx->num_commit_graphs_before++;\n+\t\tg = g->base_graph;\n+\t}\n+\n+\tg = ctx->r->objects->commit_graph;\n+\tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n+\n+\twhile (g && (g->data_len <= 2 * expected_size || num_commits > MAX_SPLIT_COMMITS)) {\n+\t\tnum_commits += g->num_commits;\n+\t\texpected_size = expected_commit_graph_size(num_commits);\n+\t\tg = g->base_graph;\n+\t\tctx->num_commit_graphs_after--;\n+\t}\n+}\n+\n+static void merge_commit_graph(struct write_commit_graph_context *ctx,\n+\t\t\t struct commit_graph *g)\n+{\n+\tuint32_t i;\n+\tuint32_t offset = g->num_commits_in_base;\n+\n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct object_id oid;\n+\t\tstruct commit *result;\n+\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\n+\t\tload_oid_from_graph(g, i + offset, &oid);\n+\n+\t\t/* only add commits if they still exist in the repo */\n+\t\tresult = lookup_commit_reference_gently(ctx->r, &oid, 1);\n+\n+\t\tif (result) {\n+\t\t\tALLOC_GROW(ctx->commits.list, ctx->commits.nr + 1, ctx->commits.alloc);\n+\t\t\tctx->commits.list[ctx->commits.nr] = result;\n+\t\t\tctx->commits.nr++;\n+\t\t}\n+\t}\n+}\n+\n+static int commit_compare(const void *_a, const void *_b)\n+{\n+\tconst struct commit *a = *(const struct commit **)_a;\n+\tconst struct commit *b = *(const struct commit **)_b;\n+\treturn oidcmp(&a->object.oid, &b->object.oid);\n+}\n+\n+static void deduplicate_commits(struct write_commit_graph_context *ctx)\n+{\n+\tuint32_t i, num_parents, last_distinct = 0, duplicates = 0;\n+\tstruct commit_list *parent;\n+\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"De-duplicating merged commits\"),\n+\t\t\t\t\tctx->commits.nr);\n+\n+\tQSORT(ctx->commits.list, ctx->commits.nr, commit_compare);\n+\n+\tctx->num_extra_edges = 0;\n+\tfor (i = 1; i < ctx->commits.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i);\n+\n+\t\tif (oideq(&ctx->commits.list[last_distinct]->object.oid,\n+\t\t\t &ctx->commits.list[i]->object.oid)) {\n+\t\t\tduplicates++;\n+\t\t} else {\n+\t\t\tif (duplicates)\n+\t\t\t\tctx->commits.list[last_distinct + 1] = ctx->commits.list[i];\n+\t\t\tlast_distinct++;\n+\n+\t\t\tnum_parents = 0;\n+\t\t\tfor (parent = ctx->commits.list[i]->parents; parent; parent = parent->next)\n+\t\t\t\tnum_parents++;\n+\n+\t\t\tif (num_parents > 2)\n+\t\t\t\tctx->num_extra_edges += num_parents - 2;\n+\t\t}\n+\t}\n+\n+\tctx->commits.nr -= duplicates;\n+\tstop_progress(&ctx->progress);\n+}\n+\n+static void merge_commit_graphs(struct write_commit_graph_context *ctx)\n+{\n+\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\tuint32_t current_graph_number = ctx->num_commit_graphs_before;\n+\tstruct strbuf progress_title = STRBUF_INIT;\n+\n+\twhile (g && current_graph_number >= ctx->num_commit_graphs_after) {\n+\t\tcurrent_graph_number--;\n+\n+\t\tif (ctx->report_progress) {\n+\t\t\tif (current_graph_number)\n+\t\t\t\tstrbuf_addf(&progress_title,\n+\t\t\t\t\t _(\"Merging commit-graph-%d\"),\n+\t\t\t\t\t current_graph_number);\n+\t\t\telse\n+\t\t\t\tstrbuf_addstr(&progress_title,\n+\t\t\t\t\t _(\"Merging commit-graph\"));\n+\t\t\tctx->progress = start_delayed_progress(progress_title.buf, 0);\n+\t\t}\n+\n+\t\tmerge_commit_graph(ctx, g);\n+\t\tstop_progress(&ctx->progress);\n+\t\tstrbuf_release(&progress_title);\n+\n+\t\tg = g->base_graph;\n+\t}\n+\n+\tif (g) {\n+\t\tctx->new_base_graph = g;\n+\t\tctx->new_num_commits_in_base = g->num_commits + g->num_commits_in_base;\n+\t}\n+\n+\tdeduplicate_commits(ctx);\n+}\n+\n+static void collapse_all_commit_graphs(struct write_commit_graph_context *ctx)\n+{\n+\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\n+\tctx->num_commit_graphs_after = 1;\n+\tctx->num_commit_graphs_before = 0;\n+\n+\twhile (g) {\n+\t\tctx->num_commit_graphs_before++;\n+\t\tg = g->base_graph;\n+\t}\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -1189,10 +1408,17 @@ int write_commit_graph(const char *obj_dir,\n \tctx->obj_dir = obj_dir;\n \tctx->append = flags & COMMIT_GRAPH_APPEND ? 1 : 0;\n \tctx->report_progress = flags & COMMIT_GRAPH_PROGRESS ? 1 : 0;\n+\tctx->split = flags & COMMIT_GRAPH_SPLIT ? 1 : 0;\n+\n+\tif (ctx->split)\n+\t\tprepare_commit_graph(ctx->r);\n \n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n \n+\tif (ctx->split && ctx->oids.alloc > MAX_SPLIT_COMMITS)\n+\t\tctx->oids.alloc = MAX_SPLIT_COMMITS;\n+\n \tif (ctx->append) {\n \t\tprepare_commit_graph_one(ctx->r, ctx->obj_dir);\n \t\tif (ctx->r->objects->commit_graph)\n@@ -1243,6 +1469,16 @@ int write_commit_graph(const char *obj_dir,\n \t\tgoto cleanup;\n \t}\n \n+\tif (!ctx->commits.nr)\n+\t\tgoto cleanup;\n+\n+\tif (ctx->split) {\n+\t\tsplit_graph_merge_strategy(ctx);\n+\n+\t\tmerge_commit_graphs(ctx);\n+\t} else\n+\t\tcollapse_all_commit_graphs(ctx);\n+\n \tcompute_generation_numbers(ctx);\n \n \tres = write_commit_graph_file(ctx);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 170920720d..7a39ae2278 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -71,6 +71,7 @@ int generation_numbers_enabled(struct repository *r);\n \n #define COMMIT_GRAPH_APPEND (1 << 0)\n #define COMMIT_GRAPH_PROGRESS (1 << 1)\n+#define COMMIT_GRAPH_SPLIT (1 << 2)\n \n int write_commit_graph_reachable(const char *obj_dir, unsigned int flags);\n int write_commit_graph(const char *obj_dir,\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex e80c1cac02..d621608500 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -20,7 +20,7 @@ test_expect_success 'verify graph with no graph file' '\n test_expect_success 'write graph with no packs' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write --object-dir . &&\n-\ttest_path_is_file info/commit-graph\n+\ttest_path_is_missing info/commit-graph\n '\n \n test_expect_success 'create commits and repack' '\n-- \ngitgitgadget\n\n"},{"id":"375165","messageId":"3c52385e5696887c40cab4a6b9b7923d60a0567c.1557330827.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.git.gitgitgadget@gmail.com","subject":"[PATCH 17/17] fetch: add fetch.writeCommitGraph config setting","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-08T15:54:02Z","receivedAt":"2019-05-08T15:54:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSigned-off-by: Derrick Stolee \n---\n builtin/fetch.c | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex b620fd54b4..cf0944bad5 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -23,6 +23,7 @@\n #include \"packfile.h\"\n #include \"list-objects-filter-options.h\"\n #include \"commit-reach.h\"\n+#include \"commit-graph.h\"\n \n static const char * const builtin_fetch_usage[] = {\n \tN_(\"git fetch [] [ [...]]\"),\n@@ -62,6 +63,7 @@ static const char *submodule_prefix = \"\";\n static int recurse_submodules = RECURSE_SUBMODULES_DEFAULT;\n static int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;\n static int shown_url = 0;\n+static int fetch_write_commit_graph = 0;\n static struct refspec refmap = REFSPEC_INIT_FETCH;\n static struct list_objects_filter_options filter_options;\n static struct string_list server_options = STRING_LIST_INIT_DUP;\n@@ -79,6 +81,11 @@ static int git_fetch_config(const char *k, const char *v, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(k, \"fetch.writecommitgraph\")) {\n+\t\tfetch_write_commit_graph = 1;\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(k, \"submodule.recurse\")) {\n \t\tint r = git_config_bool(k, v) ?\n \t\t\tRECURSE_SUBMODULES_ON : RECURSE_SUBMODULES_OFF;\n@@ -1670,6 +1677,16 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \n \tstring_list_clear(&list, 0);\n \n+\tif (fetch_write_commit_graph) {\n+\t\tint commit_graph_flags = COMMIT_GRAPH_SPLIT;\n+\n+\t\tif (progress)\n+\t\t\tcommit_graph_flags |= COMMIT_GRAPH_PROGRESS;\n+\n+\t\twrite_commit_graph_reachable(get_object_directory(),\n+\t\t\t\t\t commit_graph_flags);\n+\t}\n+\n \tclose_all_packs(the_repository->objects);\n \n \targv_array_pushl(&argv_gc_auto, \"gc\", \"--auto\", NULL);\n-- \ngitgitgadget\n"},{"id":"375170","messageId":"20190508172029.GP14763@szeder.dev","threadId":"51052","inReplyTo":"7bbe8d9150a623ea684c94d129eda1607dd32a79.1557330827.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 12/17] Documentation: describe split commit-graphs","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-05-08T17:20:29Z","receivedAt":"2019-05-08T17:20:36Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, May 08, 2019 at 08:53:57AM -0700, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The design for the split commit-graphs uses file names to force\n> a \"stack\" of commit-graph files. This allows incremental writes\n> without updating the file format.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/technical/commit-graph.txt | 142 +++++++++++++++++++++++\n> 1 file changed, 142 insertions(+)\n> \n> diff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\n> index fb53341d5e..ca1661d2d8 100644\n> --- a/Documentation/technical/commit-graph.txt\n> +++ b/Documentation/technical/commit-graph.txt\n> @@ -127,6 +127,148 @@ Design Details\n> helpful for these clones, anyway. The commit-graph will not be read or\n> written when shallow commits are present.\n> \n> +Split Commit Graphs\n> +-------------------\n> +\n> +Typically, repos grow with near-constant velocity (commits per day). Over\n> +time, the number of commits added by a fetch operation is much smaller than\n> +the number of commits in the full history. The split commit-graph feature\n> +allows for fast writes of new commit data without rewriting the entire\n> +commit history -- at least, most of the time.\n> +\n> +## File Layout\n> +\n> +A split commit-graph uses multiple files, and we use a fixed naming\n> +convention to organize these files. The base commit-graph file is the\n> +same: `$OBJDIR/info/commit-graph`. The rest of the commit-graph files have\n> +the format `$OBJDIR/info/commit-graphs/commit-graph-` where N is a\n> +positive integer. The integers must start at 1 and grow sequentially\n> +to form a stack of files.\n> +\n> +Each `commit-graph-` file has the same format as the `commit-graph`\n> +file, including a lexicographic list of commit ids. The only difference\n> +is that this list is considered to be concatenated to the list from\n> +the lower commit-graphs. As an example, consider this diagram of three\n> +files:\n> +\n> + +-----------------------+\n> + | commit-graph-2 |\n> + +-----------------------+\n> +\t |\n> + +-----------------------+\n> + | |\n> + | commit-graph-1 |\n> + | |\n> + +-----------------------+\n> +\t |\n> + +-----------------------+\n> + | |\n> + | |\n> + | |\n> + | commit-graph |\n> + | |\n> + | |\n> + | |\n> + +-----------------------+\n> +\n> +Let X0 be the number of commits in `commit-graph`, X1 be the number\n> +of commits in commit-graph-1, and X2 be the number of commits in\n> +commit-graph-2. If a commit appears in position i in `commit-graph-2`,\n> +then we interpret this as being the commit in position (X0 + X1 + i),\n> +and that will be used as its \"graph position\". The commits in\n> +commit-graph-2 use these positions to refer to their parents, which\n> +may be in commit-graph-1 or commit-graph. We can navigate to an\n> +arbitrary commit in position j by checking its containment in the\n> +intervals [0, X0), [X0, X0 + X1), [X0 + X1, X0 + X1 + X2).\n> +\n> +When Git reads from these files, it starts by acquiring a read handle\n> +on the `commit-graph` file. On success, it continues acquiring read\n> +handles on the `commit-graph-` files in increasing order. This\n> +order is important for how we replace the files.\n> +\n> +## Merging commit-graph files\n> +\n> +If we only added a `commit-graph-` file on every write, we would\n> +run into a linear search problem through many commit-graph files.\n> +Instead, we use a merge strategy to decide when the stack should\n> +collapse some number of levels.\n> +\n> +The diagram below shows such a collapse. As a set of new commits\n> +are added, it is determined by the merge strategy that the files\n> +should collapse to `commit-graph-1`. Thus, the new commits, the\n> +commits in `commit-graph-2` and the commits in `commit-graph-1`\n> +should be combined into a new `commit-graph-1` file.\n> +\n> +\t\t\t +---------------------+\n> +\t\t\t | |\n> +\t\t\t | (new commits) |\n> +\t\t\t | |\n> +\t\t\t +---------------------+\n> +\t\t\t | |\n> + +-----------------------+ +---------------------+\n> + | commit-graph-2 |->| |\n> + +-----------------------+ +---------------------+\n> +\t | | |\n> + +-----------------------+ +---------------------+\n> + | | | |\n> + | commit-graph-1 |->| |\n> + | | | |\n> + +-----------------------+ +---------------------+\n> +\t | commit-graph-1.lock\n> + +-----------------------+\n> + | |\n> + | |\n> + | |\n> + | commit-graph |\n> + | |\n> + | |\n> + | |\n> + +-----------------------+\n> +\n> +During this process, the commits to write are combined, sorted\n> +and we write the contents to the `commit-graph-1.lock` file.\n> +When the file is flushed and ready to swap to `commit-graph-1`,\n> +we first unlink the files above our target file. This unlinking\n> +is done from the top of the stack, the reverse direction that\n> +another process would use to read the stack.\n> +\n> +During this time window, another process trying to read the\n> +commit-graph stack could read `commit-graph-1` before the swap\n> +but try to read `commit-graph-2` after it is unlinked. That\n> +process would then believe that this stack is complete, but\n> +will miss out on the performance benefits of the commits in\n> +`commit-graph-2`. For this reason, the stack above the\n> +`commit-graph` file should be small.\n\nConsider the following sequence of events:\n\n 1. There are three commit-graph files in the repository.\n\n 2. A git process opens the base commit-graph and commit-graph-1 for\n reading. It doesn't yet open commit-graph-2, because the (for\n arguments sake not very fair) scheduler takes the CPU away.\n\n 3. Meanwhile, a 'git fetch', well, fetches from a remote, and\n upon noticing that it got a lot of commits it decides to collapse\n commit-graph-1 and -2 and the new commits, writing a brand new\n commit-graph-1.\n\n 4. A second fetch fetches from a second remote, and writes\n commit-graph-2 (no collapsing this time).\n\n 5. Now the crappy scheduler finally decides that it's time to wake\n up the waiting git process from step 2, which then finds the new\n commit-graph-2 file and opens it for reading.\n\n 6. At this point this poor git process has file handles for:\n \n - the base commit-graph file, which is unchanged.\n\n - the old commit-graph-1 which has since been replaced, and does\n not yet contain info about the old commit-graph-2 or the\n commits received in the first fetch.\n\n - the new commit-graph-2, containing info only about commits\n received in the second fetch, and whose parents' graph\n positions point either to the base commitg-graph (good, since\n unchanged) or to the new commit-graph-1 (uh-oh).\n\nWhat happens next? If this process tries to access the parent of a\ncommit from commit-graph-2, and the metadata about this parent is in\nthe new commit-graph-1, then I expect all kinds of weird bugs.\n\nBut will a git process ever try to access a commit that didn't yet\nexisted in the repository when it started opening the commit-graph\nfiles?\n\n\n\n> +## Merge Strategy\n> +\n> +When writing a set of commits that do not exist in the\n> +commit-graph stack of height N, we default to creating\n> +a new file at level N + 1. We then decide to merge\n> +with the Nth level if one of two conditions hold:\n> +\n> + 1. The expected file size for level N + 1 is at\n> + least half the file size for level N.\n> +\n> + 2. Level N + 1 contains more than MAX_SPLIT_COMMITS\n> + commits (64,0000 commits).\n> +\n> +This decision cascades down the levels: when we\n> +merge a level we create a new set of commits that\n> +then compares to the next level.\n> +\n> +The first condition bounds the number of levels\n> +to be logarithmic in the total number of commits.\n> +The second condition bounds the total number of\n> +commits in a `commit-graph-N` file and not in\n> +the `commit-graph` file, preventing significant\n> +performance issues when the stack merges and another\n> +process only partially reads the previous stack.\n> +\n> +The merge strategy values (2 for the size multiple,\n> +64,000 for the maximum number of commits) could be\n> +extracted into config settings for full flexibility.\n> +\n> Related Links\n> -------------\n> [0] https://bugs.chromium.org/p/git/issues/detail?id=8\n> -- \n> gitgitgadget\n> \n"},{"id":"375178","messageId":"d5ef1bcb-a910-295f-fc95-e461d34f3883@gmail.com","threadId":"51052","inReplyTo":"20190508172029.GP14763@szeder.dev","subject":"Re: [PATCH 12/17] Documentation: describe split commit-graphs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-08T19:00:11Z","receivedAt":"2019-05-08T19:00:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/8/2019 1:20 PM, SZEDER Gábor wrote:\n> On Wed, May 08, 2019 at 08:53:57AM -0700, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n> Consider the following sequence of events:\n> \n> 1. There are three commit-graph files in the repository.\n> \n> 2. A git process opens the base commit-graph and commit-graph-1 for\n> reading. It doesn't yet open commit-graph-2, because the (for\n> arguments sake not very fair) scheduler takes the CPU away.\n> \n> 3. Meanwhile, a 'git fetch', well, fetches from a remote, and\n> upon noticing that it got a lot of commits it decides to collapse\n> commit-graph-1 and -2 and the new commits, writing a brand new\n> commit-graph-1.\n> \n> 4. A second fetch fetches from a second remote, and writes\n> commit-graph-2 (no collapsing this time).\n> \n> 5. Now the crappy scheduler finally decides that it's time to wake\n> up the waiting git process from step 2, which then finds the new\n> commit-graph-2 file and opens it for reading.\n> \n> 6. At this point this poor git process has file handles for:\n> \n> - the base commit-graph file, which is unchanged.\n> \n> - the old commit-graph-1 which has since been replaced, and does\n> not yet contain info about the old commit-graph-2 or the\n> commits received in the first fetch.\n> \n> - the new commit-graph-2, containing info only about commits\n> received in the second fetch, and whose parents' graph\n> positions point either to the base commitg-graph (good, since\n> unchanged) or to the new commit-graph-1 (uh-oh).\n> \n> What happens next? If this process tries to access the parent of a\n> commit from commit-graph-2, and the metadata about this parent is in\n> the new commit-graph-1, then I expect all kinds of weird bugs.\n> \n> But will a git process ever try to access a commit that didn't yet\n> existed in the repository when it started opening the commit-graph\n> files?\n\nI'll ignore the improbability of this turn of events (two writes happening\nduring the span of trying to read two files) and focus on the fact that\nwe can prevent issues here using the 4th TODO item in my cover letter:\n\n 4. It would be helpful to add a new optional chunk that contains the\n trailing hash for the lower level of the commit-graph stack. This chunk\n would only be for the commit-graph-N files, and would provide a simple\n way to check that the stack is valid on read, in case we are still\n worried about other processes reading/writing in the wrong order.\n\nIf we have this chunk -- you have convinced me that we need it -- then we\ncould ignore the \"new\" commit-graph-2 because its base graph hash does not\nmatch. We can continue without dying because we can always parse the \"missing\"\ncommits from the packs.\n\nThanks,\n-Stolee\n"},{"id":"375181","messageId":"87r298hhlc.fsf@evledraar.gmail.com","threadId":"51052","inReplyTo":"pull.184.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 00/17] [RFC] Commit-graph: Write incremental files","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-05-08T19:27:43Z","receivedAt":"2019-05-08T19:27:49Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, May 08 2019, Derrick Stolee via GitGitGadget wrote:\n\n> This patch series is marked as RFC quality because it is missing some key\n> features and tests, but hopefully starts a concrete discussion of how the\n> incremental commit-graph writes can work. If this is a good direction, then\n> it would replace ds/commit-graph-format-v2.\n\nI have some comments on 12/17 that I'll put there.\n\nI think it would be best to start by submitting 1-11 for inclusion so we\ncan get minor cleanups/refactoring out of the way. I've only skimmed\nthose patches, but they seem to be obviously correct, although the diff\nmove detection (and with -w) doesn't help much with them.\n\nThis next bit sounds petty, but I honestly don't mean it that way :)\n\nOne minor thing I want to note is 04/17. The change itself I 100% agree\non (in-tree docs are bad places for TODO lists), but the commit message\n*still* says that a \"verify\" is just as slow as \"write\", even though I\nnoted a ~40% difference in [1].\n\nDo I care about that tiny isolated thing? Nope. But I *do* think it's\nindicative of a general thing that could be improved in these RFC\niterations that I found a bit frustrating in reading through\nit. I.e. you're getting some of the \"C[comments]\", but then there's\nre-rolled patches that don't address those things.\n\nWhat we say in the commit message for 4/17 obviously doesn't matter much\nat all. But there's other outstanding feedback on the last iteration\nthat from reading this one still mostly/entirely applies.\n\nSo I'll just leave this reply at \"I have a lot of comments\", but that\nthey're still sitting there.\n\n1. https://public-inbox.org/git/87o94mql0a.fsf@evledraar.gmail.com/\n"},{"id":"375184","messageId":"87pnoshfkq.fsf@evledraar.gmail.com","threadId":"51052","inReplyTo":"d5ef1bcb-a910-295f-fc95-e461d34f3883@gmail.com","subject":"Re: [PATCH 12/17] Documentation: describe split commit-graphs","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-05-08T20:11:17Z","receivedAt":"2019-05-08T20:11:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, May 08 2019, Derrick Stolee wrote:\n\n> On 5/8/2019 1:20 PM, SZEDER Gábor wrote:\n>> On Wed, May 08, 2019 at 08:53:57AM -0700, Derrick Stolee via GitGitGadget wrote:\n>>> From: Derrick Stolee \n>> Consider the following sequence of events:\n>>\n>> 1. There are three commit-graph files in the repository.\n>>\n>> 2. A git process opens the base commit-graph and commit-graph-1 for\n>> reading. It doesn't yet open commit-graph-2, because the (for\n>> arguments sake not very fair) scheduler takes the CPU away.\n>>\n>> 3. Meanwhile, a 'git fetch', well, fetches from a remote, and\n>> upon noticing that it got a lot of commits it decides to collapse\n>> commit-graph-1 and -2 and the new commits, writing a brand new\n>> commit-graph-1.\n>>\n>> 4. A second fetch fetches from a second remote, and writes\n>> commit-graph-2 (no collapsing this time).\n>>\n>> 5. Now the crappy scheduler finally decides that it's time to wake\n>> up the waiting git process from step 2, which then finds the new\n>> commit-graph-2 file and opens it for reading.\n>>\n>> 6. At this point this poor git process has file handles for:\n>>\n>> - the base commit-graph file, which is unchanged.\n>>\n>> - the old commit-graph-1 which has since been replaced, and does\n>> not yet contain info about the old commit-graph-2 or the\n>> commits received in the first fetch.\n>>\n>> - the new commit-graph-2, containing info only about commits\n>> received in the second fetch, and whose parents' graph\n>> positions point either to the base commitg-graph (good, since\n>> unchanged) or to the new commit-graph-1 (uh-oh).\n>>\n>> What happens next? If this process tries to access the parent of a\n>> commit from commit-graph-2, and the metadata about this parent is in\n>> the new commit-graph-1, then I expect all kinds of weird bugs.\n>>\n>> But will a git process ever try to access a commit that didn't yet\n>> existed in the repository when it started opening the commit-graph\n>> files?\n>\n> I'll ignore the improbability of this turn of events (two writes happening\n> during the span of trying to read two files) and focus on the fact that\n> we can prevent issues here using the 4th TODO item in my cover letter:\n\nFWIW the sort of scenario SZEDER is describing is something I deal with\nin production a lot. It doesn't require an unfair scheduler, just that\nyou have differently nice(1)'d processes accessing the same repo.\n\nSo if you have batch \"cron\" processes their IO scheduling follows their\nnice(1) scheduling. It's not atypical to e.g. have some background\nthingy sit for seconds or even minutes on an I/O syscall while the\nkernel decides everyone else has right of way, since you nice'd that not\ncaring if it finishes in 10 seconds or 10 hours.\n\n> 4. It would be helpful to add a new optional chunk that contains the\n> trailing hash for the lower level of the commit-graph stack. This chunk\n> would only be for the commit-graph-N files, and would provide a simple\n> way to check that the stack is valid on read, in case we are still\n> worried about other processes reading/writing in the wrong order.\n>\n> If we have this chunk -- you have convinced me that we need it -- then we\n> could ignore the \"new\" commit-graph-2 because its base graph hash does not\n> match. We can continue without dying because we can always parse the \"missing\"\n> commits from the packs.\n\nHaving read the actual code for this & what the format will look like\n(last time I didn't have that context[1]) I really don't get why we\ndon't save ourselves a lot of trouble and do away with this \"N\" in the\nfilename idea from the start.\n\nIt's just a slight change to the iteration in\nprepare_commit_graph_one().\n\nInstead of looping through files N at a time we'd have a discovery step\nwhere we'd need to open() all the files, see which ones say \"my parent\nhash hash X\", and then create a list of those hashes in order to read a\nbunch of commit-graph- files.\n\nIs that a bit painful? Sure, but way less painful than dealing with the\ncaveats I'd mentioned in [1] and SZEDER details here.\n\nAnd the obvious thing would be to save ourselves most of that work every\ntime we read by writing a .git/objects/commit-graphs/info on\n\"commit-graph write\", which would be the of the end of the\n\"latest\" chain. We could also have some side-index listing the whole\n\"current\" chain in order (but I'm more paranoid about locking/updates to\nsuch a thing, maybe we could put it in the last file in a new chunk\n....).\n\nIf we didn't have such side-indexes then the way the current loading\nworks it would need to traverse the files back to front, and *then* load\nthem front to back (as it does now), so that's a slight pain,\nobviously. But not a big deal.\n\nWith commit-graph- all these unlink() race conditions go away,\npartial reads due to concurrent graph writing becomes a non-issue (we'd\njust leave the old files, \"gc\" deals with them later..), no need to\ncarefully fsync() files/dirs etc as we need to carefully juggle N and\nN+1 files.\n\nIt also becomes easy to \"chain\" graphs across repos e.g. via\nalternates. Say in the scenario github/gitlab have where they have a\n\"main\" repo and other objects on another delta island.\n\nIn that case the repo would have a local \"tip\" file with the last link\nin its chain, some of which would then refer back to in other\n\"parent\" alternates.\n\nAs long as such a setup has a \"gc\" process that's not overly eager about\npruning old stuff and considers that constellation of repos as a whole\nthat should just work. You can freely optimize and rewrite graphs across\nrepos, just be careful about unlinking old stuff.\n\nI don't see how it would work with commit-graph-N without a *lot* of\npainful orchestration (where e.g. you *must* guarantee that the parent\nrepo ends in N, all child repos start at N+1).\n\n> This allows incremental writes without updating the file format.\n\nFWIW this is some of what I was talking about in [2]. In\nds/commit-graph-format-v2 I had feedback to the effect[3] that the\nparticular way in which you proposed to update the format (changing the\nheader) wouldn't be worth it, since old clients dealt with it so badly.\n\nBut as noted in [3] I see zero reason for why we can't update the\nexisting format, we just add new chunks. That allows us to add any new\ndata in backwards-compatible ways.\n\nI see nothing wrong with solution that has split files in principle,\njust with the currently proposed commit-graph-N way of doing that.\n\nI just wonder if we're looking at a \"Y\" solution to an \"X-Y\" problem\nwhere \"X\" was unduly dismissed. If updating the format was a non-issue\n(which seems to me to be the case), what then?\n\nI imagine we'd still have just a \"commit-graph\" file, to write a new one\nwe'd \"cp\" that one, then munge that existing file to write something new\nto the and and \"mv\" it in-place. It seems to me a (sane) split-file plan\nis better, but I'm not in your head, maybe it was a much better for\nreasons I'm not imagining before I apparently talked you out of changing\nthe format itself :)\n\n1. https://public-inbox.org/git/87bm0jirjj.fsf@evledraar.gmail.com/\n2. https://public-inbox.org/git/87r298hhlc.fsf@evledraar.gmail.com/\n3. https://public-inbox.org/git/87lfzprkfc.fsf@evledraar.gmail.com/\n"},{"id":"375200","messageId":"xmqq8svgfczw.fsf@gitster-ct.c.googlers.com","threadId":"51052","inReplyTo":"87pnoshfkq.fsf@evledraar.gmail.com","subject":"Re: [PATCH 12/17] Documentation: describe split commit-graphs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-05-09T04:49:55Z","receivedAt":"2019-05-09T04:50:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> With commit-graph- all these unlink() race conditions go away,\n> partial reads due to concurrent graph writing becomes a non-issue (we'd\n> just leave the old files, \"gc\" deals with them later..), no need to\n> carefully fsync() files/dirs etc as we need to carefully juggle N and\n> N+1 files.\n\nThe above would give a nice course correction to be in line with the\nrest of the system, like how split index knows about and chains to\nits base. Thanks for a dose of sanity.\n\n\n"},{"id":"375207","messageId":"87imukgier.fsf@evledraar.gmail.com","threadId":"51052","inReplyTo":"3c52385e5696887c40cab4a6b9b7923d60a0567c.1557330827.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 17/17] fetch: add fetch.writeCommitGraph config setting","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-05-09T08:07:40Z","receivedAt":"2019-05-09T08:07:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, May 08 2019, Derrick Stolee via GitGitGadget wrote:\n\n> From: Derrick Stolee \n>\n> Signed-off-by: Derrick Stolee \n> ---\n> builtin/fetch.c | 17 +++++++++++++++++\n> 1 file changed, 17 insertions(+)\n>\n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index b620fd54b4..cf0944bad5 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -23,6 +23,7 @@\n> #include \"packfile.h\"\n> #include \"list-objects-filter-options.h\"\n> #include \"commit-reach.h\"\n> +#include \"commit-graph.h\"\n>\n> static const char * const builtin_fetch_usage[] = {\n> \tN_(\"git fetch [] [ [...]]\"),\n> @@ -62,6 +63,7 @@ static const char *submodule_prefix = \"\";\n> static int recurse_submodules = RECURSE_SUBMODULES_DEFAULT;\n> static int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;\n> static int shown_url = 0;\n> +static int fetch_write_commit_graph = 0;\n> static struct refspec refmap = REFSPEC_INIT_FETCH;\n> static struct list_objects_filter_options filter_options;\n> static struct string_list server_options = STRING_LIST_INIT_DUP;\n> @@ -79,6 +81,11 @@ static int git_fetch_config(const char *k, const char *v, void *cb)\n> \t\treturn 0;\n> \t}\n>\n> +\tif (!strcmp(k, \"fetch.writecommitgraph\")) {\n> +\t\tfetch_write_commit_graph = 1;\n> +\t\treturn 0;\n> +\t}\n> +\n> \tif (!strcmp(k, \"submodule.recurse\")) {\n> \t\tint r = git_config_bool(k, v) ?\n> \t\t\tRECURSE_SUBMODULES_ON : RECURSE_SUBMODULES_OFF;\n> @@ -1670,6 +1677,16 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n>\n> \tstring_list_clear(&list, 0);\n>\n> +\tif (fetch_write_commit_graph) {\n> +\t\tint commit_graph_flags = COMMIT_GRAPH_SPLIT;\n> +\n> +\t\tif (progress)\n> +\t\t\tcommit_graph_flags |= COMMIT_GRAPH_PROGRESS;\n> +\n> +\t\twrite_commit_graph_reachable(get_object_directory(),\n> +\t\t\t\t\t commit_graph_flags);\n> +\t}\n> +\n> \tclose_all_packs(the_repository->objects);\n>\n> \targv_array_pushl(&argv_gc_auto, \"gc\", \"--auto\", NULL);\n\nI'm keen in general to refactor \"git gc --auto\" a bit so it moves away\nfrom being an all-or-nothing to something where we can do an\n\"incremental\" gc.\n\nI'm happy to do that work, the main thing it's been blocked on is not\nhaving some fast easy-to-lookup heuristic for one of those \"incremental\"\nthings.\n\nThe two obvious candidates are the commit-graph (I mainly wanted this on\n\"gc --auto\" after clone), and pack-refs (but doing that is more\nexpensive).\n\nSo rather than have this patch I'd like to as noted in 00/17 get the\nrefactoring bits of the commit-graph in first.\n\nThen some version of my WIP patch in\nhttps://public-inbox.org/git/87lfzprkfc.fsf@evledraar.gmail.com/ where\nwe'd note the number of objects we had when we did the last commit-graph\nin the graph itself.\n\nThen \"gc --auto\" would look at that, then approximate_object_count(),\nand have some percentage threshhold for doing a \"do some of the gc\"\ntask, which would just be a small change to need_to_gc() to make it\nreturn/populate a \"what needs to be done\" rather than \"yes/no\".\n\nThat would give you what you want here, but also be a more general\nsolution. E.g. we'd write the graph on \"clone\" once \"gc --auto\" was\ncalled there, as well as on \"fetch\".\n"},{"id":"375221","messageId":"8195f9a4-1808-e23e-f633-7e6809cc3656@gmail.com","threadId":"51052","inReplyTo":"xmqq8svgfczw.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 12/17] Documentation: describe split commit-graphs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-09T12:25:01Z","receivedAt":"2019-05-09T12:25:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/9/2019 12:49 AM, Junio C Hamano wrote:\n> Ævar Arnfjörð Bjarmason writes:\n> \n>> With commit-graph- all these unlink() race conditions go away,\n>> partial reads due to concurrent graph writing becomes a non-issue (we'd\n>> just leave the old files, \"gc\" deals with them later..), no need to\n>> carefully fsync() files/dirs etc as we need to carefully juggle N and\n>> N+1 files.\n> \n> The above would give a nice course correction to be in line with the\n> rest of the system, like how split index knows about and chains to\n> its base. Thanks for a dose of sanity.\n\nI'm working on a detailed response to Ævar's ideas, to be sure we are\ntalking about the same thing, because the original motivation for the\ncommit-graph format v2 was to allow the 'commit-graph' file to point\nto a chain of base files by a list of hashes (like the split index\ndoes). The current proposal was created in response to an unwillingness\nto break the file format for the 'commit-graph' file.\n\n-Stolee\n"},{"id":"375225","messageId":"f27db281-abad-5043-6d71-cbb083b1c877@gmail.com","threadId":"51052","inReplyTo":"87pnoshfkq.fsf@evledraar.gmail.com","subject":"Re: [PATCH 12/17] Documentation: describe split commit-graphs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-09T13:45:05Z","receivedAt":"2019-05-09T13:45:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/8/2019 4:11 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, May 08 2019, Derrick Stolee wrote:\n>\n>> I'll ignore the improbability of this turn of events (two writes happening\n>> during the span of trying to read two files) and focus on the fact that\n>> we can prevent issues here using the 4th TODO item in my cover letter:\n> \n> FWIW the sort of scenario SZEDER is describing is something I deal with\n> in production a lot. It doesn't require an unfair scheduler, just that\n> you have differently nice(1)'d processes accessing the same repo.\n> \n> So if you have batch \"cron\" processes their IO scheduling follows their\n> nice(1) scheduling. It's not atypical to e.g. have some background\n> thingy sit for seconds or even minutes on an I/O syscall while the\n> kernel decides everyone else has right of way, since you nice'd that not\n> caring if it finishes in 10 seconds or 10 hours.\n\nThanks. I learned something today.\n\nI still don't think this is a problem, with the idea below.\n\n>> 4. It would be helpful to add a new optional chunk that contains the\n>> trailing hash for the lower level of the commit-graph stack. This chunk\n>> would only be for the commit-graph-N files, and would provide a simple\n>> way to check that the stack is valid on read, in case we are still\n>> worried about other processes reading/writing in the wrong order.\n>>\n>> If we have this chunk -- you have convinced me that we need it -- then we\n>> could ignore the \"new\" commit-graph-2 because its base graph hash does not\n>> match. We can continue without dying because we can always parse the \"missing\"\n>> commits from the packs.\n\nSo, let's set that idea aside. You have other concerns.\n\n> Instead of looping through files N at a time we'd have a discovery step\n> where we'd need to open() all the files, see which ones say \"my parent\n> hash hash X\", and then create a list of those hashes in order to read a\n> bunch of commit-graph- files.\n> \n> Is that a bit painful? Sure, but way less painful than dealing with the\n> caveats I'd mentioned in [1] and SZEDER details here.\n\nI don't see how this step is less painful than the one I am describing.\nYou'll need to be a bit more specific to convince me.\n\nI'll try to be specific with a few ideas that have been thrown around,\nso we can compare and contrast (and point out where I am misunderstanding\nwhat you are trying to say).\n\nOption 0 (this series): commit-graph-N\n--------------------------------------\n\nOn read, we look for the 'info/commit-graph' file and acquire a read handle.\nWe set that commit_graph struct as our tip commit-graph. Then, for each\nincreasing N (until we fail) we acquire a read handle on\n'info/commit-graphs/commit-graph-N' and check that its base hash matches\nour current tip commit-graph. If the file doesn't exist, or the base\nhash doesn't match, then we stop and continue with our current tip graph.\n\nOn write, use a 'mv' to swap our .lock file with whatever level we are\nmerging, THEN can unlink() the higher layers in decreasing order. (This\n\"mv-then-unlink\" order is different than what is implemented by this\nseries, but is enabled by the chunk containing the base graph hash.)\n\nOption 1 (needs format v2): commit-graph -> graph-{hash}.graph\n--------------------------------------------------------------\n\nOn read, we load the 'info/commit-graph' file and inspect the byte saying\nhow many base files we have. We load their hashes from the base file chunk\nand read 'info/graph-{hash}.graph' for each. If _any_ fail, then we need\nto ignore anything \"above\" the failure in the chain (and specifically this\nincludes the 'commit-graph' file), or consider reloading the commit-graph\nfile altogether and hope it works this time. [Note: if an older version\nof Git doesn't understand the incremental file format, it will fail to\nproperly understand the graph positions and either fail with an\n\"invalid parent position\" error, or worse give garbage results.]\n\nOn write, if we are creating a new layer to our chain, we need to _copy_\nthe existing commit-graph file to a graph-{hash}.graph file before renaming\nthe .lock file. If we are merging layers, then we either (a) clean up the\ndangling chains after moving our commit-graph file, or (b) have something\nlike 'gc' clean up the files later. I think using 'gc' for this is not\na good idea, since I would expect these files to be written and merged\nmuch more frequently (say, after a 'fetch') than a 'gc' is run. Cleaning\nup the dangling chains leads to our concurrency issues. Further, this\n'copy the base' no longer keeps our large base file at rest.\n\nOption 2: grovel commit-graphs directory for graph-{hash}.graph\n---------------------------------------------------------------\n\nOn read, we load the 'info/commit-graph' file and assume it is never\nan incremental file. Then, scan the 'info/commit-graphs' directory\nfor 'graph-{hash}.graph' files and open them _all_ up to construct\na \"graph forest\" (each graph has a single parent, given by a chunk\nspecifying its base graph hash). If we don't have an instance of a\ngraph with a given hash, then ignore any graphs pointing to that hash.\nWe now have a decision to make: which leaf of this forest should we\nuse as our tip commit-graph? That could be given by the\ncommit-graphs/info file. But what happens when we have timing issues\naround scanning the directory and the commit-graphs/info file? Do\nwe fall back to modified time?\n\nOn write, if we are not merging, then we just create a new\ngraph-{hash}.graph file. If we are merging, but still have a base graph,\nthen create a new graph-{hash}.graph file. Finally, if we are merging\nall layers, then we rename our .lock file to 'info/commit-graph'.\nTo clean up, we need to grovel the directory to look for graph-{hash}.graph\nfiles whose base chains no longer match the new, \"best\" chain and unlink()\nthem. This clean-up step can happen at any time.\n\n--[end description of options]--\n\nDid I accurately describe the options we are considering?\n\nOption 1 was the design I was planning, and I think it matches how the\nsplit-index feature works. Please correct me if I am missing something.\nIt _requires_ updating the file format version. But it also has a flaw\nthat the other options do not have: the copy of the base file. One\nthing I want to enable is for whatever machinery is handling these\nfile writes to run a 'verify' immediately after, and have that be fast\nmost of the time. With a model that changes only the \"tip\" file, we\ncan verify only the new files and have confidence that the base file\ndid not change. I think options 0 and 2 both improve in this direction.\n \n> With commit-graph- all these unlink() race conditions go away,\n> partial reads due to concurrent graph writing becomes a non-issue (we'd\n> just leave the old files, \"gc\" deals with them later..), no need to\n> carefully fsync() files/dirs etc as we need to carefully juggle N and\n> N+1 files.\n\nCalling this a non-issue is an exaggeration, especially if you are\nclaiming we need to be robust to multi-hour gaps between reading files.\n\n> It also becomes easy to \"chain\" graphs across repos e.g. via\n> alternates. Say in the scenario github/gitlab have where they have a\n> \"main\" repo and other objects on another delta island.\n> \n> In that case the repo would have a local \"tip\" file with the last link\n> in its chain, some of which would then refer back to in other\n> \"parent\" alternates.\n> \n> As long as such a setup has a \"gc\" process that's not overly eager about\n> pruning old stuff and considers that constellation of repos as a whole\n> that should just work. You can freely optimize and rewrite graphs across\n> repos, just be careful about unlinking old stuff.\n> \n> I don't see how it would work with commit-graph-N without a *lot* of\n> painful orchestration (where e.g. you *must* guarantee that the parent\n> repo ends in N, all child repos start at N+1).\n\nYou're right that Option 0 does not work in this model where some graph\ninformation is stored in an alternate _and_ more information is stored\noutside the alternate. My perspective is biased, because I consider the\nalternate to be \"almost everything\" and the local object store to be\nsmall. But in a fork network, this is not always the case. I appreciate\nyour feedback for this environment, and I've always hoped that someone\nwith server experience would come and say \"this feature is great, but\nwe need X, Y, and Z to make best use of it in our environment. Here's\na patch that moves us in that direction!\" At least you are doing the\nnext-best thing: stopping me from making mistakes that would block\nadoption.\n\nSo let's consider how Option 2 would work in this \"multi-tip\" case.\nEach object directory would have some number of graph files, and one\n'commit-graphs/info' file pointing to some hash. When we read, we\ntry to pick the info file that is \"closest\" to us.\n\nThis does create some complications that I don't think you gave enough\nattention to. These may be solvable, but they are non-trivial:\n\n* When we 'gc' the \"core\" repo, we need to enumerate all of the\n \"leaf\" repos to check their tip commit-graph files and make a\n decision if we should keep their bases around or delete those tips.\n Perhaps I'm over-stating the difficulty here, since we need to do\n something similar to find still-reachable objects, anyway. But if\n we are doing that reachability calculation, then why are we not\n just putting all of the commit-graph data in the core repo? Note\n that we don't get the same value as delta islands because this data\n isn't being shared across the protocol. The issue with storing all\n graph data in the core repo is that the core repo doesn't actually\n have all of the commits, which makes 'verify' on the graph a bit\n annoying.\n\n* If we choose a local tip instead of the \"core\" tip, then that chain\n of commit-graphs can be far behind the core repo. In the world where\n a fork moves only at the speed of a single developer, but the core\n project moves quickly, then computing a merge base with the core's\n master branch becomes slow as our local chain doesn't contain most\n of the commits.\n\n* We can't take all of the \"core\" chain _and_ the local chain, because\n the concept of \"graph position\" no longer makes sense. The only way\n I see out of this is to make the graph position two-dimensional:\n commit -> (index of tip chain, position in that chain). Perhaps this\n is a valuable thing to do in the future? Or perhaps, we shouldn't\n have incremental chains spanning object directories and instead\n introduce \"parents-by-ref\" where we mark some parents as included\n by object id instead of by graph position. This would allow the\n core repo to gc without caring about the external repos. It also\n wouldn't care about how the graph files are stored (Option 0 would\n work, as graph chains would not cross object store boundaries) and\n more closely resembles the independence of the pack-files in each\n object store. The \"parents-by-ref\" would require updating the\n file format version.\n\n--[end discussion of incremental files]--\n\nI'll move forward applying your existing feedback on patches 1-11 and\nsubmit as a full series to replace ds/commit-graph-format-v2. We can\nwork on reviewing that code while we continue to think critically on\nthe topic of incremental files.\n\nThanks,\n-Stolee\n\n\n"},{"id":"375227","messageId":"b1de6af2-c015-098e-a656-e1b68056e037@gmail.com","threadId":"51052","inReplyTo":"87imukgier.fsf@evledraar.gmail.com","subject":"Re: [PATCH 17/17] fetch: add fetch.writeCommitGraph config setting","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-09T14:21:23Z","receivedAt":"2019-05-09T14:21:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/9/2019 4:07 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> So rather than have this patch I'd like to as noted in 00/17 get the\n> refactoring bits of the commit-graph in first.\n\nRefactor-only series coming soon.\n\n> Then some version of my WIP patch in\n> https://public-inbox.org/git/87lfzprkfc.fsf@evledraar.gmail.com/ where\n> we'd note the number of objects we had when we did the last commit-graph\n> in the graph itself.\n> \n> Then \"gc --auto\" would look at that, then approximate_object_count(),\n> and have some percentage threshhold for doing a \"do some of the gc\"\n> task, which would just be a small change to need_to_gc() to make it\n> return/populate a \"what needs to be done\" rather than \"yes/no\".\n> \n> That would give you what you want here, but also be a more general\n> solution. E.g. we'd write the graph on \"clone\" once \"gc --auto\" was\n> called there, as well as on \"fetch\".\n\nI think this \"gc --auto\" idea is solid, but I also think there is value\nin writing a commit-graph after _every_ fetch, not just one big enough\nto trigger this new gc behavior. Perhaps your new metadata in the\ncommit-graph file could store multiple values for \"number of objects since\nX was cleaned up\" where X is in { packs, reflog, commit-graph, etc. } and \nGC could consider each maintenance task independently. _That_ would make\nthis patch unnecessary.\n\nThanks,\n-Stolee\n\n"},{"id":"375241","messageId":"875zqjy6ga.fsf@evledraar.gmail.com","threadId":"51052","inReplyTo":"f27db281-abad-5043-6d71-cbb083b1c877@gmail.com","subject":"Re: [PATCH 12/17] Documentation: describe split commit-graphs","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-05-09T15:48:37Z","receivedAt":"2019-05-09T15:48:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, May 09 2019, Derrick Stolee wrote:\n\nNot a very detailed reply since I've gotta run soon, but figured I'd\nsend something (also given our timezone difference).\n\n> On 5/8/2019 4:11 PM, Ævar Arnfjörð Bjarmason wrote:\n>>\n>> On Wed, May 08 2019, Derrick Stolee wrote:\n>>\n>>> I'll ignore the improbability of this turn of events (two writes happening\n>>> during the span of trying to read two files) and focus on the fact that\n>>> we can prevent issues here using the 4th TODO item in my cover letter:\n>>\n>> FWIW the sort of scenario SZEDER is describing is something I deal with\n>> in production a lot. It doesn't require an unfair scheduler, just that\n>> you have differently nice(1)'d processes accessing the same repo.\n>>\n>> So if you have batch \"cron\" processes their IO scheduling follows their\n>> nice(1) scheduling. It's not atypical to e.g. have some background\n>> thingy sit for seconds or even minutes on an I/O syscall while the\n>> kernel decides everyone else has right of way, since you nice'd that not\n>> caring if it finishes in 10 seconds or 10 hours.\n>\n> Thanks. I learned something today.\n>\n> I still don't think this is a problem, with the idea below.\n>\n>>> 4. It would be helpful to add a new optional chunk that contains the\n>>> trailing hash for the lower level of the commit-graph stack. This chunk\n>>> would only be for the commit-graph-N files, and would provide a simple\n>>> way to check that the stack is valid on read, in case we are still\n>>> worried about other processes reading/writing in the wrong order.\n>>>\n>>> If we have this chunk -- you have convinced me that we need it -- then we\n>>> could ignore the \"new\" commit-graph-2 because its base graph hash does not\n>>> match. We can continue without dying because we can always parse the \"missing\"\n>>> commits from the packs.\n>\n> So, let's set that idea aside. You have other concerns.\n>\n>> Instead of looping through files N at a time we'd have a discovery step\n>> where we'd need to open() all the files, see which ones say \"my parent\n>> hash hash X\", and then create a list of those hashes in order to read a\n>> bunch of commit-graph- files.\n>>\n>> Is that a bit painful? Sure, but way less painful than dealing with the\n>> caveats I'd mentioned in [1] and SZEDER details here.\n>\n> I don't see how this step is less painful than the one I am describing.\n> You'll need to be a bit more specific to convince me.\n>\n> I'll try to be specific with a few ideas that have been thrown around,\n> so we can compare and contrast (and point out where I am misunderstanding\n> what you are trying to say).\n>\n> Option 0 (this series): commit-graph-N\n> --------------------------------------\n>\n> On read, we look for the 'info/commit-graph' file and acquire a read handle.\n> We set that commit_graph struct as our tip commit-graph. Then, for each\n> increasing N (until we fail) we acquire a read handle on\n> 'info/commit-graphs/commit-graph-N' and check that its base hash matches\n> our current tip commit-graph. If the file doesn't exist, or the base\n> hash doesn't match, then we stop and continue with our current tip graph.\n\nThis \"base hash\" is something I may have missed. I *thought* we'd\nhappy-go-lucky load e.g. commit-graph-1, commit-graph-2 in that order,\nand if (e.g. due to fsync/dir entry issues noted earlier) if we got the\n\"wrong\" commit-graph-2 we'd be oblivious to that. Is that not the case?\n\nI was trying (and failing) to make the test suite vomit by hacking up\nthe code to load \"commit-graph-1\" first, then \"commit-graph\", so maybe I\nran into that safeguard, whatever it is. I've just seen how we carry the\n\"num_commits_in_base\" forward, not some expected checksum...\n\n> On write, use a 'mv' to swap our .lock file with whatever level we are\n> merging, THEN can unlink() the higher layers in decreasing order. (This\n> \"mv-then-unlink\" order is different than what is implemented by this\n> series, but is enabled by the chunk containing the base graph hash.)\n\nSo one of the things that make me paranoid about this (as noted\nupthread/earlier) is that at least on POSIX filesystems just because you\nobserve that you create and fsync \"foo\" and \"bar\" in that order doesn't\nmean that a concurrent reader of that directory will get the updates in\nthat order. I.e. file update != dir entry update.\n\nIt can be made to work, and core.fsyncObjectFiles is a partial solution,\nbut better to avoid it entirely. Also the failure with\ncore.fsyncObjectFiles is much more graceful, since in that case we don't\nhave a \"deadbeef[...]\" loose object *yet* but will (presumably) see the\n*right* thing Real Soon Now since it's content-addressable, whereas with\nthis design we might see the *wrong* thing. So this would be the first\narea of git (that I know of) where we'd be sensitive to a combination of\nsyncing dir entries and file content in order.\n\n> Option 1 (needs format v2): commit-graph -> graph-{hash}.graph\n> --------------------------------------------------------------\n>\n> On read, we load the 'info/commit-graph' file and inspect the byte saying\n> how many base files we have. We load their hashes from the base file chunk\n> and read 'info/graph-{hash}.graph' for each. If _any_ fail, then we need\n> to ignore anything \"above\" the failure in the chain (and specifically this\n> includes the 'commit-graph' file), or consider reloading the commit-graph\n> file altogether and hope it works this time. [Note: if an older version\n> of Git doesn't understand the incremental file format, it will fail to\n> properly understand the graph positions and either fail with an\n> \"invalid parent position\" error, or worse give garbage results.]\n\nRight, except as noted before I'd add (at least per my understanding so\nfar) \"s/needs format v2//\". I.e. we could stick this data in new v1\nbackwards-compatible chunks, which also addresses the \"older version...\"\ncaveat.\n\nI.e. we'd always guarantee that the \"commit-graph\" file was\nunderstandable/valid for older versions (even though eventually their\nview on it would be \"oh, this has no data\" (but really it's all in a new\nchunk they don't grok).\n\n> On write, if we are creating a new layer to our chain, we need to _copy_\n> the existing commit-graph file to a graph-{hash}.graph file before renaming\n> the .lock file. If we are merging layers, then we either (a) clean up the\n> dangling chains after moving our commit-graph file, or (b) have something\n> like 'gc' clean up the files later. I think using 'gc' for this is not\n> a good idea, since I would expect these files to be written and merged\n> much more frequently (say, after a 'fetch') than a 'gc' is run. Cleaning\n> up the dangling chains leads to our concurrency issues. Further, this\n> 'copy the base' no longer keeps our large base file at rest.\n\nSo aside from the specifics of implementation both this and the\ncommit-graph-N way of doing it involves juggling a sequence of chunks\nthat \"point upwards\", which IMO has worse caveats than \"point\ndownwards\". I.e. we'd need to rewrite \"base\" files to point \"upwards\" to\nnew stuff, and there's no (sane) way in option 0 to split commit-graphs\nacross repos, and no way at all in this scenario (if I understand it\ncorrectly...).\n\n> Option 2: grovel commit-graphs directory for graph-{hash}.graph\n> ---------------------------------------------------------------\n>\n> On read, we load the 'info/commit-graph' file and assume it is never\n> an incremental file. Then, scan the 'info/commit-graphs' directory\n> for 'graph-{hash}.graph' files and open them _all_ up to construct\n> a \"graph forest\" (each graph has a single parent, given by a chunk\n> specifying its base graph hash). If we don't have an instance of a\n> graph with a given hash, then ignore any graphs pointing to that hash.\n> We now have a decision to make: which leaf of this forest should we\n> use as our tip commit-graph? That could be given by the\n> commit-graphs/info file. But what happens when we have timing issues\n> around scanning the directory and the commit-graphs/info file? Do\n> we fall back to modified time?\n\nI think in a worst-case scenario we shrug and just pick whatever chain\nlooks most recent/largest or whatever, but in practice I think it's a\nnon-issue.\n\nCritical for that \"it's a non-issue\" is the suggestion I had on 17/17 of\nnot doing the incremental write in \"fetch\", but instead just rely on \"gc\n--auto\" *and* to address your \"I would expect these files to be written\nand merged much more frequently\" we'd just (and I'm keen to hack this\nup) teach \"gc --auto\" such an incremental mode.\n\nThis means that within a single repo all updates to the commit-graph\nwill go through the gc.lock, whereas with your current 17/17 we'd\npotentially have a \"commit-graph write\" racing a concurrent \"gc\", with\nboth aiming to update the commit-graph.\n\n> On write, if we are not merging, then we just create a new\n> graph-{hash}.graph file. If we are merging, but still have a base graph,\n> then create a new graph-{hash}.graph file. Finally, if we are merging\n> all layers, then we rename our .lock file to 'info/commit-graph'.\n> To clean up, we need to grovel the directory to look for graph-{hash}.graph\n> files whose base chains no longer match the new, \"best\" chain and unlink()\n> them. This clean-up step can happen at any time.\n>\n> --[end description of options]--\n>\n> Did I accurately describe the options we are considering?\n>\n> Option 1 was the design I was planning, and I think it matches how the\n> split-index feature works. Please correct me if I am missing something.\n> It _requires_ updating the file format version. But it also has a flaw\n> that the other options do not have: the copy of the base file. One\n> thing I want to enable is for whatever machinery is handling these\n> file writes to run a 'verify' immediately after, and have that be fast\n> most of the time. With a model that changes only the \"tip\" file, we\n> can verify only the new files and have confidence that the base file\n> did not change. I think options 0 and 2 both improve in this direction.\n>\n>> With commit-graph- all these unlink() race conditions go away,\n>> partial reads due to concurrent graph writing becomes a non-issue (we'd\n>> just leave the old files, \"gc\" deals with them later..), no need to\n>> carefully fsync() files/dirs etc as we need to carefully juggle N and\n>> N+1 files.\n>\n> Calling this a non-issue is an exaggeration, especially if you are\n> claiming we need to be robust to multi-hour gaps between reading files.\n\nWe always have a race, but they're very different races.\n\nWith \"N\" files (option #0) we'd have races of the type where within the\nsame few milliseconds of a commit-graph being merged/updated a\ne.g. concurrent \"tag --contains\" would either be much slower (couldn't\nget one of the incremental graphs it expected), or produce the wrong\nanswer (this may be wrong on my part, see my \"base hash\" comment above).\n\nWhereas if I run something with \"ionice -c 3\" I could possibly hang for\nhowever many hours/days/weeks we wait until another \"gc\" comes along and\nunlinks those old files, but if I'm running it like that I'm not\nexpecting it to be fast, so it's OK if the files went away, and it won't\never get the wrong file (since the filenames are hash-addressible).\n\n>> It also becomes easy to \"chain\" graphs across repos e.g. via\n>> alternates. Say in the scenario github/gitlab have where they have a\n>> \"main\" repo and other objects on another delta island.\n>>\n>> In that case the repo would have a local \"tip\" file with the last link\n>> in its chain, some of which would then refer back to in other\n>> \"parent\" alternates.\n>>\n>> As long as such a setup has a \"gc\" process that's not overly eager about\n>> pruning old stuff and considers that constellation of repos as a whole\n>> that should just work. You can freely optimize and rewrite graphs across\n>> repos, just be careful about unlinking old stuff.\n>>\n>> I don't see how it would work with commit-graph-N without a *lot* of\n>> painful orchestration (where e.g. you *must* guarantee that the parent\n>> repo ends in N, all child repos start at N+1).\n>\n> You're right that Option 0 does not work in this model where some graph\n> information is stored in an alternate _and_ more information is stored\n> outside the alternate. My perspective is biased, because I consider the\n> alternate to be \"almost everything\" and the local object store to be\n> small. But in a fork network, this is not always the case. I appreciate\n> your feedback for this environment, and I've always hoped that someone\n> with server experience would come and say \"this feature is great, but\n> we need X, Y, and Z to make best use of it in our environment. Here's\n> a patch that moves us in that direction!\" At least you are doing the\n> next-best thing: stopping me from making mistakes that would block\n> adoption.\n\nI'm happy to write some patches, just want to talk about it first (and\nif I'm lucky convince you to write them for me :) ).\n\nOne example is the git-for-windows commit-graph is 5.6MB, the git.git\none is 3.1MB. Should GitHub just stick that all in the one \"parent\"\ngraph, maybe. But nice to have the flexibility of stacking them.\n\nThere's also more disconnected cases, e.g. I have some \"staging\" boxes\nwhere where I have a cronjob running around re-pointing clones of a big\nmonorepo to a shared \"alternates\" store where I guarantee objects are\nonly ever added, never removed.\n\nIt would be nice to have a way to provide a commit-graph there that's\n\"stable\" that clients could point to, and they'd just generate the\ndifference.\n\nI.e. now I have a shared .git/objects which contains gigabytes, a\ncrapload of stuff in /home where .git/objects is 10-50MB, and each one\nhas a commit-graph that's around the same 50-100MB size (since it needs\nto contain the metadata for the full set).\n\n> So let's consider how Option 2 would work in this \"multi-tip\" case.\n> Each object directory would have some number of graph files, and one\n> 'commit-graphs/info' file pointing to some hash. When we read, we\n> try to pick the info file that is \"closest\" to us.\n>\n> This does create some complications that I don't think you gave enough\n> attention to. These may be solvable, but they are non-trivial:\n>\n> * When we 'gc' the \"core\" repo, we need to enumerate all of the\n> \"leaf\" repos to check their tip commit-graph files and make a\n> decision if we should keep their bases around or delete those tips.\n> Perhaps I'm over-stating the difficulty here, since we need to do\n> something similar to find still-reachable objects, anyway. But if\n> we are doing that reachability calculation, then why are we not\n> just putting all of the commit-graph data in the core repo? Note\n> that we don't get the same value as delta islands because this data\n> isn't being shared across the protocol. The issue with storing all\n> graph data in the core repo is that the core repo doesn't actually\n> have all of the commits, which makes 'verify' on the graph a bit\n> annoying.\n\nYeah I think similar to \"alternates\" it would be annoying to have a case\nwhere a given repo has metadata on objects it doesn't have, and there's\ncases (see the \"staging\" case I mentioned above) where that \"parent\"\nrepo won't have access to those things.\n\n> * If we choose a local tip instead of the \"core\" tip, then that chain\n> of commit-graphs can be far behind the core repo. In the world where\n> a fork moves only at the speed of a single developer, but the core\n> project moves quickly, then computing a merge base with the core's\n> master branch becomes slow as our local chain doesn't contain most\n> of the commits.\n\nThat's a good point and a case where pointing \"upwards\" or just having\ncommit-graphs in the \"base\" repo is better, i.e. the \"fork\" has almost\nno objects.\n\nBut solvable by triggering \"gc\" on these child projects so their\ncommit-graph keeps being re-pointed to a later version.\n\nAnd we'd have the reverse problem with a git-for-windows wouldn't we?\nI.e. the fork is \"far ahead\".\n\n> * We can't take all of the \"core\" chain _and_ the local chain, because\n> the concept of \"graph position\" no longer makes sense. The only way\n> I see out of this is to make the graph position two-dimensional:\n> commit -> (index of tip chain, position in that chain). Perhaps this\n> is a valuable thing to do in the future? Or perhaps, we shouldn't\n> have incremental chains spanning object directories and instead\n> introduce \"parents-by-ref\" where we mark some parents as included\n> by object id instead of by graph position. This would allow the\n> core repo to gc without caring about the external repos. It also\n> wouldn't care about how the graph files are stored (Option 0 would\n> work, as graph chains would not cross object store boundaries) and\n> more closely resembles the independence of the pack-files in each\n> object store. The \"parents-by-ref\" would require updating the\n> file format version.\n\nThe parent repo can \"gc\" without caring/inspecting \"child\" repos with\nthe \"point downwards\" of option #2, as long as it promises to retain its\nold commit-graph files for some retention period of X, and the \"child\"\nrepos promise to \"gc\" (and re-point to new graphs if necessary) at rate\nthat's faster than that.\n\nThis makes it easy to e.g. say \"we retain old commit-graph files for 2\nweeks\", and \"we re-gc everything in cron weekly\".\n\nIt would work best if we can also pull this trick on the \"base\"\ncommit-graph file, which I believe we could do in a backwards-compatible\nway by making \"commit-graph\" a symlink to whatever \"commit-graph-\"\nis the current \"base\".\n\n> --[end discussion of incremental files]--\n>\n> I'll move forward applying your existing feedback on patches 1-11 and\n> submit as a full series to replace ds/commit-graph-format-v2. We can\n> work on reviewing that code while we continue to think critically on\n> the topic of incremental files.\n>\n> Thanks,\n> -Stolee\n"},{"id":"375243","messageId":"caefe73d-828f-5668-91bf-1f5a82931ad1@gmail.com","threadId":"51052","inReplyTo":"875zqjy6ga.fsf@evledraar.gmail.com","subject":"Re: [PATCH 12/17] Documentation: describe split commit-graphs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-09T17:08:54Z","receivedAt":"2019-05-09T17:09:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/9/2019 11:48 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, May 09 2019, Derrick Stolee wrote:\n> [snip]\n>>\n>> I still don't think this is a problem, with the idea below.\n>>\n>>>> 4. It would be helpful to add a new optional chunk that contains the\n>>>> trailing hash for the lower level of the commit-graph stack. This chunk\n>>>> would only be for the commit-graph-N files, and would provide a simple\n>>>> way to check that the stack is valid on read, in case we are still\n>>>> worried about other processes reading/writing in the wrong order.\n>>>>\n>>>> If we have this chunk -- you have convinced me that we need it -- then we\n>>>> could ignore the \"new\" commit-graph-2 because its base graph hash does not\n>>>> match. We can continue without dying because we can always parse the \"missing\"\n>>>> commits from the packs.\n>>\n>> So, let's set that idea aside. You have other concerns.\n>>\n>>> Instead of looping through files N at a time we'd have a discovery step\n>>> where we'd need to open() all the files, see which ones say \"my parent\n>>> hash hash X\", and then create a list of those hashes in order to read a\n>>> bunch of commit-graph- files.\n>>>\n>>> Is that a bit painful? Sure, but way less painful than dealing with the\n>>> caveats I'd mentioned in [1] and SZEDER details here.\n>>\n>> I don't see how this step is less painful than the one I am describing.\n>> You'll need to be a bit more specific to convince me.\n>>\n>> I'll try to be specific with a few ideas that have been thrown around,\n>> so we can compare and contrast (and point out where I am misunderstanding\n>> what you are trying to say).\n>>\n>> Option 0 (this series): commit-graph-N\n>> --------------------------------------\n>>\n>> On read, we look for the 'info/commit-graph' file and acquire a read handle.\n>> We set that commit_graph struct as our tip commit-graph. Then, for each\n>> increasing N (until we fail) we acquire a read handle on\n>> 'info/commit-graphs/commit-graph-N' and check that its base hash matches\n>> our current tip commit-graph. If the file doesn't exist, or the base\n>> hash doesn't match, then we stop and continue with our current tip graph.\n> \n> This \"base hash\" is something I may have missed. I *thought* we'd\n> happy-go-lucky load e.g. commit-graph-1, commit-graph-2 in that order,\n> and if (e.g. due to fsync/dir entry issues noted earlier) if we got the\n> \"wrong\" commit-graph-2 we'd be oblivious to that. Is that not the case?> \n> I was trying (and failing) to make the test suite vomit by hacking up\n> the code to load \"commit-graph-1\" first, then \"commit-graph\", so maybe I\n> ran into that safeguard, whatever it is. I've just seen how we carry the\n> \"num_commits_in_base\" forward, not some expected checksum...\n\nIt's not implemented in the patch series code, but included as an idea\nin the cover letter, and the discussion above convinced me it should be\nrequired. The discussion here assumes that is part of the design.\n\n> >> On write, use a 'mv' to swap our .lock file with whatever level we are\n>> merging, THEN can unlink() the higher layers in decreasing order. (This\n>> \"mv-then-unlink\" order is different than what is implemented by this\n>> series, but is enabled by the chunk containing the base graph hash.)\n> \n> So one of the things that make me paranoid about this (as noted\n> upthread/earlier) is that at least on POSIX filesystems just because you\n> observe that you create and fsync \"foo\" and \"bar\" in that order doesn't\n> mean that a concurrent reader of that directory will get the updates in\n> that order. I.e. file update != dir entry update.\n\nHow far apart can these concurrency issues happen in the file system?\nOne benefit to Option 0 is that there is only one file _write_ that matters.\nThe other options require at least two writes.\n\n> It can be made to work, and core.fsyncObjectFiles is a partial solution,\n> but better to avoid it entirely. Also the failure with\n> core.fsyncObjectFiles is much more graceful, since in that case we don't\n> have a \"deadbeef[...]\" loose object *yet* but will (presumably) see the\n> *right* thing Real Soon Now since it's content-addressable, whereas with\n> this design we might see the *wrong* thing. So this would be the first\n> area of git (that I know of) where we'd be sensitive to a combination of\n> syncing dir entries and file content in order.\n> \n>> Option 1 (needs format v2): commit-graph -> graph-{hash}.graph\n>> --------------------------------------------------------------\n>>\n>> On read, we load the 'info/commit-graph' file and inspect the byte saying\n>> how many base files we have. We load their hashes from the base file chunk\n>> and read 'info/graph-{hash}.graph' for each. If _any_ fail, then we need\n>> to ignore anything \"above\" the failure in the chain (and specifically this\n>> includes the 'commit-graph' file), or consider reloading the commit-graph\n>> file altogether and hope it works this time. [Note: if an older version\n>> of Git doesn't understand the incremental file format, it will fail to\n>> properly understand the graph positions and either fail with an\n>> \"invalid parent position\" error, or worse give garbage results.]\n> \n> Right, except as noted before I'd add (at least per my understanding so\n> far) \"s/needs format v2//\". I.e. we could stick this data in new v1\n> backwards-compatible chunks, which also addresses the \"older version...\"\n> caveat.\n> \n> I.e. we'd always guarantee that the \"commit-graph\" file was\n> understandable/valid for older versions (even though eventually their\n> view on it would be \"oh, this has no data\" (but really it's all in a new\n> chunk they don't grok).\n\nI see. Something that I gather from your paragraph above but cannot find\nwritten explicitly anywhere is \"write a commit-graph with zero commits,\nand it contains extra information that we can use to find the rest of the\ncommits.\" In essence, the entire purpose of the file would be to contain\nthe information of the 'commit-graphs/info' file from Option 2. It would\nmean we don't need to \"copy then move\" because the tip commit-graph file\nhas no content to persist (except for the first incremental write).\n\nThat would prevent breaking old clients, but they would also not have any\ncommit-graph data to use. Option 0 and Option 2 can leave a valid v1\ncommit-graph file with the majority of the commit data. This is only a\nperformance issue for a narrow case, so maybe that's worth ignoring.\n\n(See Ævar's idea about symlinks at the bottom of this message for more.)\n\n>> On write, if we are creating a new layer to our chain, we need to _copy_\n>> the existing commit-graph file to a graph-{hash}.graph file before renaming\n>> the .lock file. If we are merging layers, then we either (a) clean up the\n>> dangling chains after moving our commit-graph file, or (b) have something\n>> like 'gc' clean up the files later. I think using 'gc' for this is not\n>> a good idea, since I would expect these files to be written and merged\n>> much more frequently (say, after a 'fetch') than a 'gc' is run. Cleaning\n>> up the dangling chains leads to our concurrency issues. Further, this\n>> 'copy the base' no longer keeps our large base file at rest.\n> \n> So aside from the specifics of implementation both this and the\n> commit-graph-N way of doing it involves juggling a sequence of chunks\n> that \"point upwards\", which IMO has worse caveats than \"point\n> downwards\". I.e. we'd need to rewrite \"base\" files to point \"upwards\" to\n> new stuff, and there's no (sane) way in option 0 to split commit-graphs\n> across repos, and no way at all in this scenario (if I understand it\n> correctly...).\n\nI don't understand the value of rewriting base files to point upwards.\nThe whole point is to not change the base files.\n\n>> Option 2: grovel commit-graphs directory for graph-{hash}.graph\n>> ---------------------------------------------------------------\n>>\n>> On read, we load the 'info/commit-graph' file and assume it is never\n>> an incremental file. Then, scan the 'info/commit-graphs' directory\n>> for 'graph-{hash}.graph' files and open them _all_ up to construct\n>> a \"graph forest\" (each graph has a single parent, given by a chunk\n>> specifying its base graph hash). If we don't have an instance of a\n>> graph with a given hash, then ignore any graphs pointing to that hash.\n>> We now have a decision to make: which leaf of this forest should we\n>> use as our tip commit-graph? That could be given by the\n>> commit-graphs/info file. But what happens when we have timing issues\n>> around scanning the directory and the commit-graphs/info file? Do\n>> we fall back to modified time?\n> \n> I think in a worst-case scenario we shrug and just pick whatever chain\n> looks most recent/largest or whatever, but in practice I think it's a\n> non-issue.\n> \n> Critical for that \"it's a non-issue\" is the suggestion I had on 17/17 of\n> not doing the incremental write in \"fetch\", but instead just rely on \"gc\n> --auto\" *and* to address your \"I would expect these files to be written\n> and merged much more frequently\" we'd just (and I'm keen to hack this\n> up) teach \"gc --auto\" such an incremental mode.\n\nI look forward to this incremental mode, and its \"partial maintenance\".\n\n> This means that within a single repo all updates to the commit-graph\n> will go through the gc.lock, whereas with your current 17/17 we'd\n> potentially have a \"commit-graph write\" racing a concurrent \"gc\", with\n> both aiming to update the commit-graph.\n\nExcept for writes that happen in `git commit-graph write`, of course.\n\nVFS for Git never runs 'gc' and instead maintains the commit-graph directly.\n\nShould the commit-graph builtin take a gc lock on write?\n\n>> On write, if we are not merging, then we just create a new\n>> graph-{hash}.graph file. If we are merging, but still have a base graph,\n>> then create a new graph-{hash}.graph file. Finally, if we are merging\n>> all layers, then we rename our .lock file to 'info/commit-graph'.\n>> To clean up, we need to grovel the directory to look for graph-{hash}.graph\n>> files whose base chains no longer match the new, \"best\" chain and unlink()\n>> them. This clean-up step can happen at any time.\n>>\n>> --[end description of options]--\n>>\n>> Did I accurately describe the options we are considering?\n>>\n>> Option 1 was the design I was planning, and I think it matches how the\n>> split-index feature works. Please correct me if I am missing something.\n>> It _requires_ updating the file format version. But it also has a flaw\n>> that the other options do not have: the copy of the base file. One\n>> thing I want to enable is for whatever machinery is handling these\n>> file writes to run a 'verify' immediately after, and have that be fast\n>> most of the time. With a model that changes only the \"tip\" file, we\n>> can verify only the new files and have confidence that the base file\n>> did not change. I think options 0 and 2 both improve in this direction.\n>>\n>>> With commit-graph- all these unlink() race conditions go away,\n>>> partial reads due to concurrent graph writing becomes a non-issue (we'd\n>>> just leave the old files, \"gc\" deals with them later..), no need to\n>>> carefully fsync() files/dirs etc as we need to carefully juggle N and\n>>> N+1 files.\n>>\n>> Calling this a non-issue is an exaggeration, especially if you are\n>> claiming we need to be robust to multi-hour gaps between reading files.\n> \n> We always have a race, but they're very different races.\n> \n> With \"N\" files (option #0) we'd have races of the type where within the\n> same few milliseconds of a commit-graph being merged/updated a\n> e.g. concurrent \"tag --contains\" would either be much slower (couldn't\n> get one of the incremental graphs it expected), or produce the wrong\n> answer (this may be wrong on my part, see my \"base hash\" comment above).\n\nWith the \"base hash\" feature (planned, not implemented) we won't be wrong.\n\nWith the \"max number of commits in the incremental files\" feature, we\nwould limit the performance issues from missing an update. BUT, this also\nimplies we are rewriting the base commit-graph more often. With a limit\nof 64,000 commits, that's still only once every few weeks in the Windows\nOS repo.\n\n> Whereas if I run something with \"ionice -c 3\" I could possibly hang for\n> however many hours/days/weeks we wait until another \"gc\" comes along and\n> unlinks those old files, but if I'm running it like that I'm not\n> expecting it to be fast, so it's OK if the files went away, and it won't\n> ever get the wrong file (since the filenames are hash-addressible).\n\nHow do we recover from this situation? Have no commit-graph at all? Or\ndo we try again?\n\n>>> It also becomes easy to \"chain\" graphs across repos e.g. via\n>>> alternates. Say in the scenario github/gitlab have where they have a\n>>> \"main\" repo and other objects on another delta island.\n>>>\n>>> In that case the repo would have a local \"tip\" file with the last link\n>>> in its chain, some of which would then refer back to in other\n>>> \"parent\" alternates.\n>>>\n>>> As long as such a setup has a \"gc\" process that's not overly eager about\n>>> pruning old stuff and considers that constellation of repos as a whole\n>>> that should just work. You can freely optimize and rewrite graphs across\n>>> repos, just be careful about unlinking old stuff.\n>>>\n>>> I don't see how it would work with commit-graph-N without a *lot* of\n>>> painful orchestration (where e.g. you *must* guarantee that the parent\n>>> repo ends in N, all child repos start at N+1).\n>>\n>> You're right that Option 0 does not work in this model where some graph\n>> information is stored in an alternate _and_ more information is stored\n>> outside the alternate. My perspective is biased, because I consider the\n>> alternate to be \"almost everything\" and the local object store to be\n>> small. But in a fork network, this is not always the case. I appreciate\n>> your feedback for this environment, and I've always hoped that someone\n>> with server experience would come and say \"this feature is great, but\n>> we need X, Y, and Z to make best use of it in our environment. Here's\n>> a patch that moves us in that direction!\" At least you are doing the\n>> next-best thing: stopping me from making mistakes that would block\n>> adoption.\n> \n> I'm happy to write some patches, just want to talk about it first (and\n> if I'm lucky convince you to write them for me :) ).\n> \n> One example is the git-for-windows commit-graph is 5.6MB, the git.git\n> one is 3.1MB. Should GitHub just stick that all in the one \"parent\"\n> graph, maybe. But nice to have the flexibility of stacking them.\n> \n> There's also more disconnected cases, e.g. I have some \"staging\" boxes\n> where where I have a cronjob running around re-pointing clones of a big\n> monorepo to a shared \"alternates\" store where I guarantee objects are\n> only ever added, never removed.\n> \n> It would be nice to have a way to provide a commit-graph there that's\n> \"stable\" that clients could point to, and they'd just generate the\n> difference.\n> \n> I.e. now I have a shared .git/objects which contains gigabytes, a\n> crapload of stuff in /home where .git/objects is 10-50MB, and each one\n> has a commit-graph that's around the same 50-100MB size (since it needs\n> to contain the metadata for the full set).\n>\n>> So let's consider how Option 2 would work in this \"multi-tip\" case.\n>> Each object directory would have some number of graph files, and one\n>> 'commit-graphs/info' file pointing to some hash. When we read, we\n>> try to pick the info file that is \"closest\" to us.\n>>\n>> This does create some complications that I don't think you gave enough\n>> attention to. These may be solvable, but they are non-trivial:\n>>\n>> * When we 'gc' the \"core\" repo, we need to enumerate all of the\n>> \"leaf\" repos to check their tip commit-graph files and make a\n>> decision if we should keep their bases around or delete those tips.\n>> Perhaps I'm over-stating the difficulty here, since we need to do\n>> something similar to find still-reachable objects, anyway. But if\n>> we are doing that reachability calculation, then why are we not\n>> just putting all of the commit-graph data in the core repo? Note\n>> that we don't get the same value as delta islands because this data\n>> isn't being shared across the protocol. The issue with storing all\n>> graph data in the core repo is that the core repo doesn't actually\n>> have all of the commits, which makes 'verify' on the graph a bit\n>> annoying.\n> \n> Yeah I think similar to \"alternates\" it would be annoying to have a case\n> where a given repo has metadata on objects it doesn't have, and there's\n> cases (see the \"staging\" case I mentioned above) where that \"parent\"\n> repo won't have access to those things.\n> \n>> * If we choose a local tip instead of the \"core\" tip, then that chain\n>> of commit-graphs can be far behind the core repo. In the world where\n>> a fork moves only at the speed of a single developer, but the core\n>> project moves quickly, then computing a merge base with the core's\n>> master branch becomes slow as our local chain doesn't contain most\n>> of the commits.\n> \n> That's a good point and a case where pointing \"upwards\" or just having\n> commit-graphs in the \"base\" repo is better, i.e. the \"fork\" has almost\n> no objects.\n\nIs your idea of \"upwards\" different than mine? I think of pointing to\na base file as pointing \"down\", and the opposite would be \"up\". In the\ncase of a fork network or multiple repos using an alternate, pointing\nupwards would not even be well-defined.\n\n> But solvable by triggering \"gc\" on these child projects so their\n> commit-graph keeps being re-pointed to a later version.\n> \n> And we'd have the reverse problem with a git-for-windows wouldn't we?\n> I.e. the fork is \"far ahead\".\n\nThis is the quintessential example for why we can't have a single chain\nof commit-graphs long-term. It deviates from most fork networks enough\nthat we can't say \"just take the base repo's commit-graph\" but typical\nfork networks can't say \"just take my local commit-graph chain\". The\ntwo-dimensional graph position would be valuable to help both shapes.\n\n>> * We can't take all of the \"core\" chain _and_ the local chain, because\n>> the concept of \"graph position\" no longer makes sense. The only way\n>> I see out of this is to make the graph position two-dimensional:\n>> commit -> (index of tip chain, position in that chain). Perhaps this\n>> is a valuable thing to do in the future? Or perhaps, we shouldn't\n>> have incremental chains spanning object directories and instead\n>> introduce \"parents-by-ref\" where we mark some parents as included\n>> by object id instead of by graph position. This would allow the\n>> core repo to gc without caring about the external repos. It also\n>> wouldn't care about how the graph files are stored (Option 0 would\n>> work, as graph chains would not cross object store boundaries) and\n>> more closely resembles the independence of the pack-files in each\n>> object store. The \"parents-by-ref\" would require updating the\n>> file format version.\n> \n> The parent repo can \"gc\" without caring/inspecting \"child\" repos with\n> the \"point downwards\" of option #2, as long as it promises to retain its\n> old commit-graph files for some retention period of X, and the \"child\"\n> repos promise to \"gc\" (and re-point to new graphs if necessary) at rate\n> that's faster than that.\n> \n> This makes it easy to e.g. say \"we retain old commit-graph files for 2\n> weeks\", and \"we re-gc everything in cron weekly\".\n\nHere, I think, is the most crucial point of why Option 2 may be worth the\nadded complexity over Option 0. Option 0 _requires_ that the files be\nreplaced immediately on a new write, while Option 2 provides a way to\nleave old files around and be cleaned up later.\n\nBut how should we actually perform this cleanup? I would imagine a\n'git commit-graph gc' subcommand that cleans up old files. A 'git gc'\nrun would perform the same logic, but we need a way to do this outside\nof 'gc'. It needs to use the modified time as an indicator, since we\ncould run 'git commit-graph write' twice an hour before our two-week\ncleanup job and need to keep our hour-old stale file. Perhaps the\n'git commit-graph gc' subcommand could take a '--window' parameter that\ncan be 0, while 'git gc' uses a config setting.\n\nThe decision can then be: \"is this file not in our graph chain and\nolder than from now?\"\n\nBut also, I expect the stale commit-graph files will pile up quickly.\nWe rebuild the commit-graph file roughly every hour. I would write\nour maintenance to call these subcommands in order with no delay:\n\n \"write\" -> \"verify --shallow\" -> \"gc --window=0\"\n\n(Here, \"verify --shallow\" would only verify the tip of the\ncommit-graph chain.)\n\n> It would work best if we can also pull this trick on the \"base\"\n> commit-graph file, which I believe we could do in a backwards-compatible\n> way by making \"commit-graph\" a symlink to whatever \"commit-graph-\"\n> is the current \"base\".\n\nCould we do this, anyway? Use 'commit-graphs/info' to point to the tip\nand let the symlink 'commit-graph' point to the base. Then, old clients\nwould load a full commit-graph and new clients would get the full chain.\n"},{"id":"375271","messageId":"874l63xpwx.fsf@evledraar.gmail.com","threadId":"51052","inReplyTo":"caefe73d-828f-5668-91bf-1f5a82931ad1@gmail.com","subject":"Re: [PATCH 12/17] Documentation: describe split commit-graphs","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-05-09T21:45:50Z","receivedAt":"2019-05-09T21:45:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, May 09 2019, Derrick Stolee wrote:\n\n> On 5/9/2019 11:48 AM, Ævar Arnfjörð Bjarmason wrote:\n>>\n>> On Thu, May 09 2019, Derrick Stolee wrote:\n>> [snip]\n>>>\n>>> I still don't think this is a problem, with the idea below.\n>>>\n>>>>> 4. It would be helpful to add a new optional chunk that contains the\n>>>>> trailing hash for the lower level of the commit-graph stack. This chunk\n>>>>> would only be for the commit-graph-N files, and would provide a simple\n>>>>> way to check that the stack is valid on read, in case we are still\n>>>>> worried about other processes reading/writing in the wrong order.\n>>>>>\n>>>>> If we have this chunk -- you have convinced me that we need it -- then we\n>>>>> could ignore the \"new\" commit-graph-2 because its base graph hash does not\n>>>>> match. We can continue without dying because we can always parse the \"missing\"\n>>>>> commits from the packs.\n>>>\n>>> So, let's set that idea aside. You have other concerns.\n>>>\n>>>> Instead of looping through files N at a time we'd have a discovery step\n>>>> where we'd need to open() all the files, see which ones say \"my parent\n>>>> hash hash X\", and then create a list of those hashes in order to read a\n>>>> bunch of commit-graph- files.\n>>>>\n>>>> Is that a bit painful? Sure, but way less painful than dealing with the\n>>>> caveats I'd mentioned in [1] and SZEDER details here.\n>>>\n>>> I don't see how this step is less painful than the one I am describing.\n>>> You'll need to be a bit more specific to convince me.\n>>>\n>>> I'll try to be specific with a few ideas that have been thrown around,\n>>> so we can compare and contrast (and point out where I am misunderstanding\n>>> what you are trying to say).\n>>>\n>>> Option 0 (this series): commit-graph-N\n>>> --------------------------------------\n>>>\n>>> On read, we look for the 'info/commit-graph' file and acquire a read handle.\n>>> We set that commit_graph struct as our tip commit-graph. Then, for each\n>>> increasing N (until we fail) we acquire a read handle on\n>>> 'info/commit-graphs/commit-graph-N' and check that its base hash matches\n>>> our current tip commit-graph. If the file doesn't exist, or the base\n>>> hash doesn't match, then we stop and continue with our current tip graph.\n>>\n>> This \"base hash\" is something I may have missed. I *thought* we'd\n>> happy-go-lucky load e.g. commit-graph-1, commit-graph-2 in that order,\n>> and if (e.g. due to fsync/dir entry issues noted earlier) if we got the\n>> \"wrong\" commit-graph-2 we'd be oblivious to that. Is that not the case?>\n>> I was trying (and failing) to make the test suite vomit by hacking up\n>> the code to load \"commit-graph-1\" first, then \"commit-graph\", so maybe I\n>> ran into that safeguard, whatever it is. I've just seen how we carry the\n>> \"num_commits_in_base\" forward, not some expected checksum...\n>\n> It's not implemented in the patch series code, but included as an idea\n> in the cover letter, and the discussion above convinced me it should be\n> required. The discussion here assumes that is part of the design.\n\nGot it. Hashing like that would definitely mitigate the \"wrong file\"\nfailure scenario, although as discussed leaving some others...\n\n>> >> On write, use a 'mv' to swap our .lock file with whatever level we are\n>>> merging, THEN can unlink() the higher layers in decreasing order. (This\n>>> \"mv-then-unlink\" order is different than what is implemented by this\n>>> series, but is enabled by the chunk containing the base graph hash.)\n>>\n>> So one of the things that make me paranoid about this (as noted\n>> upthread/earlier) is that at least on POSIX filesystems just because you\n>> observe that you create and fsync \"foo\" and \"bar\" in that order doesn't\n>> mean that a concurrent reader of that directory will get the updates in\n>> that order. I.e. file update != dir entry update.\n>\n> How far apart can these concurrency issues happen in the file system?\n> One benefit to Option 0 is that there is only one file _write_ that matters.\n> The other options require at least two writes.\n\nYou need to write() and then fsync()/close() to be guaranteed that the\ndata was written to the file.\n\nAs an aside I see that the current commit-graph uses CSUM_FSYNC (but\nwithout CSUM_CLOSE, probably doesn't matter), I thought it\ndidn't. Maybe we should remove that unless core.fsyncObjectFiles=true\n(or have another \"looser\" fsync config).\n\nSo writing is cheap, but it's asking the OS to sync to disk that\ngenerally hurts. As noted in the discussions when core.fsyncObjectFiles\nwas introduced some FSs are really stupid about it, so it's better to\navoid these caveats if you can.\n\nAs noted in the fsync(2) manpage on Linux (ditto POSIX) an fsync to a\n*file* doesn't guarantee that the directory entry is updated. So we'd\nalso need to opendir() the containing directory and fsync that FD as we\njuggle these renames/replaces.\n\nBut stepping away from strict POSIX for a second, many users of git use\nthe object store over something like NFS. It's very handy to be able to\nmount such a volume with e.g. lookupcache=positive\n(https://linux.die.net/man/5/nfs).\n\nI.e. if you request a commit-graph- you don't need a round-trip to\nthe server to server to see if it's still there.\n\nSo it's not that we can't do it, but rather that there's some usage\npatterns that are much friendlier to performance & caching than\nothers. Just throwing things at the FS and having it flush stuff in its\nown time is ideal, and e.g. commercial FS appliances have a lot of\nperformance knobs you can use if you have access patterns like what we\nhave in objects/pack/.\n\nI.e. \"any file here you can cache forever, they never change (they\ndisappear, but no hurry!), but new ones might show up\".\n\nThat sort of thing becomes more true with other dumber FS-alike things,\ne.g. if we go for some \"grab a graph from the server\" implementation\nsuch a thing is way simpler as a list of hashes that can be cached\nforever (http proxies et al) than N files sensitive to their update\nsequence.\n\n>> It can be made to work, and core.fsyncObjectFiles is a partial solution,\n>> but better to avoid it entirely. Also the failure with\n>> core.fsyncObjectFiles is much more graceful, since in that case we don't\n>> have a \"deadbeef[...]\" loose object *yet* but will (presumably) see the\n>> *right* thing Real Soon Now since it's content-addressable, whereas with\n>> this design we might see the *wrong* thing. So this would be the first\n>> area of git (that I know of) where we'd be sensitive to a combination of\n>> syncing dir entries and file content in order.\n>>\n>>> Option 1 (needs format v2): commit-graph -> graph-{hash}.graph\n>>> --------------------------------------------------------------\n>>>\n>>> On read, we load the 'info/commit-graph' file and inspect the byte saying\n>>> how many base files we have. We load their hashes from the base file chunk\n>>> and read 'info/graph-{hash}.graph' for each. If _any_ fail, then we need\n>>> to ignore anything \"above\" the failure in the chain (and specifically this\n>>> includes the 'commit-graph' file), or consider reloading the commit-graph\n>>> file altogether and hope it works this time. [Note: if an older version\n>>> of Git doesn't understand the incremental file format, it will fail to\n>>> properly understand the graph positions and either fail with an\n>>> \"invalid parent position\" error, or worse give garbage results.]\n>>\n>> Right, except as noted before I'd add (at least per my understanding so\n>> far) \"s/needs format v2//\". I.e. we could stick this data in new v1\n>> backwards-compatible chunks, which also addresses the \"older version...\"\n>> caveat.\n>>\n>> I.e. we'd always guarantee that the \"commit-graph\" file was\n>> understandable/valid for older versions (even though eventually their\n>> view on it would be \"oh, this has no data\" (but really it's all in a new\n>> chunk they don't grok).\n>\n> I see. Something that I gather from your paragraph above but cannot find\n> written explicitly anywhere is \"write a commit-graph with zero commits,\n> and it contains extra information that we can use to find the rest of the\n> commits.\" In essence, the entire purpose of the file would be to contain\n> the information of the 'commit-graphs/info' file from Option 2. It would\n> mean we don't need to \"copy then move\" because the tip commit-graph file\n> has no content to persist (except for the first incremental write).\n>\n> That would prevent breaking old clients, but they would also not have any\n> commit-graph data to use. Option 0 and Option 2 can leave a valid v1\n> commit-graph file with the majority of the commit data. This is only a\n> performance issue for a narrow case, so maybe that's worth ignoring.\n\nIndeed. As noted in the thread about the v1->v2 format if we just append\nnew chunks we have leeway like that, i.e. it's completely OK if we\nchoose to from the POV of old clients write no data to them so they're\nnot helped by the optimization, that's a more graceful way than them\ndying on a format change.\n\nAnd yes, we could stick the proposed commit-graphs/info \"index\" in a\nchunk there. I've got a preference for a \\n-delimited list similar to\nobjects/info/packs, but that's mostly for aesthetic reasons.\n\n> (See Ævar's idea about symlinks at the bottom of this message for more.)\n>\n>>> On write, if we are creating a new layer to our chain, we need to _copy_\n>>> the existing commit-graph file to a graph-{hash}.graph file before renaming\n>>> the .lock file. If we are merging layers, then we either (a) clean up the\n>>> dangling chains after moving our commit-graph file, or (b) have something\n>>> like 'gc' clean up the files later. I think using 'gc' for this is not\n>>> a good idea, since I would expect these files to be written and merged\n>>> much more frequently (say, after a 'fetch') than a 'gc' is run. Cleaning\n>>> up the dangling chains leads to our concurrency issues. Further, this\n>>> 'copy the base' no longer keeps our large base file at rest.\n>>\n>> So aside from the specifics of implementation both this and the\n>> commit-graph-N way of doing it involves juggling a sequence of chunks\n>> that \"point upwards\", which IMO has worse caveats than \"point\n>> downwards\". I.e. we'd need to rewrite \"base\" files to point \"upwards\" to\n>> new stuff, and there's no (sane) way in option 0 to split commit-graphs\n>> across repos, and no way at all in this scenario (if I understand it\n>> correctly...).\n>\n> I don't understand the value of rewriting base files to point upwards.\n> The whole point is to not change the base files.\n\nYeah I don't think it makes sense, but maybe I misread the\ndescription...\n\n>>> Option 2: grovel commit-graphs directory for graph-{hash}.graph\n>>> ---------------------------------------------------------------\n>>>\n>>> On read, we load the 'info/commit-graph' file and assume it is never\n>>> an incremental file. Then, scan the 'info/commit-graphs' directory\n>>> for 'graph-{hash}.graph' files and open them _all_ up to construct\n>>> a \"graph forest\" (each graph has a single parent, given by a chunk\n>>> specifying its base graph hash). If we don't have an instance of a\n>>> graph with a given hash, then ignore any graphs pointing to that hash.\n>>> We now have a decision to make: which leaf of this forest should we\n>>> use as our tip commit-graph? That could be given by the\n>>> commit-graphs/info file. But what happens when we have timing issues\n>>> around scanning the directory and the commit-graphs/info file? Do\n>>> we fall back to modified time?\n>>\n>> I think in a worst-case scenario we shrug and just pick whatever chain\n>> looks most recent/largest or whatever, but in practice I think it's a\n>> non-issue.\n>>\n>> Critical for that \"it's a non-issue\" is the suggestion I had on 17/17 of\n>> not doing the incremental write in \"fetch\", but instead just rely on \"gc\n>> --auto\" *and* to address your \"I would expect these files to be written\n>> and merged much more frequently\" we'd just (and I'm keen to hack this\n>> up) teach \"gc --auto\" such an incremental mode.\n>\n> I look forward to this incremental mode, and its \"partial maintenance\".\n>\n>> This means that within a single repo all updates to the commit-graph\n>> will go through the gc.lock, whereas with your current 17/17 we'd\n>> potentially have a \"commit-graph write\" racing a concurrent \"gc\", with\n>> both aiming to update the commit-graph.\n>\n> Except for writes that happen in `git commit-graph write`, of course.\n>\n> VFS for Git never runs 'gc' and instead maintains the commit-graph directly.\n>\n> Should the commit-graph builtin take a gc lock on write?\n\nIf we make the file management dumb enough it shouldn't matter, we'd\nalways have a lock for updating the \"meta\" file, but if we didn't we'd\njust have some harmless duplicate work.\n\nI think a lock in \"gc\" is enough, it's our default, so it should be\nsensible. People doing custom GC generally do/want their own management\nof that, no need for us to hold their hand beyond the point where\ne.g. concurrent \"pack-refs\" might corrupt refs if the ref update itself\nisn't locked.\n\n>>> On write, if we are not merging, then we just create a new\n>>> graph-{hash}.graph file. If we are merging, but still have a base graph,\n>>> then create a new graph-{hash}.graph file. Finally, if we are merging\n>>> all layers, then we rename our .lock file to 'info/commit-graph'.\n>>> To clean up, we need to grovel the directory to look for graph-{hash}.graph\n>>> files whose base chains no longer match the new, \"best\" chain and unlink()\n>>> them. This clean-up step can happen at any time.\n>>>\n>>> --[end description of options]--\n>>>\n>>> Did I accurately describe the options we are considering?\n>>>\n>>> Option 1 was the design I was planning, and I think it matches how the\n>>> split-index feature works. Please correct me if I am missing something.\n>>> It _requires_ updating the file format version. But it also has a flaw\n>>> that the other options do not have: the copy of the base file. One\n>>> thing I want to enable is for whatever machinery is handling these\n>>> file writes to run a 'verify' immediately after, and have that be fast\n>>> most of the time. With a model that changes only the \"tip\" file, we\n>>> can verify only the new files and have confidence that the base file\n>>> did not change. I think options 0 and 2 both improve in this direction.\n>>>\n>>>> With commit-graph- all these unlink() race conditions go away,\n>>>> partial reads due to concurrent graph writing becomes a non-issue (we'd\n>>>> just leave the old files, \"gc\" deals with them later..), no need to\n>>>> carefully fsync() files/dirs etc as we need to carefully juggle N and\n>>>> N+1 files.\n>>>\n>>> Calling this a non-issue is an exaggeration, especially if you are\n>>> claiming we need to be robust to multi-hour gaps between reading files.\n>>\n>> We always have a race, but they're very different races.\n>>\n>> With \"N\" files (option #0) we'd have races of the type where within the\n>> same few milliseconds of a commit-graph being merged/updated a\n>> e.g. concurrent \"tag --contains\" would either be much slower (couldn't\n>> get one of the incremental graphs it expected), or produce the wrong\n>> answer (this may be wrong on my part, see my \"base hash\" comment above).\n>\n> With the \"base hash\" feature (planned, not implemented) we won't be wrong.\n\n*nod*\n\n> With the \"max number of commits in the incremental files\" feature, we\n> would limit the performance issues from missing an update. BUT, this also\n> implies we are rewriting the base commit-graph more often. With a limit\n> of 64,000 commits, that's still only once every few weeks in the Windows\n> OS repo.\n>\n>> Whereas if I run something with \"ionice -c 3\" I could possibly hang for\n>> however many hours/days/weeks we wait until another \"gc\" comes along and\n>> unlinks those old files, but if I'm running it like that I'm not\n>> expecting it to be fast, so it's OK if the files went away, and it won't\n>> ever get the wrong file (since the filenames are hash-addressible).\n>\n> How do we recover from this situation? Have no commit-graph at all? Or\n> do we try again?\n\nI think just no commit-graph at all is fine. If it lazily took you N\nhours from reading the \"commit-graphs/info\" to getting around to looking\nat now-disappeared files it sucks to be you.\n\n>>>> It also becomes easy to \"chain\" graphs across repos e.g. via\n>>>> alternates. Say in the scenario github/gitlab have where they have a\n>>>> \"main\" repo and other objects on another delta island.\n>>>>\n>>>> In that case the repo would have a local \"tip\" file with the last link\n>>>> in its chain, some of which would then refer back to in other\n>>>> \"parent\" alternates.\n>>>>\n>>>> As long as such a setup has a \"gc\" process that's not overly eager about\n>>>> pruning old stuff and considers that constellation of repos as a whole\n>>>> that should just work. You can freely optimize and rewrite graphs across\n>>>> repos, just be careful about unlinking old stuff.\n>>>>\n>>>> I don't see how it would work with commit-graph-N without a *lot* of\n>>>> painful orchestration (where e.g. you *must* guarantee that the parent\n>>>> repo ends in N, all child repos start at N+1).\n>>>\n>>> You're right that Option 0 does not work in this model where some graph\n>>> information is stored in an alternate _and_ more information is stored\n>>> outside the alternate. My perspective is biased, because I consider the\n>>> alternate to be \"almost everything\" and the local object store to be\n>>> small. But in a fork network, this is not always the case. I appreciate\n>>> your feedback for this environment, and I've always hoped that someone\n>>> with server experience would come and say \"this feature is great, but\n>>> we need X, Y, and Z to make best use of it in our environment. Here's\n>>> a patch that moves us in that direction!\" At least you are doing the\n>>> next-best thing: stopping me from making mistakes that would block\n>>> adoption.\n>>\n>> I'm happy to write some patches, just want to talk about it first (and\n>> if I'm lucky convince you to write them for me :) ).\n>>\n>> One example is the git-for-windows commit-graph is 5.6MB, the git.git\n>> one is 3.1MB. Should GitHub just stick that all in the one \"parent\"\n>> graph, maybe. But nice to have the flexibility of stacking them.\n>>\n>> There's also more disconnected cases, e.g. I have some \"staging\" boxes\n>> where where I have a cronjob running around re-pointing clones of a big\n>> monorepo to a shared \"alternates\" store where I guarantee objects are\n>> only ever added, never removed.\n>>\n>> It would be nice to have a way to provide a commit-graph there that's\n>> \"stable\" that clients could point to, and they'd just generate the\n>> difference.\n>>\n>> I.e. now I have a shared .git/objects which contains gigabytes, a\n>> crapload of stuff in /home where .git/objects is 10-50MB, and each one\n>> has a commit-graph that's around the same 50-100MB size (since it needs\n>> to contain the metadata for the full set).\n>>\n>>> So let's consider how Option 2 would work in this \"multi-tip\" case.\n>>> Each object directory would have some number of graph files, and one\n>>> 'commit-graphs/info' file pointing to some hash. When we read, we\n>>> try to pick the info file that is \"closest\" to us.\n>>>\n>>> This does create some complications that I don't think you gave enough\n>>> attention to. These may be solvable, but they are non-trivial:\n>>>\n>>> * When we 'gc' the \"core\" repo, we need to enumerate all of the\n>>> \"leaf\" repos to check their tip commit-graph files and make a\n>>> decision if we should keep their bases around or delete those tips.\n>>> Perhaps I'm over-stating the difficulty here, since we need to do\n>>> something similar to find still-reachable objects, anyway. But if\n>>> we are doing that reachability calculation, then why are we not\n>>> just putting all of the commit-graph data in the core repo? Note\n>>> that we don't get the same value as delta islands because this data\n>>> isn't being shared across the protocol. The issue with storing all\n>>> graph data in the core repo is that the core repo doesn't actually\n>>> have all of the commits, which makes 'verify' on the graph a bit\n>>> annoying.\n>>\n>> Yeah I think similar to \"alternates\" it would be annoying to have a case\n>> where a given repo has metadata on objects it doesn't have, and there's\n>> cases (see the \"staging\" case I mentioned above) where that \"parent\"\n>> repo won't have access to those things.\n>>\n>>> * If we choose a local tip instead of the \"core\" tip, then that chain\n>>> of commit-graphs can be far behind the core repo. In the world where\n>>> a fork moves only at the speed of a single developer, but the core\n>>> project moves quickly, then computing a merge base with the core's\n>>> master branch becomes slow as our local chain doesn't contain most\n>>> of the commits.\n>>\n>> That's a good point and a case where pointing \"upwards\" or just having\n>> commit-graphs in the \"base\" repo is better, i.e. the \"fork\" has almost\n>> no objects.\n>\n> Is your idea of \"upwards\" different than mine? I think of pointing to\n> a base file as pointing \"down\", and the opposite would be \"up\". In the\n> case of a fork network or multiple repos using an alternate, pointing\n> upwards would not even be well-defined.\n\nWe've got the same idea, but as noted above maybe I misunderstood the\n\"option 1\" description.\n\n>> But solvable by triggering \"gc\" on these child projects so their\n>> commit-graph keeps being re-pointed to a later version.\n>>\n>> And we'd have the reverse problem with a git-for-windows wouldn't we?\n>> I.e. the fork is \"far ahead\".\n>\n> This is the quintessential example for why we can't have a single chain\n> of commit-graphs long-term. It deviates from most fork networks enough\n> that we can't say \"just take the base repo's commit-graph\" but typical\n> fork networks can't say \"just take my local commit-graph chain\". The\n> two-dimensional graph position would be valuable to help both shapes.\n\nIndeed. I just thought about e.g. a [branch|tag] --contains for the\ncurrent repo, but for things like \"how ahead of gitster/pu is my GFW\ntopic?\" one graph for the network is needed.\n\nIf we can help it it would be useful to not unduly box the user in and\noffer them flexibility to choose. I.e. some (e.g. my staging server\nuse-case) might want base+local repo and never need 2x local repos\nv.s. each other, whereas github/gitlab might need one giant graph etc.\n\n>>> * We can't take all of the \"core\" chain _and_ the local chain, because\n>>> the concept of \"graph position\" no longer makes sense. The only way\n>>> I see out of this is to make the graph position two-dimensional:\n>>> commit -> (index of tip chain, position in that chain). Perhaps this\n>>> is a valuable thing to do in the future? Or perhaps, we shouldn't\n>>> have incremental chains spanning object directories and instead\n>>> introduce \"parents-by-ref\" where we mark some parents as included\n>>> by object id instead of by graph position. This would allow the\n>>> core repo to gc without caring about the external repos. It also\n>>> wouldn't care about how the graph files are stored (Option 0 would\n>>> work, as graph chains would not cross object store boundaries) and\n>>> more closely resembles the independence of the pack-files in each\n>>> object store. The \"parents-by-ref\" would require updating the\n>>> file format version.\n>>\n>> The parent repo can \"gc\" without caring/inspecting \"child\" repos with\n>> the \"point downwards\" of option #2, as long as it promises to retain its\n>> old commit-graph files for some retention period of X, and the \"child\"\n>> repos promise to \"gc\" (and re-point to new graphs if necessary) at rate\n>> that's faster than that.\n>>\n>> This makes it easy to e.g. say \"we retain old commit-graph files for 2\n>> weeks\", and \"we re-gc everything in cron weekly\".\n>\n> Here, I think, is the most crucial point of why Option 2 may be worth the\n> added complexity over Option 0. Option 0 _requires_ that the files be\n> replaced immediately on a new write, while Option 2 provides a way to\n> leave old files around and be cleaned up later.\n>\n> But how should we actually perform this cleanup? I would imagine a\n> 'git commit-graph gc' subcommand that cleans up old files. A 'git gc'\n> run would perform the same logic, but we need a way to do this outside\n> of 'gc'. It needs to use the modified time as an indicator, since we\n> could run 'git commit-graph write' twice an hour before our two-week\n> cleanup job and need to keep our hour-old stale file. Perhaps the\n> 'git commit-graph gc' subcommand could take a '--window' parameter that\n> can be 0, while 'git gc' uses a config setting.\n>\n> The decision can then be: \"is this file not in our graph chain and\n> older than from now?\"\n>\n> But also, I expect the stale commit-graph files will pile up quickly.\n> We rebuild the commit-graph file roughly every hour. I would write\n> our maintenance to call these subcommands in order with no delay:\n>\n> \"write\" -> \"verify --shallow\" -> \"gc --window=0\"\n>\n> (Here, \"verify --shallow\" would only verify the tip of the\n> commit-graph chain.)\n\nI think a sane default would be to just unlink() the old ones as soon as\nwe're done writing the new ones & writing the commit-graphs/info file\nsaying \"here's your current ones\".\n\nI.e. as noted in what I said about fsync above it's not that we need to\nkeep the old ones around, but avoiding the tight dance with N & N+1\nupdates, and being friendlier to stuff like lookupcache=positive.\n\nUsers with more advanced use-cases (e.g. cross-repo graphs) could then\nalways increase such an expiry.\n\n>> It would work best if we can also pull this trick on the \"base\"\n>> commit-graph file, which I believe we could do in a backwards-compatible\n>> way by making \"commit-graph\" a symlink to whatever \"commit-graph-\"\n>> is the current \"base\".\n>\n> Could we do this, anyway? Use 'commit-graphs/info' to point to the tip\n> and let the symlink 'commit-graph' point to the base. Then, old clients\n> would load a full commit-graph and new clients would get the full chain.\n\nHow's the Windows support for symlinks? We don't symlink anything in\n.git/objects/ ourselves now (but see[1]).\n\nOn *nix just manually symlinking it works fine (you need to go out of\nyour way not to support it, which we didn't).\n\nSo something like this would be desirable:\n\n $ tree -a .git/objects/info/\n .git/objects/info/\n ├── commit-graph -> commit-graphs/commit-graph-2492e0ef38643d4cb6369f76443e6a814d616258\n ├── commit-graphs\n │   ├── commit-graph-2492e0ef38643d4cb6369f76443e6a814d616258\n │   ├── commit-graph-988881adc9fc3655077dc2d4d757d480b5ea0e11\n │   └── info\n └── packs\n $ cat .git/objects/info/commit-graphs/info\n 2492e0ef38643d4cb6369f76443e6a814d616258\n 988881adc9fc3655077dc2d4d757d480b5ea0e11\n\nI.e. create new ones as needed, and when done say what sequence they\nshould be read in in the \"info\" file, and symlink \"commit-graph\" to\nwhatever the latest \"base\" is as a courtesy to old clients (or not, or\neventually don't bother).\n\n1. https://public-inbox.org/git/20190502144829.4394-1-matheus.bernardino@usp.br/\n"},{"id":"375291","messageId":"6d9c0911-6f36-3fb7-2be9-2be9bc68fc69@gmail.com","threadId":"51052","inReplyTo":"874l63xpwx.fsf@evledraar.gmail.com","subject":"Re: [PATCH 12/17] Documentation: describe split commit-graphs","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-10T12:44:30Z","receivedAt":"2019-05-10T12:44:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/9/2019 5:45 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, May 09 2019, Derrick Stolee wrote:\n> \n>> How far apart can these concurrency issues happen in the file system?\n>> One benefit to Option 0 is that there is only one file _write_ that matters.\n>> The other options require at least two writes.\n> \n> You need to write() and then fsync()/close() to be guaranteed that the\n> data was written to the file.\n> \n> As an aside I see that the current commit-graph uses CSUM_FSYNC (but\n> without CSUM_CLOSE, probably doesn't matter), I thought it\n> didn't. Maybe we should remove that unless core.fsyncObjectFiles=true\n> (or have another \"looser\" fsync config).\n> \n> So writing is cheap, but it's asking the OS to sync to disk that\n> generally hurts. As noted in the discussions when core.fsyncObjectFiles\n> was introduced some FSs are really stupid about it, so it's better to\n> avoid these caveats if you can.\n> \n> As noted in the fsync(2) manpage on Linux (ditto POSIX) an fsync to a\n> *file* doesn't guarantee that the directory entry is updated. So we'd\n> also need to opendir() the containing directory and fsync that FD as we\n> juggle these renames/replaces.\n\nI think any plan should not rely on directory scanning for this reason.\nStarting from a known file (info/commit-graphs/head?) and then navigating\ndirectly to files would avoid these issues, right?\n\n>> That would prevent breaking old clients, but they would also not have any\n>> commit-graph data to use. Option 0 and Option 2 can leave a valid v1\n>> commit-graph file with the majority of the commit data. This is only a\n>> performance issue for a narrow case, so maybe that's worth ignoring.\n> \n> Indeed. As noted in the thread about the v1->v2 format if we just append\n> new chunks we have leeway like that, i.e. it's completely OK if we\n> choose to from the POV of old clients write no data to them so they're\n> not helped by the optimization, that's a more graceful way than them\n> dying on a format change.\n> \n> And yes, we could stick the proposed commit-graphs/info \"index\" in a\n> chunk there. I've got a preference for a \\n-delimited list similar to\n> objects/info/packs, but that's mostly for aesthetic reasons.\n\nI think a \\n-delimited list would be better because using the commit-graph\nformat with zero commits creates at least a kilobyte of data for a\nzero-valued fanout table. If we relax the format to say \"the head file\nhas zero commits, so the fanout, commit oids, and commit data chunks\nshould not exist\" then we would be fine, and get versioning \"for free\".\nBut the file serves a different purpose.\n\n>>> Whereas if I run something with \"ionice -c 3\" I could possibly hang for\n>>> however many hours/days/weeks we wait until another \"gc\" comes along and\n>>> unlinks those old files, but if I'm running it like that I'm not\n>>> expecting it to be fast, so it's OK if the files went away, and it won't\n>>> ever get the wrong file (since the filenames are hash-addressible).\n>>\n>> How do we recover from this situation? Have no commit-graph at all? Or\n>> do we try again?\n> \n> I think just no commit-graph at all is fine. If it lazily took you N\n> hours from reading the \"commit-graphs/info\" to getting around to looking\n> at now-disappeared files it sucks to be you.\n\nSounds good. We can revisit if this actually starts hurting any real\nscenarios.\n>>> And we'd have the reverse problem with a git-for-windows wouldn't we?\n>>> I.e. the fork is \"far ahead\".\n>>\n>> This is the quintessential example for why we can't have a single chain\n>> of commit-graphs long-term. It deviates from most fork networks enough\n>> that we can't say \"just take the base repo's commit-graph\" but typical\n>> fork networks can't say \"just take my local commit-graph chain\". The\n>> two-dimensional graph position would be valuable to help both shapes.\n> \n> Indeed. I just thought about e.g. a [branch|tag] --contains for the\n> current repo, but for things like \"how ahead of gitster/pu is my GFW\n> topic?\" one graph for the network is needed.\n> \n> If we can help it it would be useful to not unduly box the user in and\n> offer them flexibility to choose. I.e. some (e.g. my staging server\n> use-case) might want base+local repo and never need 2x local repos\n> v.s. each other, whereas github/gitlab might need one giant graph etc.\n\nI made a note to follow up with the two-dimensional graph position [1].\n\n[1] https://github.com/microsoft/git/issues/138\n\n>>> This makes it easy to e.g. say \"we retain old commit-graph files for 2\n>>> weeks\", and \"we re-gc everything in cron weekly\".\n>>\n>> Here, I think, is the most crucial point of why Option 2 may be worth the\n>> added complexity over Option 0. Option 0 _requires_ that the files be\n>> replaced immediately on a new write, while Option 2 provides a way to\n>> leave old files around and be cleaned up later.\n>>\n>> But how should we actually perform this cleanup? I would imagine a\n>> 'git commit-graph gc' subcommand that cleans up old files. A 'git gc'\n>> run would perform the same logic, but we need a way to do this outside\n>> of 'gc'. It needs to use the modified time as an indicator, since we\n>> could run 'git commit-graph write' twice an hour before our two-week\n>> cleanup job and need to keep our hour-old stale file. Perhaps the\n>> 'git commit-graph gc' subcommand could take a '--window' parameter that\n>> can be 0, while 'git gc' uses a config setting.\n>>\n>> The decision can then be: \"is this file not in our graph chain and\n>> older than from now?\"\n>>\n>> But also, I expect the stale commit-graph files will pile up quickly.\n>> We rebuild the commit-graph file roughly every hour. I would write\n>> our maintenance to call these subcommands in order with no delay:\n>>\n>> \"write\" -> \"verify --shallow\" -> \"gc --window=0\"\n>>\n>> (Here, \"verify --shallow\" would only verify the tip of the\n>> commit-graph chain.)\n> \n> I think a sane default would be to just unlink() the old ones as soon as\n> we're done writing the new ones & writing the commit-graphs/info file\n> saying \"here's your current ones\".\n> \n> I.e. as noted in what I said about fsync above it's not that we need to\n> keep the old ones around, but avoiding the tight dance with N & N+1\n> updates, and being friendlier to stuff like lookupcache=positive.\n> \n> Users with more advanced use-cases (e.g. cross-repo graphs) could then\n> always increase such an expiry.\n\nNoted. Thanks.\n\n>>> It would work best if we can also pull this trick on the \"base\"\n>>> commit-graph file, which I believe we could do in a backwards-compatible\n>>> way by making \"commit-graph\" a symlink to whatever \"commit-graph-\"\n>>> is the current \"base\".\n>>\n>> Could we do this, anyway? Use 'commit-graphs/info' to point to the tip\n>> and let the symlink 'commit-graph' point to the base. Then, old clients\n>> would load a full commit-graph and new clients would get the full chain.\n> \n> How's the Windows support for symlinks? We don't symlink anything in\n> .git/objects/ ourselves now (but see[1]).\n\nMaybe this won't work in all scenarios, but it could be left as a future\nenhancement if anyone cares. The \"support two versions\" scenario is rare\nenough to avoid building something specifically for it, but often enough\nto not _break_ it.\n\n> On *nix just manually symlinking it works fine (you need to go out of\n> your way not to support it, which we didn't).\n> \n> So something like this would be desirable:\n> \n> $ tree -a .git/objects/info/\n> .git/objects/info/\n> ├── commit-graph -> commit-graphs/commit-graph-2492e0ef38643d4cb6369f76443e6a814d616258\n> ├── commit-graphs\n> │   ├── commit-graph-2492e0ef38643d4cb6369f76443e6a814d616258\n> │   ├── commit-graph-988881adc9fc3655077dc2d4d757d480b5ea0e11\n> │   └── info\n> └── packs\n> $ cat .git/objects/info/commit-graphs/info\n> 2492e0ef38643d4cb6369f76443e6a814d616258\n> 988881adc9fc3655077dc2d4d757d480b5ea0e11\n> \n> I.e. create new ones as needed, and when done say what sequence they\n> should be read in in the \"info\" file, and symlink \"commit-graph\" to\n> whatever the latest \"base\" is as a courtesy to old clients (or not, or\n> eventually don't bother).\n\nOK. Glad that the idea would work.\n\nThanks,\n-Stolee\n\n"},{"id":"376067","messageId":"a423afbfdd4d5591bdef83c4603b2d8de5e64e21.1558554800.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 01/11] commit-graph: document commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-22T19:53:22Z","receivedAt":"2019-05-22T19:53:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd a basic description of commit-graph chains. More details about the\nfeature will be added as we add functionality. This introduction gives a\nhigh-level overview to the goals of the feature and the basic layout of\ncommit-graph chains.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 59 ++++++++++++++++++++++++\n 1 file changed, 59 insertions(+)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex fb53341d5e..1dca3bd8fe 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -127,6 +127,65 @@ Design Details\n helpful for these clones, anyway. The commit-graph will not be read or\n written when shallow commits are present.\n \n+Commit Graphs Chains\n+--------------------\n+\n+Typically, repos grow with near-constant velocity (commits per day). Over time,\n+the number of commits added by a fetch operation is much smaller than the\n+number of commits in the full history. By creating a \"chain\" of commit-graphs,\n+we enable fast writes of new commit data without rewriting the entire commit\n+history -- at least, most of the time.\n+\n+## File Layout\n+\n+A commit-graph chain uses multiple files, and we use a fixed naming convention\n+to organize these files. Each commit-graph file has a name\n+`$OBJDIR/info/commit-graphs/graph-{hash}.graph` where `{hash}` is the hex-\n+valued hash stored in the footer of that file (which is a hash of the file's\n+contents before that hash). For a chain of commit-graph files, a plain-text\n+file at `$OBJDIR/info/commit-graphs/commit-graph-chain` contains the\n+hashes for the files in order from \"lowest\" to \"highest\".\n+\n+For example, if the `commit-graph-chain` file contains the lines\n+\n+```\n+\t{hash0}\n+\t{hash1}\n+\t{hash2}\n+```\n+\n+then the commit-graph chain looks like the following diagram:\n+\n+ +-----------------------+\n+ | graph-{hash2}.graph |\n+ +-----------------------+\n+\t |\n+ +-----------------------+\n+ | |\n+ | graph-{hash1}.graph |\n+ | |\n+ +-----------------------+\n+\t |\n+ +-----------------------+\n+ | |\n+ | |\n+ | |\n+ | graph-{hash0}.graph |\n+ | |\n+ | |\n+ | |\n+ +-----------------------+\n+\n+Let X0 be the number of commits in `graph-{hash0}.graph`, X1 be the number of\n+commits in `graph-{hash1}.graph`, and X2 be the number of commits in\n+`graph-{hash2}.graph`. If a commit appears in position i in `graph-{hash2}.graph`,\n+then we interpret this as being the commit in position (X0 + X1 + i), and that\n+will be used as its \"graph position\". The commits in `graph-{hash2}.graph` use these\n+positions to refer to their parents, which may be in `graph-{hash1}.graph` or\n+`graph-{hash0}.graph`. We can navigate to an arbitrary commit in position j by checking\n+its containment in the intervals [0, X0), [X0, X0 + X1), [X0 + X1, X0 + X1 +\n+X2).\n+\n Related Links\n -------------\n [0] https://bugs.chromium.org/p/git/issues/detail?id=8\n-- \ngitgitgadget\n\n"},{"id":"376068","messageId":"249668fc92133df684acc57963109eb75ac6d8fa.1558554800.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 02/11] commit-graph: prepare for commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-22T19:53:23Z","receivedAt":"2019-05-22T19:53:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTo prepare for a chain of commit-graph files, augment the\ncommit_graph struct to point to a base commit_graph. As we load\ncommits from the graph, we may actually want to read from a base\nfile according to the graph position.\n\nThe \"graph position\" of a commit is given by concatenating the\nlexicographic commit orders from each of the commit-graph files in\nthe chain. This means that we must distinguish two values:\n\n * lexicographic index : the position within the lexicographic\n order in a single commit-graph file.\n\n * graph position: the posiiton within the concatenated order\n of multiple commit-graph files\n\nGiven the lexicographic index of a commit in a graph, we can\ncompute the graph position by adding the number of commits in\nthe lower-level graphs. To find the lexicographic index of\na commit, we subtract the number of commits in lower-level graphs.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 74 ++++++++++++++++++++++++++++++++++++++++++++------\n commit-graph.h | 3 ++\n 2 files changed, 69 insertions(+), 8 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 7723156964..3afedcd7f5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -371,6 +371,25 @@ static int bsearch_graph(struct commit_graph *g, struct object_id *oid, uint32_t\n \t\t\t g->chunk_oid_lookup, g->hash_len, pos);\n }\n \n+static void load_oid_from_graph(struct commit_graph *g, int pos, struct object_id *oid)\n+{\n+\tuint32_t lex_index;\n+\n+\tif (!g)\n+\t\tBUG(\"NULL commit-graph\");\n+\n+\twhile (pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tif (pos >= g->num_commits + g->num_commits_in_base)\n+\t\tBUG(\"position %d is beyond the scope of this commit-graph (%d local + %d base commits)\",\n+\t\t pos, g->num_commits, g->num_commits_in_base);\n+\n+\tlex_index = pos - g->num_commits_in_base;\n+\n+\thashcpy(oid->hash, g->chunk_oid_lookup + g->hash_len * lex_index);\n+}\n+\n static struct commit_list **insert_parent_or_die(struct repository *r,\n \t\t\t\t\t\t struct commit_graph *g,\n \t\t\t\t\t\t uint64_t pos,\n@@ -379,10 +398,10 @@ static struct commit_list **insert_parent_or_die(struct repository *r,\n \tstruct commit *c;\n \tstruct object_id oid;\n \n-\tif (pos >= g->num_commits)\n+\tif (pos >= g->num_commits + g->num_commits_in_base)\n \t\tdie(\"invalid parent position %\"PRIu64, pos);\n \n-\thashcpy(oid.hash, g->chunk_oid_lookup + g->hash_len * pos);\n+\tload_oid_from_graph(g, pos, &oid);\n \tc = lookup_commit(r, &oid);\n \tif (!c)\n \t\tdie(_(\"could not find commit %s\"), oid_to_hex(&oid));\n@@ -392,7 +411,14 @@ static struct commit_list **insert_parent_or_die(struct repository *r,\n \n static void fill_commit_graph_info(struct commit *item, struct commit_graph *g, uint32_t pos)\n {\n-\tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n+\tconst unsigned char *commit_data;\n+\tuint32_t lex_index;\n+\n+\twhile (pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tlex_index = pos - g->num_commits_in_base;\n+\tcommit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * lex_index;\n \titem->graph_pos = pos;\n \titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n }\n@@ -405,10 +431,26 @@ static int fill_commit_in_graph(struct repository *r,\n \tuint32_t *parent_data_ptr;\n \tuint64_t date_low, date_high;\n \tstruct commit_list **pptr;\n-\tconst unsigned char *commit_data = g->chunk_commit_data + (g->hash_len + 16) * pos;\n+\tconst unsigned char *commit_data;\n+\tuint32_t lex_index;\n \n-\titem->object.parsed = 1;\n+\twhile (pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tif (pos >= g->num_commits + g->num_commits_in_base)\n+\t\tBUG(\"position %d is beyond the scope of this commit-graph (%d local + %d base commits)\",\n+\t\t pos, g->num_commits, g->num_commits_in_base);\n+\n+\t/*\n+\t * Store the \"full\" position, but then use the\n+\t * \"local\" position for the rest of the calculation.\n+\t */\n \titem->graph_pos = pos;\n+\tlex_index = pos - g->num_commits_in_base;\n+\n+\tcommit_data = g->chunk_commit_data + (g->hash_len + 16) * lex_index;\n+\n+\titem->object.parsed = 1;\n \n \titem->maybe_tree = NULL;\n \n@@ -452,7 +494,18 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \t\t*pos = item->graph_pos;\n \t\treturn 1;\n \t} else {\n-\t\treturn bsearch_graph(g, &(item->object.oid), pos);\n+\t\tstruct commit_graph *cur_g = g;\n+\t\tuint32_t lex_index;\n+\n+\t\twhile (cur_g && !bsearch_graph(cur_g, &(item->object.oid), &lex_index))\n+\t\t\tcur_g = cur_g->base_graph;\n+\n+\t\tif (cur_g) {\n+\t\t\t*pos = lex_index + cur_g->num_commits_in_base;\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\treturn 0;\n \t}\n }\n \n@@ -492,8 +545,13 @@ static struct tree *load_tree_for_commit(struct repository *r,\n \t\t\t\t\t struct commit *c)\n {\n \tstruct object_id oid;\n-\tconst unsigned char *commit_data = g->chunk_commit_data +\n-\t\t\t\t\t GRAPH_DATA_WIDTH * (c->graph_pos);\n+\tconst unsigned char *commit_data;\n+\n+\twhile (c->graph_pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tcommit_data = g->chunk_commit_data +\n+\t\t\tGRAPH_DATA_WIDTH * (c->graph_pos - g->num_commits_in_base);\n \n \thashcpy(oid.hash, commit_data);\n \tc->maybe_tree = lookup_tree(r, &oid);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 70f4caf0c7..f9fe32ebe3 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -48,6 +48,9 @@ struct commit_graph {\n \tuint32_t num_commits;\n \tstruct object_id oid;\n \n+\tuint32_t num_commits_in_base;\n+\tstruct commit_graph *base_graph;\n+\n \tconst uint32_t *chunk_oid_fanout;\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_commit_data;\n-- \ngitgitgadget\n\n"},{"id":"376069","messageId":"809fa7ad804834fa7250958ac328d3716e5fcf80.1558554800.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 03/11] commit-graph: rename commit_compare to oid_compare","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-22T19:53:24Z","receivedAt":"2019-05-22T19:53:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe helper function commit_compare() actually compares object_id\nstructs, not commits. A future change to commit-graph.c will need\nto sort commit structs, so rename this function in advance.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 3afedcd7f5..e2f438f6a3 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -761,7 +761,7 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n \t}\n }\n \n-static int commit_compare(const void *_a, const void *_b)\n+static int oid_compare(const void *_a, const void *_b)\n {\n \tconst struct object_id *a = (const struct object_id *)_a;\n \tconst struct object_id *b = (const struct object_id *)_b;\n@@ -1030,7 +1030,7 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n \t\t\t_(\"Counting distinct commits in commit graph\"),\n \t\t\tctx->oids.nr);\n \tdisplay_progress(ctx->progress, 0); /* TODO: Measure QSORT() progress */\n-\tQSORT(ctx->oids.list, ctx->oids.nr, commit_compare);\n+\tQSORT(ctx->oids.list, ctx->oids.nr, oid_compare);\n \n \tfor (i = 1; i < ctx->oids.nr; i++) {\n \t\tdisplay_progress(ctx->progress, i + 1);\n-- \ngitgitgadget\n\n"},{"id":"376070","messageId":"a8c0b47c8af44e2216e4a9462b384424ce94a2c9.1558554800.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 04/11] commit-graph: load commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-22T19:53:25Z","receivedAt":"2019-05-22T19:53:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nPrepare the logic for reading a chain of commit-graphs.\n\nFirst, look for a file at $OBJDIR/info/commit-graph. If it exists,\nthen use that file and stop.\n\nNext, look for the chain file at $OBJDIR/info/commit-graphs/commit-graph-chain.\nIf this file exists, then load the hash values as line-separated values in that\nfile and load $OBJDIR/info/commit-graphs/graph-{hash[i]}.graph for each hash[i]\nin that file. The file is given in order, so the first hash corresponds to the\n\"base\" file and the final hash corresponds to the \"tip\" file.\n\nThis implementation assumes that all of the graph-{hash}.graph files are in\nthe same object directory as the commit-graph-chain file. This will be updated\nin a future change. This change is purposefully simple so we can isolate the\ndifferent concerns.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 98 +++++++++++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 93 insertions(+), 5 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex e2f438f6a3..70e44393b8 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -45,6 +45,19 @@ char *get_commit_graph_filename(const char *obj_dir)\n \treturn xstrfmt(\"%s/info/commit-graph\", obj_dir);\n }\n \n+static char *get_split_graph_filename(const char *obj_dir,\n+\t\t\t\t const char *oid_hex)\n+{\n+\treturn xstrfmt(\"%s/info/commit-graphs/graph-%s.graph\",\n+\t\t obj_dir,\n+\t\t oid_hex);\n+}\n+\n+static char *get_chain_filename(const char *obj_dir)\n+{\n+\treturn xstrfmt(\"%s/info/commit-graphs/commit-graph-chain\", obj_dir);\n+}\n+\n static uint8_t oid_version(void)\n {\n \treturn 1;\n@@ -286,18 +299,93 @@ static struct commit_graph *load_commit_graph_one(const char *graph_file)\n \treturn load_commit_graph_one_fd_st(fd, &st);\n }\n \n+static int prepare_commit_graph_v1(struct repository *r, const char *obj_dir)\n+{\n+\tchar *graph_name = get_commit_graph_filename(obj_dir);\n+\tr->objects->commit_graph = load_commit_graph_one(graph_name);\n+\tfree(graph_name);\n+\n+\treturn r->objects->commit_graph ? 0 : -1;\n+}\n+\n+static int add_graph_to_chain(struct commit_graph *g,\n+\t\t\t struct commit_graph *chain,\n+\t\t\t struct object_id *oids,\n+\t\t\t int n)\n+{\n+\tstruct commit_graph *cur_g = chain;\n+\n+\twhile (n) {\n+\t\tn--;\n+\t\tcur_g = cur_g->base_graph;\n+\t}\n+\n+\tg->base_graph = chain;\n+\n+\tif (chain)\n+\t\tg->num_commits_in_base = chain->num_commits + chain->num_commits_in_base;\n+\n+\treturn 1;\n+}\n+\n+static void prepare_commit_graph_chain(struct repository *r, const char *obj_dir)\n+{\n+\tstruct strbuf line = STRBUF_INIT;\n+\tstruct stat st;\n+\tstruct object_id *oids;\n+\tint i = 0, valid = 1;\n+\tchar *chain_name = get_chain_filename(obj_dir);\n+\tFILE *fp;\n+\n+\tif (stat(chain_name, &st))\n+\t\treturn;\n+\n+\tif (st.st_size <= the_hash_algo->hexsz)\n+\t\treturn;\n+\n+\tfp = fopen(chain_name, \"r\");\n+\tfree(chain_name);\n+\n+\tif (!fp)\n+\t\treturn;\n+\n+\toids = xcalloc(st.st_size / (the_hash_algo->hexsz + 1), sizeof(struct object_id));\n+\n+\twhile (strbuf_getline_lf(&line, fp) != EOF && valid) {\n+\t\tchar *graph_name;\n+\t\tstruct commit_graph *g;\n+\n+\t\tif (get_oid_hex(line.buf, &oids[i])) {\n+\t\t\twarning(_(\"invalid commit-graph chain: line '%s' not a hash\"),\n+\t\t\t\tline.buf);\n+\t\t\tvalid = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tgraph_name = get_split_graph_filename(obj_dir, line.buf);\n+\t\tg = load_commit_graph_one(graph_name);\n+\t\tfree(graph_name);\n+\n+\t\tif (g && add_graph_to_chain(g, r->objects->commit_graph, oids, i))\n+\t\t\tr->objects->commit_graph = g;\n+\t\telse\n+\t\t\tvalid = 0;\n+\t}\n+\n+\tfree(oids);\n+\tfclose(fp);\n+}\n+\n static void prepare_commit_graph_one(struct repository *r, const char *obj_dir)\n {\n-\tchar *graph_name;\n \n \tif (r->objects->commit_graph)\n \t\treturn;\n \n-\tgraph_name = get_commit_graph_filename(obj_dir);\n-\tr->objects->commit_graph =\n-\t\tload_commit_graph_one(graph_name);\n+\tif (!prepare_commit_graph_v1(r, obj_dir))\n+\t\treturn;\n \n-\tFREE_AND_NULL(graph_name);\n+\tprepare_commit_graph_chain(r, obj_dir);\n }\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"376071","messageId":"4fefd0a654f6c296aad90b49df59af7a7877e3bf.1558554800.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 05/11] commit-graph: add base graphs chunk","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-22T19:53:25Z","receivedAt":"2019-05-22T19:53:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTo quickly verify a commit-graph chain is valid on load, we will\nread from the new \"Base Graphs Chunk\" of each file in the chain.\nThis will prevent accidentally loading incorrect data from manually\nediting the commit-graph-chain file or renaming graph-{hash}.graph\nfiles.\n\nThe commit_graph struct already had an object_id struct \"oid\", but\nit was never initialized or used. Add a line to read the hash from\nthe end of the commit-graph file and into the oid member.\n\nSigned-off-by: Derrick Stolee \n---\n .../technical/commit-graph-format.txt | 11 ++++++++--\n commit-graph.c | 22 +++++++++++++++++++\n commit-graph.h | 1 +\n 3 files changed, 32 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex 16452a0504..a4f17441ae 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -44,8 +44,9 @@ HEADER:\n \n 1-byte number (C) of \"chunks\"\n \n- 1-byte (reserved for later use)\n- Current clients should ignore this value.\n+ 1-byte number (B) of base commit-graphs\n+ We infer the length (H*B) of the Base Graphs chunk\n+ from this value.\n \n CHUNK LOOKUP:\n \n@@ -92,6 +93,12 @@ CHUNK DATA:\n positions for the parents until reaching a value with the most-significant\n bit on. The other bits correspond to the position of the last parent.\n \n+ Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]\n+ This list of H-byte hashes describe a set of B commit-graph files that\n+ form a commit-graph chain. The graph position for the ith commit in this\n+ file's OID Lookup chunk is equal to i plus the number of commits in all\n+ base graphs. If B is non-zero, this chunk must exist.\n+\n TRAILER:\n \n \tH-byte HASH-checksum of all of the above.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 70e44393b8..060897fff0 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -22,6 +22,7 @@\n #define GRAPH_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define GRAPH_CHUNKID_DATA 0x43444154 /* \"CDAT\" */\n #define GRAPH_CHUNKID_EXTRAEDGES 0x45444745 /* \"EDGE\" */\n+#define GRAPH_CHUNKID_BASE 0x42415345 /* \"BASE\" */\n \n #define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n \n@@ -262,6 +263,12 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t\t\telse\n \t\t\t\tgraph->chunk_extra_edges = data + chunk_offset;\n \t\t\tbreak;\n+\n+\t\tcase GRAPH_CHUNKID_BASE:\n+\t\t\tif (graph->chunk_base_graphs)\n+\t\t\t\tchunk_repeated = 1;\n+\t\t\telse\n+\t\t\t\tgraph->chunk_base_graphs = data + chunk_offset;\n \t\t}\n \n \t\tif (chunk_repeated) {\n@@ -280,6 +287,8 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t\tlast_chunk_offset = chunk_offset;\n \t}\n \n+\thashcpy(graph->oid.hash, graph->data + graph->data_len - graph->hash_len);\n+\n \tif (verify_commit_graph_lite(graph))\n \t\treturn NULL;\n \n@@ -315,8 +324,21 @@ static int add_graph_to_chain(struct commit_graph *g,\n {\n \tstruct commit_graph *cur_g = chain;\n \n+\tif (n && !g->chunk_base_graphs) {\n+\t\twarning(_(\"commit-graph has no base graphs chunk\"));\n+\t\treturn 0;\n+\t}\n+\n \twhile (n) {\n \t\tn--;\n+\n+\t\tif (!oideq(&oids[n], &cur_g->oid) ||\n+\t\t !hasheq(oids[n].hash, g->chunk_base_graphs + g->hash_len * n)) {\n+\t\t\twarning(_(\"commit-graph chain does not match\"));\n+\t\t\treturn 0;\n+\t\t}\n+\n+\n \t\tcur_g = cur_g->base_graph;\n \t}\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex f9fe32ebe3..80f4917ddb 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -55,6 +55,7 @@ struct commit_graph {\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_commit_data;\n \tconst unsigned char *chunk_extra_edges;\n+\tconst unsigned char *chunk_base_graphs;\n };\n \n struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st);\n-- \ngitgitgadget\n\n"},{"id":"376072","messageId":"a595a1eb65168fad4fae21c347ecac063a29c6f5.1558554800.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 06/11] commit-graph: rearrange chunk count logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-22T19:53:26Z","receivedAt":"2019-05-22T19:53:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe number of chunks in a commit-graph file can change depending on\nwhether we need the Extra Edges Chunk. We are going to add more optional\nchunks, and it will be helpful to rearrange this logic around the chunk\ncount before doing so.\n\nSpecifically, we need to finalize the number of chunks before writing\nthe commit-graph header. Further, we also need to fill out the chunk\nlookup table dynamically and using \"num_chunks\" as we add optional\nchunks is useful for adding optional chunks in the future.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 35 +++++++++++++++++++++--------------\n 1 file changed, 21 insertions(+), 14 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 060897fff0..b8a1444217 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1192,7 +1192,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tuint64_t chunk_offsets[5];\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n-\tint num_chunks = ctx->num_extra_edges ? 4 : 3;\n+\tint num_chunks = 3;\n \n \tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n \tif (safe_create_leading_directories(ctx->graph_name)) {\n@@ -1205,27 +1205,34 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \n-\thashwrite_be32(f, GRAPH_SIGNATURE);\n-\n-\thashwrite_u8(f, GRAPH_VERSION);\n-\thashwrite_u8(f, oid_version());\n-\thashwrite_u8(f, num_chunks);\n-\thashwrite_u8(f, 0); /* unused padding byte */\n-\n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n \tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n-\tif (ctx->num_extra_edges)\n-\t\tchunk_ids[3] = GRAPH_CHUNKID_EXTRAEDGES;\n-\telse\n-\t\tchunk_ids[3] = 0;\n-\tchunk_ids[4] = 0;\n+\tif (ctx->num_extra_edges) {\n+\t\tchunk_ids[num_chunks] = GRAPH_CHUNKID_EXTRAEDGES;\n+\t\tnum_chunks++;\n+\t}\n+\n+\tchunk_ids[num_chunks] = 0;\n \n \tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n \tchunk_offsets[2] = chunk_offsets[1] + hashsz * ctx->commits.nr;\n \tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * ctx->commits.nr;\n-\tchunk_offsets[4] = chunk_offsets[3] + 4 * ctx->num_extra_edges;\n+\n+\tnum_chunks = 3;\n+\tif (ctx->num_extra_edges) {\n+\t\tchunk_offsets[num_chunks + 1] = chunk_offsets[num_chunks] +\n+\t\t\t\t\t\t4 * ctx->num_extra_edges;\n+\t\tnum_chunks++;\n+\t}\n+\n+\thashwrite_be32(f, GRAPH_SIGNATURE);\n+\n+\thashwrite_u8(f, GRAPH_VERSION);\n+\thashwrite_u8(f, oid_version());\n+\thashwrite_u8(f, num_chunks);\n+\thashwrite_u8(f, 0);\n \n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tuint32_t chunk_write[3];\n-- \ngitgitgadget\n\n"},{"id":"376073","messageId":"5ad14f574b6ffa5e233d61098dbcfc29c3632be5.1558554800.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 08/11] commit-graph: add --split option to builtin","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-22T19:53:27Z","receivedAt":"2019-05-22T19:53:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd a new \"--split\" option to the 'git commit-graph write' subcommand. This\noption allows the optional behavior of writing a commit-graph chain.\n\nThe current behavior will add a tip commit-graph containing any commits that\nare not in the existing commit-graph or commit-graph chain. Later changes\nwill allow merging the chain and expiring out-dated files.\n\nAdd a new test script (t5323-split-commit-graph.sh) that demonstrates this\nbehavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 10 ++-\n t/t5323-split-commit-graph.sh | 122 ++++++++++++++++++++++++++++++++++\n 2 files changed, 129 insertions(+), 3 deletions(-)\n create mode 100755 t/t5323-split-commit-graph.sh\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 828b1a713f..c2c07d3917 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -10,7 +10,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph verify [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -25,7 +25,7 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -35,9 +35,9 @@ static struct opts_commit_graph {\n \tint stdin_packs;\n \tint stdin_commits;\n \tint append;\n+\tint split;\n } opts;\n \n-\n static int graph_verify(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n@@ -156,6 +156,8 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"start walk at commits listed by stdin\")),\n \t\tOPT_BOOL(0, \"append\", &opts.append,\n \t\t\tN_(\"include all commits already in the commit-graph file\")),\n+\t\tOPT_BOOL(0, \"split\", &opts.split,\n+\t\t\tN_(\"allow writing an incremental commit-graph file\")),\n \t\tOPT_END(),\n \t};\n \n@@ -169,6 +171,8 @@ static int graph_write(int argc, const char **argv)\n \t\topts.obj_dir = get_object_directory();\n \tif (opts.append)\n \t\tflags |= COMMIT_GRAPH_APPEND;\n+\tif (opts.split)\n+\t\tflags |= COMMIT_GRAPH_SPLIT;\n \n \tread_replace_refs = 0;\n \ndiff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\nnew file mode 100755\nindex 0000000000..96704b9f5b\n--- /dev/null\n+++ b/t/t5323-split-commit-graph.sh\n@@ -0,0 +1,122 @@\n+#!/bin/sh\n+\n+test_description='split commit graph'\n+. ./test-lib.sh\n+\n+GIT_TEST_COMMIT_GRAPH=0\n+\n+test_expect_success 'setup repo' '\n+\tgit init &&\n+\tgit config core.commitGraph true &&\n+\tinfodir=\".git/objects/info\" &&\n+\tgraphdir=\"$infodir/commit-graphs\" &&\n+\ttest_oid_init\n+'\n+\n+graph_read_expect() {\n+\tNUM_BASE=0\n+\tif test ! -z $2\n+\tthen\n+\t\tNUM_BASE=$2\n+\tfi\n+\tcat >expect <<- EOF\n+\theader: 43475048 1 1 3 $NUM_BASE\n+\tnum_commits: $1\n+\tchunks: oid_fanout oid_lookup commit_metadata\n+\tEOF\n+\tgit commit-graph read >output &&\n+\ttest_cmp expect output\n+}\n+\n+test_expect_success 'create commits and write commit-graph' '\n+\tfor i in $(test_seq 3)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i\n+\tdone &&\n+\tgit commit-graph write --reachable &&\n+\ttest_path_is_file $infodir/commit-graph &&\n+\tgraph_read_expect 3\n+'\n+\n+graph_git_two_modes() {\n+\tgit -c core.commitGraph=true $1 >output\n+\tgit -c core.commitGraph=false $1 >expect\n+\ttest_cmp expect output\n+}\n+\n+graph_git_behavior() {\n+\tMSG=$1\n+\tBRANCH=$2\n+\tCOMPARE=$3\n+\ttest_expect_success \"check normal git operations: $MSG\" '\n+\t\tgraph_git_two_modes \"log --oneline $BRANCH\" &&\n+\t\tgraph_git_two_modes \"log --topo-order $BRANCH\" &&\n+\t\tgraph_git_two_modes \"log --graph $COMPARE..$BRANCH\" &&\n+\t\tgraph_git_two_modes \"branch -vv\" &&\n+\t\tgraph_git_two_modes \"merge-base -a $BRANCH $COMPARE\"\n+\t'\n+}\n+\n+graph_git_behavior 'graph exists' commits/3 commits/1\n+\n+verify_chain_files_exist() {\n+\tfor hash in $(cat $1/commit-graph-chain)\n+\tdo\n+\t\ttest_path_is_file $1/graph-$hash.graph\n+\tdone\n+}\n+\n+test_expect_success 'add more commits, and write a new base graph' '\n+\tgit reset --hard commits/1 &&\n+\tfor i in $(test_seq 4 5)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i\n+\tdone &&\n+\tgit reset --hard commits/2 &&\n+\tfor i in $(test_seq 6 10)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i\n+\tdone &&\n+\tgit reset --hard commits/2 &&\n+\tgit merge commits/4 &&\n+\tgit branch merge/1 &&\n+\tgit reset --hard commits/4 &&\n+\tgit merge commits/6 &&\n+\tgit branch merge/2 &&\n+\tgit commit-graph write --reachable &&\n+\tgraph_read_expect 12\n+'\n+\n+test_expect_success 'add three more commits, write a tip graph' '\n+\tgit reset --hard commits/3 &&\n+\tgit merge merge/1 &&\n+\tgit merge commits/5 &&\n+\tgit merge merge/2 &&\n+\tgit branch merge/3 &&\n+\tgit commit-graph write --reachable --split &&\n+\ttest_path_is_missing $infodir/commit-graph &&\n+\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\tls $graphdir/graph-*.graph >graph-files &&\n+\ttest_line_count = 2 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n+graph_git_behavior 'split commit-graph: merge 3 vs 2' merge/3 merge/2\n+\n+test_expect_success 'add one commit, write a tip graph' '\n+\ttest_commit 11 &&\n+\tgit branch commits/11 &&\n+\tgit commit-graph write --reachable --split &&\n+\ttest_path_is_missing $infodir/commit-graph &&\n+\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\tls $graphdir/graph-*.graph >graph-files &&\n+\ttest_line_count = 3 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n+graph_git_behavior 'three-layer commit-graph: commit 11 vs 6' commits/11 commits/6\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"376074","messageId":"9cbfb656b3e12d11b4709e72246af478072c9496.1558554800.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 07/11] commit-graph: write commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-22T19:53:27Z","receivedAt":"2019-05-22T19:53:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nExtend write_commit_graph() to write a commit-graph chain when given the\nCOMMIT_GRAPH_SPLIT flag.\n\nThis implementation is purposefully simplistic in how it creates a new\nchain. The commits not already in the chain are added to a new tip\ncommit-graph file.\n\nMuch of the logic around writing a graph-{hash}.graph file and updating\nthe commit-graph-chain file is the same as the commit-graph file case.\nHowever, there are several places where we need to do some extra logic\nin the split case.\n\nTrack the list of graph filenames before and after the planned write.\nThis will be more important when we start merging graph files, but it\nalso allows us to upgrade our commit-graph file to the appropriate\ngraph-{hash}.graph file when we upgrade to a chain of commit-graphs.\n\nNote that we use the eighth byte of the commit-graph header to store the\nnumber of base graph files. This determines the length of the base\ngraphs chunk.\n\nA subtle change of behavior with the new logic is that we do not write a\ncommit-graph if we our commit list is empty. This extends to the typical\ncase, which is reflected in t5318-commit-graph.sh.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 286 ++++++++++++++++++++++++++++++++++++++--\n commit-graph.h | 2 +\n t/t5318-commit-graph.sh | 2 +-\n 3 files changed, 278 insertions(+), 12 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex b8a1444217..fb972c0bc2 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -300,12 +300,18 @@ static struct commit_graph *load_commit_graph_one(const char *graph_file)\n \n \tstruct stat st;\n \tint fd;\n+\tstruct commit_graph *g;\n \tint open_ok = open_commit_graph(graph_file, &fd, &st);\n \n \tif (!open_ok)\n \t\treturn NULL;\n \n-\treturn load_commit_graph_one_fd_st(fd, &st);\n+\tg = load_commit_graph_one_fd_st(fd, &st);\n+\n+\tif (g)\n+\t\tg->filename = xstrdup(graph_file);\n+\n+\treturn g;\n }\n \n static int prepare_commit_graph_v1(struct repository *r, const char *obj_dir)\n@@ -709,8 +715,19 @@ struct write_commit_graph_context {\n \tstruct progress *progress;\n \tint progress_done;\n \tuint64_t progress_cnt;\n+\n+\tchar *base_graph_name;\n+\tint num_commit_graphs_before;\n+\tint num_commit_graphs_after;\n+\tchar **commit_graph_filenames_before;\n+\tchar **commit_graph_filenames_after;\n+\tchar **commit_graph_hash_after;\n+\tuint32_t new_num_commits_in_base;\n+\tstruct commit_graph *new_base_graph;\n+\n \tunsigned append:1,\n-\t\t report_progress:1;\n+\t\t report_progress:1,\n+\t\t split:1;\n };\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n@@ -780,6 +797,16 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t commit_to_sha1);\n \n+\t\t\tif (edge_value >= 0)\n+\t\t\t\tedge_value += ctx->new_num_commits_in_base;\n+\t\t\telse {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (find_commit_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n+\t\t\t\t\tedge_value = pos;\n+\t\t\t}\n+\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n \t\t\t\t oid_to_hex(&parent->item->object.oid),\n@@ -800,6 +827,17 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\t\t\t ctx->commits.list,\n \t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t commit_to_sha1);\n+\n+\t\t\tif (edge_value >= 0)\n+\t\t\t\tedge_value += ctx->new_num_commits_in_base;\n+\t\t\telse {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (find_commit_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n+\t\t\t\t\tedge_value = pos;\n+\t\t\t}\n+\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n \t\t\t\t oid_to_hex(&parent->item->object.oid),\n@@ -857,6 +895,16 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n \t\t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t\t commit_to_sha1);\n \n+\t\t\tif (edge_value >= 0)\n+\t\t\t\tedge_value += ctx->new_num_commits_in_base;\n+\t\t\telse {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (find_commit_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n+\t\t\t\t\tedge_value = pos;\n+\t\t\t}\n+\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n \t\t\t\t oid_to_hex(&parent->item->object.oid),\n@@ -948,7 +996,13 @@ static void close_reachable(struct write_commit_graph_context *ctx)\n \t\tdisplay_progress(ctx->progress, i + 1);\n \t\tcommit = lookup_commit(ctx->r, &ctx->oids.list[i]);\n \n-\t\tif (commit && !parse_commit_no_graph(commit))\n+\t\tif (!commit)\n+\t\t\tcontinue;\n+\t\tif (ctx->split) {\n+\t\t\tif (!parse_commit(commit) &&\n+\t\t\t commit->graph_pos == COMMIT_NOT_FROM_GRAPH)\n+\t\t\t\tadd_missing_parents(ctx, commit);\n+\t\t} else if (!parse_commit_no_graph(commit))\n \t\t\tadd_missing_parents(ctx, commit);\n \t}\n \tstop_progress(&ctx->progress);\n@@ -1144,8 +1198,16 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n \n \tfor (i = 1; i < ctx->oids.nr; i++) {\n \t\tdisplay_progress(ctx->progress, i + 1);\n-\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n+\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i])) {\n+\t\t\tif (ctx->split) {\n+\t\t\t\tstruct commit *c = lookup_commit(ctx->r, &ctx->oids.list[i]);\n+\n+\t\t\t\tif (!c || c->graph_pos != COMMIT_NOT_FROM_GRAPH)\n+\t\t\t\t\tcontinue;\n+\t\t\t}\n+\n \t\t\tcount_distinct++;\n+\t\t}\n \t}\n \tstop_progress(&ctx->progress);\n \n@@ -1168,7 +1230,13 @@ static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n \t\tif (i > 0 && oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n \t\t\tcontinue;\n \n+\t\tALLOC_GROW(ctx->commits.list, ctx->commits.nr + 1, ctx->commits.alloc);\n \t\tctx->commits.list[ctx->commits.nr] = lookup_commit(ctx->r, &ctx->oids.list[i]);\n+\n+\t\tif (ctx->split &&\n+\t\t ctx->commits.list[ctx->commits.nr]->graph_pos != COMMIT_NOT_FROM_GRAPH)\n+\t\t\tcontinue;\n+\n \t\tparse_commit_no_graph(ctx->commits.list[ctx->commits.nr]);\n \n \t\tfor (parent = ctx->commits.list[ctx->commits.nr]->parents;\n@@ -1183,18 +1251,86 @@ static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n+static int write_graph_chunk_base_1(struct hashfile *f,\n+\t\t\t\t struct commit_graph *g)\n+{\n+\tint num = 0;\n+\n+\tif (!g)\n+\t\treturn 0;\n+\n+\tnum = write_graph_chunk_base_1(f, g->base_graph);\n+\thashwrite(f, g->oid.hash, the_hash_algo->rawsz);\n+\treturn num + 1;\n+}\n+\n+static int write_graph_chunk_base(struct hashfile *f,\n+\t\t\t\t struct write_commit_graph_context *ctx)\n+{\n+\tint num = write_graph_chunk_base_1(f, ctx->new_base_graph);\n+\n+\tif (num != ctx->num_commit_graphs_after - 1) {\n+\t\terror(_(\"failed to write correct number of base graph ids\"));\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void init_commit_graph_chain(struct write_commit_graph_context *ctx)\n+{\n+\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\tuint32_t i;\n+\n+\tctx->new_base_graph = g;\n+\tctx->base_graph_name = xstrdup(g->filename);\n+\tctx->new_num_commits_in_base = g->num_commits + g->num_commits_in_base;\n+\n+\tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n+\n+\tALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);\n+\tALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);\n+\n+\tfor (i = 0; i < ctx->num_commit_graphs_before - 1; i++)\n+\t\tctx->commit_graph_filenames_after[i] = xstrdup(ctx->commit_graph_filenames_before[i]);\n+\n+\tif (ctx->num_commit_graphs_before)\n+\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_before - 1] =\n+\t\t\tget_split_graph_filename(ctx->obj_dir, oid_to_hex(&g->oid));\n+\n+\ti = ctx->num_commit_graphs_before - 1;\n+\n+\twhile (g) {\n+\t\tctx->commit_graph_hash_after[i] = xstrdup(oid_to_hex(&g->oid));\n+\t\ti--;\n+\t\tg = g->base_graph;\n+\t}\n+}\n+\n static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n+\tint fd;\n \tstruct hashfile *f;\n \tstruct lock_file lk = LOCK_INIT;\n-\tuint32_t chunk_ids[5];\n-\tuint64_t chunk_offsets[5];\n+\tuint32_t chunk_ids[6];\n+\tuint64_t chunk_offsets[6];\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tint num_chunks = 3;\n+\tstruct object_id file_hash;\n+\n+\tif (ctx->split) {\n+\t\tstruct strbuf tmp_file = STRBUF_INIT;\n+\n+\t\tstrbuf_addf(&tmp_file,\n+\t\t\t \"%s/info/commit-graphs/tmp_graph_XXXXXX\",\n+\t\t\t ctx->obj_dir);\n+\t\tctx->graph_name = strbuf_detach(&tmp_file, NULL);\n+\t} else {\n+\t\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n+\t}\n \n-\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n \tif (safe_create_leading_directories(ctx->graph_name)) {\n \t\tUNLEAK(ctx->graph_name);\n \t\terror(_(\"unable to create leading directories of %s\"),\n@@ -1202,8 +1338,23 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\treturn errno;\n \t}\n \n-\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n-\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n+\tif (ctx->split) {\n+\t\tchar *lock_name = get_chain_filename(ctx->obj_dir);\n+\n+\t\thold_lock_file_for_update(&lk, lock_name, LOCK_DIE_ON_ERROR);\n+\n+\t\tfd = git_mkstemp_mode(ctx->graph_name, 0444);\n+\t\tif (fd < 0) {\n+\t\t\terror(_(\"unable to create '%s'\"), ctx->graph_name);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tf = hashfd(fd, ctx->graph_name);\n+\t} else {\n+\t\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n+\t\tfd = lk.tempfile->fd;\n+\t\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n+\t}\n \n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n@@ -1212,6 +1363,10 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\tchunk_ids[num_chunks] = GRAPH_CHUNKID_EXTRAEDGES;\n \t\tnum_chunks++;\n \t}\n+\tif (ctx->num_commit_graphs_after > 1) {\n+\t\tchunk_ids[num_chunks] = GRAPH_CHUNKID_BASE;\n+\t\tnum_chunks++;\n+\t}\n \n \tchunk_ids[num_chunks] = 0;\n \n@@ -1226,13 +1381,18 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\t\t\t\t4 * ctx->num_extra_edges;\n \t\tnum_chunks++;\n \t}\n+\tif (ctx->num_commit_graphs_after > 1) {\n+\t\tchunk_offsets[num_chunks + 1] = chunk_offsets[num_chunks] +\n+\t\t\t\t\t\thashsz * (ctx->num_commit_graphs_after - 1);\n+\t\tnum_chunks++;\n+\t}\n \n \thashwrite_be32(f, GRAPH_SIGNATURE);\n \n \thashwrite_u8(f, GRAPH_VERSION);\n \thashwrite_u8(f, oid_version());\n \thashwrite_u8(f, num_chunks);\n-\thashwrite_u8(f, 0);\n+\thashwrite_u8(f, ctx->num_commit_graphs_after - 1);\n \n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tuint32_t chunk_write[3];\n@@ -1258,11 +1418,67 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \twrite_graph_chunk_data(f, hashsz, ctx);\n \tif (ctx->num_extra_edges)\n \t\twrite_graph_chunk_extra_edges(f, ctx);\n+\tif (ctx->num_commit_graphs_after > 1 &&\n+\t write_graph_chunk_base(f, ctx)) {\n+\t\treturn -1;\n+\t}\n \tstop_progress(&ctx->progress);\n \tstrbuf_release(&progress_title);\n \n+\tif (ctx->split && ctx->base_graph_name && ctx->num_commit_graphs_after > 1) {\n+\t\tchar *new_base_hash = xstrdup(oid_to_hex(&ctx->new_base_graph->oid));\n+\t\tchar *new_base_name = get_split_graph_filename(ctx->obj_dir, new_base_hash);\n+\n+\t\tfree(ctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2]);\n+\t\tfree(ctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 2]);\n+\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2] = new_base_name;\n+\t\tctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 2] = new_base_hash;\n+\t}\n+\n \tclose_commit_graph(ctx->r);\n-\tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n+\tfinalize_hashfile(f, file_hash.hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n+\n+\tif (ctx->split) {\n+\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n+\t\tchar *final_graph_name;\n+\t\tint result;\n+\n+\t\tclose(fd);\n+\n+\t\tif (!chainf) {\n+\t\t\terror(_(\"unable to open commit-graph chain file\"));\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (ctx->base_graph_name) {\n+\t\t\tresult = rename(ctx->base_graph_name,\n+\t\t\t\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2]);\n+\n+\t\t\tif (result) {\n+\t\t\t\terror(_(\"failed to rename base commit-graph file\"));\n+\t\t\t\treturn -1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tchar *graph_name = get_commit_graph_filename(ctx->obj_dir);\n+\t\t\tunlink(graph_name);\n+\t\t}\n+\n+\t\tctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 1] = xstrdup(oid_to_hex(&file_hash));\n+\t\tfinal_graph_name = get_split_graph_filename(ctx->obj_dir,\n+\t\t\t\t\tctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 1]);\n+\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 1] = final_graph_name;\n+\n+\t\tresult = rename(ctx->graph_name, final_graph_name);\n+\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++)\n+\t\t\tfprintf(lk.tempfile->fp, \"%s\\n\", ctx->commit_graph_hash_after[i]);\n+\n+\t\tif (result) {\n+\t\t\terror(_(\"failed to rename temporary commit-graph file\"));\n+\t\t\treturn -1;\n+\t\t}\n+\t}\n+\n \tcommit_lock_file(&lk);\n \n \treturn 0;\n@@ -1285,6 +1501,30 @@ int write_commit_graph(const char *obj_dir,\n \tctx->obj_dir = obj_dir;\n \tctx->append = flags & COMMIT_GRAPH_APPEND ? 1 : 0;\n \tctx->report_progress = flags & COMMIT_GRAPH_PROGRESS ? 1 : 0;\n+\tctx->split = flags & COMMIT_GRAPH_SPLIT ? 1 : 0;\n+\n+\tif (ctx->split) {\n+\t\tstruct commit_graph *g;\n+\t\tprepare_commit_graph(ctx->r);\n+\n+\t\tg = ctx->r->objects->commit_graph;\n+\n+\t\twhile (g) {\n+\t\t\tctx->num_commit_graphs_before++;\n+\t\t\tg = g->base_graph;\n+\t\t}\n+\n+\t\tif (ctx->num_commit_graphs_before) {\n+\t\t\tALLOC_ARRAY(ctx->commit_graph_filenames_before, ctx->num_commit_graphs_before);\n+\t\t\ti = ctx->num_commit_graphs_before;\n+\t\t\tg = ctx->r->objects->commit_graph;\n+\n+\t\t\twhile (g) {\n+\t\t\t\tctx->commit_graph_filenames_before[--i] = xstrdup(g->filename);\n+\t\t\t\tg = g->base_graph;\n+\t\t\t}\n+\t\t}\n+\t}\n \n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n@@ -1339,6 +1579,14 @@ int write_commit_graph(const char *obj_dir,\n \t\tgoto cleanup;\n \t}\n \n+\tif (!ctx->commits.nr)\n+\t\tgoto cleanup;\n+\n+\tif (ctx->split)\n+\t\tinit_commit_graph_chain(ctx);\n+\telse\n+\t\tctx->num_commit_graphs_after = 1;\n+\n \tcompute_generation_numbers(ctx);\n \n \tres = write_commit_graph_file(ctx);\n@@ -1347,6 +1595,21 @@ int write_commit_graph(const char *obj_dir,\n \tfree(ctx->graph_name);\n \tfree(ctx->commits.list);\n \tfree(ctx->oids.list);\n+\n+\tif (ctx->commit_graph_filenames_after) {\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++) {\n+\t\t\tfree(ctx->commit_graph_filenames_after[i]);\n+\t\t\tfree(ctx->commit_graph_hash_after[i]);\n+\t\t}\n+\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_before; i++)\n+\t\t\tfree(ctx->commit_graph_filenames_before[i]);\n+\n+\t\tfree(ctx->commit_graph_filenames_after);\n+\t\tfree(ctx->commit_graph_filenames_before);\n+\t\tfree(ctx->commit_graph_hash_after);\n+\t}\n+\n \tfree(ctx);\n \n \treturn res;\n@@ -1534,5 +1797,6 @@ void free_commit_graph(struct commit_graph *g)\n \t\tg->data = NULL;\n \t\tclose(g->graph_fd);\n \t}\n+\tfree(g->filename);\n \tfree(g);\n }\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 80f4917ddb..5c48c4f66a 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -47,6 +47,7 @@ struct commit_graph {\n \tunsigned char num_chunks;\n \tuint32_t num_commits;\n \tstruct object_id oid;\n+\tchar *filename;\n \n \tuint32_t num_commits_in_base;\n \tstruct commit_graph *base_graph;\n@@ -71,6 +72,7 @@ int generation_numbers_enabled(struct repository *r);\n \n #define COMMIT_GRAPH_APPEND (1 << 0)\n #define COMMIT_GRAPH_PROGRESS (1 << 1)\n+#define COMMIT_GRAPH_SPLIT (1 << 2)\n \n int write_commit_graph_reachable(const char *obj_dir, unsigned int flags);\n int write_commit_graph(const char *obj_dir,\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 3b6fd0d728..063f906b3e 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -20,7 +20,7 @@ test_expect_success 'verify graph with no graph file' '\n test_expect_success 'write graph with no packs' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write --object-dir . &&\n-\ttest_path_is_file info/commit-graph\n+\ttest_path_is_missing info/commit-graph\n '\n \n test_expect_success 'close with correct error on bad input' '\n-- \ngitgitgadget\n\n"},{"id":"376075","messageId":"4cfe19a933fae40465e424cb364dad812e8283f5.1558554800.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 10/11] commit-graph: allow cross-alternate chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-22T19:53:29Z","receivedAt":"2019-05-22T19:53:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an environment like a fork network, it is helpful to have a\ncommit-graph chain that spans both the base repo and the fork repo. The\nfork is usually a small set of data on top of the large repo, but\nsometimes the fork is much larger. For example, git-for-windows/git has\nalmost double the number of commits as git/git because it rebases its\ncommits on every major version update.\n\nTo allow cross-alternate commit-graph chains, we need a few pieces:\n\n1. When looking for a graph-{hash}.graph file, check all alternates.\n\n2. When merging commit-graph chains, do not merge across alternates.\n\n3. When writing a new commit-graph chain based on a commit-graph file\n in another object directory, do not allow success if the base file\n has of the name \"commit-graph\" instead of\n \"commit-graphs/graoh-{hash}.graph\".\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 40 +++++++++++++++++++++\n commit-graph.c | 46 ++++++++++++++++++------\n commit-graph.h | 1 +\n t/t5323-split-commit-graph.sh | 37 +++++++++++++++++++\n 4 files changed, 114 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex 083fff9927..b6fe8b2321 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -267,6 +267,42 @@ The merge strategy values (2 for the size multiple, 64,000 for the maximum\n number of commits) could be extracted into config settings for full\n flexibility.\n \n+## Chains across multiple object directories\n+\n+In a repo with alternates, we look for the `commit-graph-chain` file starting\n+in the local object directory and then in each alternate. The first file that\n+exists defines our chain. As we look for the `graph-{hash}` files for\n+each `{hash}` in the chain file, we follow the same pattern for the host\n+directories.\n+\n+This allows commit-graphs to be split across multiple forks in a fork network.\n+The typical case is a large \"base\" repo with many smaller forks.\n+\n+As the base repo advances, it will likely update and merge its commit-graph\n+chain more frequently than the forks. If a fork updates their commit-graph after\n+the base repo, then it should \"reparent\" the commit-graph chain onto the new\n+chain in the base repo. When reading each `graph-{hash}` file, we track\n+the object directory containing it. During a write of a new commit-graph file,\n+we check for any changes in the source object directory and read the\n+`commit-graph-chain` file for that source and create a new file based on those\n+files. During this \"reparent\" operation, we necessarily need to collapse all\n+levels in the fork, as all of the files are invalid against the new base file.\n+\n+It is crucial to be careful when cleaning up \"unreferenced\" `graph-{hash}.graph`\n+files in this scenario. It falls to the user to define the proper settings for\n+their custom environment:\n+\n+ 1. When merging levels in the base repo, the unreferenced files may still be\n+ referenced by chains from fork repos.\n+\n+ 2. The expiry time should be set to a length of time such that every fork has\n+ time to recompute their commit-graph chain to \"reparent\" onto the new base\n+ file(s).\n+\n+ 3. If the commit-graph chain is updated in the base, the fork will not have\n+ access to the new chain until its chain is updated to reference those files.\n+ (This may change in the future [5].)\n+\n Related Links\n -------------\n [0] https://bugs.chromium.org/p/git/issues/detail?id=8\n@@ -293,3 +329,7 @@ Related Links\n \n [4] https://public-inbox.org/git/20180108154822.54829-1-git@jeffhostetler.com/T/#u\n A patch to remove the ahead-behind calculation from 'status'.\n+\n+[5] https://public-inbox.org/git/f27db281-abad-5043-6d71-cbb083b1c877@gmail.com/\n+ A discussion of a \"two-dimensional graph position\" that can allow reading\n+ multiple commit-graph chains at the same time.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex e9784cd559..0d8b942e2b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -320,6 +320,9 @@ static int prepare_commit_graph_v1(struct repository *r, const char *obj_dir)\n \tr->objects->commit_graph = load_commit_graph_one(graph_name);\n \tfree(graph_name);\n \n+\tif (r->objects->commit_graph)\n+\t\tr->objects->commit_graph->obj_dir = obj_dir;\n+\n \treturn r->objects->commit_graph ? 0 : -1;\n }\n \n@@ -380,8 +383,7 @@ static void prepare_commit_graph_chain(struct repository *r, const char *obj_dir\n \toids = xcalloc(st.st_size / (the_hash_algo->hexsz + 1), sizeof(struct object_id));\n \n \twhile (strbuf_getline_lf(&line, fp) != EOF && valid) {\n-\t\tchar *graph_name;\n-\t\tstruct commit_graph *g;\n+\t\tstruct object_directory *odb;\n \n \t\tif (get_oid_hex(line.buf, &oids[i])) {\n \t\t\twarning(_(\"invalid commit-graph chain: line '%s' not a hash\"),\n@@ -390,14 +392,23 @@ static void prepare_commit_graph_chain(struct repository *r, const char *obj_dir\n \t\t\tbreak;\n \t\t}\n \n-\t\tgraph_name = get_split_graph_filename(obj_dir, line.buf);\n-\t\tg = load_commit_graph_one(graph_name);\n-\t\tfree(graph_name);\n+\t\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n+\t\t\tchar *graph_name = get_split_graph_filename(odb->path, line.buf);\n+\t\t\tstruct commit_graph *g = load_commit_graph_one(graph_name);\n \n-\t\tif (g && add_graph_to_chain(g, r->objects->commit_graph, oids, i))\n-\t\t\tr->objects->commit_graph = g;\n-\t\telse\n-\t\t\tvalid = 0;\n+\t\t\tfree(graph_name);\n+\n+\t\t\tif (g) {\n+\t\t\t\tg->obj_dir = odb->path;\n+\n+\t\t\t\tif (add_graph_to_chain(g, r->objects->commit_graph, oids, i))\n+\t\t\t\t\tr->objects->commit_graph = g;\n+\t\t\t\telse\n+\t\t\t\t\tvalid = 0;\n+\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n \t}\n \n \tfree(oids);\n@@ -1397,7 +1408,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \n \tif (ctx->split && ctx->base_graph_name && ctx->num_commit_graphs_after > 1) {\n \t\tchar *new_base_hash = xstrdup(oid_to_hex(&ctx->new_base_graph->oid));\n-\t\tchar *new_base_name = get_split_graph_filename(ctx->obj_dir, new_base_hash);\n+\t\tchar *new_base_name = get_split_graph_filename(ctx->new_base_graph->obj_dir, new_base_hash);\n \n \t\tfree(ctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2]);\n \t\tfree(ctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 2]);\n@@ -1468,6 +1479,9 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \n \twhile (g && (g->num_commits <= split_strategy_size_mult * num_commits ||\n \t\t num_commits > split_strategy_max_commits)) {\n+\t\tif (strcmp(g->obj_dir, ctx->obj_dir))\n+\t\t\tbreak;\n+\n \t\tnum_commits += g->num_commits;\n \t\tg = g->base_graph;\n \n@@ -1476,6 +1490,18 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \n \tctx->new_base_graph = g;\n \n+\tif (ctx->num_commit_graphs_after == 2) {\n+\t\tchar *old_graph_name = get_commit_graph_filename(g->obj_dir);\n+\n+\t\tif (!strcmp(g->filename, old_graph_name) &&\n+\t\t strcmp(g->obj_dir, ctx->obj_dir)) {\n+\t\t\tctx->num_commit_graphs_after = 1;\n+\t\t\tctx->new_base_graph = NULL;\n+\t\t}\n+\n+\t\tfree(old_graph_name);\n+\t}\n+\n \tALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);\n \tALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex 5c48c4f66a..10466bc064 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -48,6 +48,7 @@ struct commit_graph {\n \tuint32_t num_commits;\n \tstruct object_id oid;\n \tchar *filename;\n+\tconst char *obj_dir;\n \n \tuint32_t num_commits_in_base;\n \tstruct commit_graph *base_graph;\ndiff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\nindex 3902fd9aee..ed5cac8617 100755\n--- a/t/t5323-split-commit-graph.sh\n+++ b/t/t5323-split-commit-graph.sh\n@@ -90,6 +90,21 @@ test_expect_success 'add more commits, and write a new base graph' '\n \tgraph_read_expect 12\n '\n \n+test_expect_success 'fork and fail to base a chain on a commit-graph file' '\n+\ttest_when_finished rm -rf fork &&\n+\tgit clone . fork &&\n+\t(\n+\t\tcd fork &&\n+\t\trm .git/objects/info/commit-graph &&\n+\t\techo \"$TRASH_DIRECTORY/.git/objects\" >.git/objects/info/alternates &&\n+\t\ttest_commit new-commit &&\n+\t\tgit commit-graph write --reachable --split &&\n+\t\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n+\t\tverify_chain_files_exist $graphdir\n+\t)\n+'\n+\n test_expect_success 'add three more commits, write a tip graph' '\n \tgit reset --hard commits/3 &&\n \tgit merge merge/1 &&\n@@ -132,4 +147,26 @@ test_expect_success 'add one commit, write a merged graph' '\n \n graph_git_behavior 'merged commit-graph: commit 12 vs 6' commits/12 commits/6\n \n+test_expect_success 'create fork and chain across alternate' '\n+\tgit clone . fork &&\n+\t(\n+\t\tcd fork &&\n+\t\tgit config core.commitGraph true &&\n+\t\trm -rf $graphdir &&\n+\t\techo \"$TRASH_DIRECTORY/.git/objects\" >.git/objects/info/alternates &&\n+\t\ttest_commit 13 &&\n+\t\tgit branch commits/13 &&\n+\t\tgit commit-graph write --reachable --split &&\n+\t\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\t\ttest_line_count = 3 $graphdir/commit-graph-chain &&\n+\t\tls $graphdir/graph-*.graph >graph-files &&\n+\t\ttest_line_count = 1 graph-files &&\n+\t\tgit -c core.commitGraph=true rev-list HEAD >expect &&\n+\t\tgit -c core.commitGraph=false rev-list HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+graph_git_behavior 'alternate: commit 13 vs 6' commits/13 commits/6\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"376076","messageId":"9567daa0b88e9fa2e755d9060341c7a39629ea86.1558554800.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 09/11] commit-graph: merge commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-22T19:53:28Z","receivedAt":"2019-05-22T19:53:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen searching for a commit in a commit-graph chain of G graphs with N\ncommits, the search takes O(G log N) time. If we always add a new tip\ngraph with every write, the linear G term will start to dominate and\nslow the lookup process.\n\nTo keep lookups fast, but also keep most incremental writes fast, create\na strategy for merging levels of the commit-graph chain. The strategy is\ndetailed in the commit-graph design document, but is summarized by these\ntwo conditions:\n\n 1. If the number of commits we are adding is more than half the number\n of commits in the graph below, then merge with that graph.\n\n 2. If we are writing more than 64,000 commits into a single graph,\n then merge with all lower graphs.\n\nThe numeric values in the conditions above are currently constant, but\ncan become config options in a future update.\n\nAs we merge levels of the commit-graph chain, check that the commits\nstill exist in the repository. A garbage-collection operation may have\nremoved those commits from the object store and we do not want to\npersist them in the commit-graph chain. This is a non-issue if the\n'git gc' process wrote a new, single-level commit-graph file.\n\nAfter we merge levels, the old graph-{hash}.graph files are no longer\nreferenced by the commit-graph-chain file. We will expire these files in\na future change.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 81 ++++++++++\n commit-graph.c | 190 +++++++++++++++++++----\n t/t5323-split-commit-graph.sh | 13 ++\n 3 files changed, 251 insertions(+), 33 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex 1dca3bd8fe..083fff9927 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -186,6 +186,87 @@ positions to refer to their parents, which may be in `graph-{hash1}.graph` or\n its containment in the intervals [0, X0), [X0, X0 + X1), [X0 + X1, X0 + X1 +\n X2).\n \n+Each commit-graph file (except the base, `graph-{hash0}.graph`) contains data\n+specifying the hashes of all files in the lower layers. In the above example,\n+`graph-{hash1}.graph` contains `{hash0}` while `graph-{hash2}.graph` contains\n+`{hash0}` and `{hash1}`.\n+\n+## Merging commit-graph files\n+\n+If we only added a new commit-graph file on every write, we would run into a\n+linear search problem through many commit-graph files. Instead, we use a merge\n+strategy to decide when the stack should collapse some number of levels.\n+\n+The diagram below shows such a collapse. As a set of new commits are added, it\n+is determined by the merge strategy that the files should collapse to\n+`graph-{hash1}`. Thus, the new commits, the commits in `graph-{hash2}` and\n+the commits in `graph-{hash1}` should be combined into a new `graph-{hash3}`\n+file.\n+\n+\t\t\t +---------------------+\n+\t\t\t | |\n+\t\t\t | (new commits) |\n+\t\t\t | |\n+\t\t\t +---------------------+\n+\t\t\t | |\n+ +-----------------------+ +---------------------+\n+ | graph-{hash2} |->| |\n+ +-----------------------+ +---------------------+\n+\t | | |\n+ +-----------------------+ +---------------------+\n+ | | | |\n+ | graph-{hash1} |->| |\n+ | | | |\n+ +-----------------------+ +---------------------+\n+\t | tmp_graphXXX\n+ +-----------------------+\n+ | |\n+ | |\n+ | |\n+ | graph-{hash0} |\n+ | |\n+ | |\n+ | |\n+ +-----------------------+\n+\n+During this process, the commits to write are combined, sorted and we write the\n+contents to a temporary file, all while holding a `commit-graph-chain.lock`\n+lock-file. When the file is flushed, we rename it to `graph-{hash3}`\n+according to the computed `{hash3}`. Finally, we write the new chain data to\n+`commit-graph-chain.lock`:\n+\n+```\n+\t{hash3}\n+\t{hash0}\n+```\n+\n+We then close the lock-file.\n+\n+## Merge Strategy\n+\n+When writing a set of commits that do not exist in the commit-graph stack of\n+height N, we default to creating a new file at level N + 1. We then decide to\n+merge with the Nth level if one of two conditions hold:\n+\n+ 1. The expected file size for level N + 1 is at least half the file size for\n+ level N.\n+\n+ 2. Level N + 1 contains more than MAX_SPLIT_COMMITS commits (64,0000\n+ commits).\n+\n+This decision cascades down the levels: when we merge a level we create a new\n+set of commits that then compares to the next level.\n+\n+The first condition bounds the number of levels to be logarithmic in the total\n+number of commits. The second condition bounds the total number of commits in\n+a `graph-{hashN}` file and not in the `commit-graph` file, preventing\n+significant performance issues when the stack merges and another process only\n+partially reads the previous stack.\n+\n+The merge strategy values (2 for the size multiple, 64,000 for the maximum\n+number of commits) could be extracted into config settings for full\n+flexibility.\n+\n Related Links\n -------------\n [0] https://bugs.chromium.org/p/git/issues/detail?id=8\ndiff --git a/commit-graph.c b/commit-graph.c\nindex fb972c0bc2..e9784cd559 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1277,36 +1277,6 @@ static int write_graph_chunk_base(struct hashfile *f,\n \treturn 0;\n }\n \n-static void init_commit_graph_chain(struct write_commit_graph_context *ctx)\n-{\n-\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n-\tuint32_t i;\n-\n-\tctx->new_base_graph = g;\n-\tctx->base_graph_name = xstrdup(g->filename);\n-\tctx->new_num_commits_in_base = g->num_commits + g->num_commits_in_base;\n-\n-\tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n-\n-\tALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);\n-\tALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);\n-\n-\tfor (i = 0; i < ctx->num_commit_graphs_before - 1; i++)\n-\t\tctx->commit_graph_filenames_after[i] = xstrdup(ctx->commit_graph_filenames_before[i]);\n-\n-\tif (ctx->num_commit_graphs_before)\n-\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_before - 1] =\n-\t\t\tget_split_graph_filename(ctx->obj_dir, oid_to_hex(&g->oid));\n-\n-\ti = ctx->num_commit_graphs_before - 1;\n-\n-\twhile (g) {\n-\t\tctx->commit_graph_hash_after[i] = xstrdup(oid_to_hex(&g->oid));\n-\t\ti--;\n-\t\tg = g->base_graph;\n-\t}\n-}\n-\n static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n@@ -1484,6 +1454,155 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \treturn 0;\n }\n \n+static int split_strategy_max_commits = 64000;\n+static float split_strategy_size_mult = 2.0f;\n+\n+static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n+{\n+\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\tuint32_t num_commits = ctx->commits.nr;\n+\tuint32_t i;\n+\n+\tg = ctx->r->objects->commit_graph;\n+\tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n+\n+\twhile (g && (g->num_commits <= split_strategy_size_mult * num_commits ||\n+\t\t num_commits > split_strategy_max_commits)) {\n+\t\tnum_commits += g->num_commits;\n+\t\tg = g->base_graph;\n+\n+\t\tctx->num_commit_graphs_after--;\n+\t}\n+\n+\tctx->new_base_graph = g;\n+\n+\tALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);\n+\tALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);\n+\n+\tfor (i = 0; i < ctx->num_commit_graphs_after &&\n+\t\t i < ctx->num_commit_graphs_before; i++)\n+\t\tctx->commit_graph_filenames_after[i] = xstrdup(ctx->commit_graph_filenames_before[i]);\n+\n+\ti = ctx->num_commit_graphs_before - 1;\n+\tg = ctx->r->objects->commit_graph;\n+\n+\twhile (g) {\n+\t\tif (i < ctx->num_commit_graphs_after)\n+\t\t\tctx->commit_graph_hash_after[i] = xstrdup(oid_to_hex(&g->oid));\n+\n+\t\ti--;\n+\t\tg = g->base_graph;\n+\t}\n+}\n+\n+static void merge_commit_graph(struct write_commit_graph_context *ctx,\n+\t\t\t struct commit_graph *g)\n+{\n+\tuint32_t i;\n+\tuint32_t offset = g->num_commits_in_base;\n+\n+\tALLOC_GROW(ctx->commits.list, ctx->commits.nr + g->num_commits, ctx->commits.alloc);\n+\n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct object_id oid;\n+\t\tstruct commit *result;\n+\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\n+\t\tload_oid_from_graph(g, i + offset, &oid);\n+\n+\t\t/* only add commits if they still exist in the repo */\n+\t\tresult = lookup_commit_reference_gently(ctx->r, &oid, 1);\n+\n+\t\tif (result) {\n+\t\t\tctx->commits.list[ctx->commits.nr] = result;\n+\t\t\tctx->commits.nr++;\n+\t\t}\n+\t}\n+}\n+\n+static int commit_compare(const void *_a, const void *_b)\n+{\n+\tconst struct commit *a = *(const struct commit **)_a;\n+\tconst struct commit *b = *(const struct commit **)_b;\n+\treturn oidcmp(&a->object.oid, &b->object.oid);\n+}\n+\n+static void deduplicate_commits(struct write_commit_graph_context *ctx)\n+{\n+\tuint32_t i, num_parents, last_distinct = 0, duplicates = 0;\n+\tstruct commit_list *parent;\n+\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"De-duplicating merged commits\"),\n+\t\t\t\t\tctx->commits.nr);\n+\n+\tQSORT(ctx->commits.list, ctx->commits.nr, commit_compare);\n+\n+\tctx->num_extra_edges = 0;\n+\tfor (i = 1; i < ctx->commits.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i);\n+\n+\t\tif (oideq(&ctx->commits.list[last_distinct]->object.oid,\n+\t\t\t &ctx->commits.list[i]->object.oid)) {\n+\t\t\tduplicates++;\n+\t\t} else {\n+\t\t\tif (duplicates)\n+\t\t\t\tctx->commits.list[last_distinct + 1] = ctx->commits.list[i];\n+\t\t\tlast_distinct++;\n+\n+\t\t\tnum_parents = 0;\n+\t\t\tfor (parent = ctx->commits.list[i]->parents; parent; parent = parent->next)\n+\t\t\t\tnum_parents++;\n+\n+\t\t\tif (num_parents > 2)\n+\t\t\t\tctx->num_extra_edges += num_parents - 2;\n+\t\t}\n+\t}\n+\n+\tctx->commits.nr -= duplicates;\n+\tstop_progress(&ctx->progress);\n+}\n+\n+static void merge_commit_graphs(struct write_commit_graph_context *ctx)\n+{\n+\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\tuint32_t current_graph_number = ctx->num_commit_graphs_before;\n+\tstruct strbuf progress_title = STRBUF_INIT;\n+\n+\twhile (g && current_graph_number >= ctx->num_commit_graphs_after) {\n+\t\tcurrent_graph_number--;\n+\n+\t\tif (ctx->report_progress) {\n+\t\t\tif (current_graph_number)\n+\t\t\t\tstrbuf_addf(&progress_title,\n+\t\t\t\t\t _(\"Merging commit-graph-%d\"),\n+\t\t\t\t\t current_graph_number);\n+\t\t\telse\n+\t\t\t\tstrbuf_addstr(&progress_title,\n+\t\t\t\t\t _(\"Merging commit-graph\"));\n+\t\t\tctx->progress = start_delayed_progress(progress_title.buf, 0);\n+\t\t}\n+\n+\t\tmerge_commit_graph(ctx, g);\n+\t\tstop_progress(&ctx->progress);\n+\t\tstrbuf_release(&progress_title);\n+\n+\t\tg = g->base_graph;\n+\t}\n+\n+\tif (g) {\n+\t\tctx->new_base_graph = g;\n+\t\tctx->new_num_commits_in_base = g->num_commits + g->num_commits_in_base;\n+\t}\n+\n+\tif (ctx->new_base_graph)\n+\t\tctx->base_graph_name = xstrdup(ctx->new_base_graph->filename);\n+\n+\tdeduplicate_commits(ctx);\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -1529,6 +1648,9 @@ int write_commit_graph(const char *obj_dir,\n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n \n+\tif (ctx->split && ctx->oids.alloc > split_strategy_max_commits)\n+\t\tctx->oids.alloc = split_strategy_max_commits;\n+\n \tif (ctx->append) {\n \t\tprepare_commit_graph_one(ctx->r, ctx->obj_dir);\n \t\tif (ctx->r->objects->commit_graph)\n@@ -1582,9 +1704,11 @@ int write_commit_graph(const char *obj_dir,\n \tif (!ctx->commits.nr)\n \t\tgoto cleanup;\n \n-\tif (ctx->split)\n-\t\tinit_commit_graph_chain(ctx);\n-\telse\n+\tif (ctx->split) {\n+\t\tsplit_graph_merge_strategy(ctx);\n+\n+\t\tmerge_commit_graphs(ctx);\n+\t} else\n \t\tctx->num_commit_graphs_after = 1;\n \n \tcompute_generation_numbers(ctx);\ndiff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\nindex 96704b9f5b..3902fd9aee 100755\n--- a/t/t5323-split-commit-graph.sh\n+++ b/t/t5323-split-commit-graph.sh\n@@ -119,4 +119,17 @@ test_expect_success 'add one commit, write a tip graph' '\n \n graph_git_behavior 'three-layer commit-graph: commit 11 vs 6' commits/11 commits/6\n \n+test_expect_success 'add one commit, write a merged graph' '\n+\ttest_commit 12 &&\n+\tgit branch commits/12 &&\n+\tgit commit-graph write --reachable --split &&\n+\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\tls $graphdir/graph-*.graph >graph-files &&\n+\ttest_line_count = 4 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n+graph_git_behavior 'merged commit-graph: commit 12 vs 6' commits/12 commits/6\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"376077","messageId":"pull.184.v2.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.git.gitgitgadget@gmail.com","subject":"[PATCH v2 00/11] [RFC] Commit-graph: Write incremental files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-22T19:53:21Z","receivedAt":"2019-05-22T19:53:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This patch series is marked as RFC quality because it is missing some key\nfeatures and tests, but hopefully starts a concrete discussion of how the\nincremental commit-graph writes can work. This version takes the design\nsuggestions from the earlier discussion and tries to work out most of the\nconcerns.\n\nThe commit-graph is a valuable performance feature for repos with large\ncommit histories, but suffers from the same problem as git repack: it\nrewrites the entire file every time. This can be slow when there are\nmillions of commits, especially after we stopped reading from the\ncommit-graph file during a write in 43d3561 (commit-graph write: don't die\nif the existing graph is corrupt).\n\nInstead, create a \"chain\" of commit-graphs in the\n.git/objects/info/commit-graphs folder with name graph-{hash}.graph. The\nlist of hashes is given by the commit-graph-chain file, and also in a \"base\ngraph chunk\" in the commit-graph format. As we read a chain, we can verify\nthat the hashes match the trailing hash of each commit-graph we read along\nthe way and each hash below a level is expected by that graph file.\n\nWhen writing, we don't always want to add a new level to the stack. This\nwould eventually result in performance degradation, especially when\nsearching for a commit (before we know its graph position). We decide to\nmerge levels of the stack when the new commits we will write satisfy two\nconditions:\n\n 1. The expected size of the new file is more than half the size of the tip\n of the stack.\n 2. The new file contains more than 64,000 commits.\n\nThe first condition alone would prevent more than a logarithmic number of\nlevels. The second condition is a stop-gap to prevent performance issues\nwhen another process starts reading the commit-graph stack as we are merging\na large stack of commit-graph files. The reading process could be in a state\nwhere the new file is not ready, but the levels above the new file were\nalready deleted. Thus, the commits that were merged down must be parsed from\npack-files.\n\nThe performance is necessarily amortized across multiple writes, so I tested\nby writing commit-graphs from the (non-rc) tags in the Linux repo. My test\nincluded 72 tags, and wrote everything reachable from the tag using \n--stdin-commits. Here are the overall perf numbers:\n\ngit commit-graph write --stdin-commits: 8m 12s\ngit commit-graph write --stdin-commits --split: 48s\n\nThe test using --split included at least six full collapses to the full\ncommit-graph. I believe the commit-graph stack had at most three levels\nduring this test.\n\nHere are a few points that still need to be addressed before this is ready\nfor full review:\n\n * The merge strategy values should be extracted into config options.\n \n \n * If we have a commit-graph chain and someone writes without \"--split\" it\n will make a new commit-graph file and not clean up the old files.\n \n \n * We need to update 'git commit-graph verify' to understand the chains, and\n test that it catches the new problems. It would be good to have a\n '--shallow' option to only verify the tip file, as if we run that after\n every write we can have some confidence that the files at rest are still\n valid and we only need to check the smaller file. (This is the main\n reason this is a priority to the VFS for Git team.)\n \n \n\nThis is based on ds/commit-graph-write-refactor.\n\nThanks, -Stolee\n\n[1] \nhttps://github.com/git/git/commit/43d356180556180b4ef6ac232a14498a5bb2b446\ncommit-graph write: don't die if the existing graph is corrupt\n\nDerrick Stolee (11):\n commit-graph: document commit-graph chains\n commit-graph: prepare for commit-graph chains\n commit-graph: rename commit_compare to oid_compare\n commit-graph: load commit-graph chains\n commit-graph: add base graphs chunk\n commit-graph: rearrange chunk count logic\n commit-graph: write commit-graph chains\n commit-graph: add --split option to builtin\n commit-graph: merge commit-graph chains\n commit-graph: allow cross-alternate chains\n commit-graph: expire commit-graph files\n\n .../technical/commit-graph-format.txt | 11 +-\n Documentation/technical/commit-graph.txt | 195 +++++\n builtin/commit-graph.c | 10 +-\n commit-graph.c | 734 +++++++++++++++++-\n commit-graph.h | 7 +\n t/t5318-commit-graph.sh | 2 +-\n t/t5323-split-commit-graph.sh | 172 ++++\n 7 files changed, 1088 insertions(+), 43 deletions(-)\n create mode 100755 t/t5323-split-commit-graph.sh\n\n\nbase-commit: 8520d7fc7c6edd4d71582c69a873436029b6cb1b\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-184%2Fderrickstolee%2Fgraph%2Fincremental-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-184/derrickstolee/graph/incremental-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/184\n\nRange-diff vs v1:\n\n 1: 0be7713a25 < -: ---------- commit-graph: fix the_repository reference\n 2: 86b18f15ba < -: ---------- commit-graph: return with errors during write\n 3: 9299a7fe25 < -: ---------- commit-graph: collapse parameters into flags\n 4: f9b719fc7a < -: ---------- commit-graph: remove Future Work section\n 5: 74e40970e0 < -: ---------- commit-graph: create write_commit_graph_context\n 6: 54817ef50b < -: ---------- commit-graph: extract fill_oids_from_packs()\n 7: cf792d38ed < -: ---------- commit-graph: extract fill_oids_from_commit_hex()\n 8: aaae85f1ec < -: ---------- commit-graph: extract fill_oids_from_all_packs()\n 9: 9d434dc38c < -: ---------- commit-graph: extract count_distinct_commits()\n 10: ebd665468e < -: ---------- commit-graph: extract copy_oids_to_commits()\n 11: 3eee3667cf < -: ---------- commit-graph: extract write_commit_graph_file()\n 12: 7bbe8d9150 < -: ---------- Documentation: describe split commit-graphs\n -: ---------- > 1: a423afbfdd commit-graph: document commit-graph chains\n 13: 9d0e966a3d ! 2: 249668fc92 commit-graph: lay groundwork for incremental files\n @@ -1,6 +1,26 @@\n Author: Derrick Stolee \n \n - commit-graph: lay groundwork for incremental files\n + commit-graph: prepare for commit-graph chains\n +\n + To prepare for a chain of commit-graph files, augment the\n + commit_graph struct to point to a base commit_graph. As we load\n + commits from the graph, we may actually want to read from a base\n + file according to the graph position.\n +\n + The \"graph position\" of a commit is given by concatenating the\n + lexicographic commit orders from each of the commit-graph files in\n + the chain. This means that we must distinguish two values:\n +\n + * lexicographic index : the position within the lexicographic\n + order in a single commit-graph file.\n +\n + * graph position: the posiiton within the concatenated order\n + of multiple commit-graph files\n +\n + Given the lexicographic index of a commit in a graph, we can\n + compute the graph position by adding the number of commits in\n + the lower-level graphs. To find the lexicographic index of\n + a commit, we subtract the number of commits in lower-level graphs.\n \n Signed-off-by: Derrick Stolee \n \n @@ -13,21 +33,21 @@\n \n +static void load_oid_from_graph(struct commit_graph *g, int pos, struct object_id *oid)\n +{\n ++\tuint32_t lex_index;\n ++\n +\tif (!g)\n +\t\tBUG(\"NULL commit-graph\");\n +\n -+\tif (pos < g->num_commits_in_base) {\n -+\t\tload_oid_from_graph(g->base_graph, pos, oid);\n -+\t\treturn;\n -+\t}\n ++\twhile (pos < g->num_commits_in_base)\n ++\t\tg = g->base_graph;\n +\n +\tif (pos >= g->num_commits + g->num_commits_in_base)\n +\t\tBUG(\"position %d is beyond the scope of this commit-graph (%d local + %d base commits)\",\n +\t\t pos, g->num_commits, g->num_commits_in_base);\n +\n -+\tpos -= g->num_commits_in_base;\n ++\tlex_index = pos - g->num_commits_in_base;\n +\n -+\thashcpy(oid->hash, g->chunk_oid_lookup + g->hash_len * pos);\n ++\thashcpy(oid->hash, g->chunk_oid_lookup + g->hash_len * lex_index);\n +}\n +\n static struct commit_list **insert_parent_or_die(struct repository *r,\n @@ -47,24 +67,32 @@\n \tif (!c)\n \t\tdie(_(\"could not find commit %s\"), oid_to_hex(&oid));\n @@\n + \n static void fill_commit_graph_info(struct commit *item, struct commit_graph *g, uint32_t pos)\n {\n - \tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n --\titem->graph_pos = pos;\n -+\titem->graph_pos = pos + g->num_commits_in_base;\n +-\tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n ++\tconst unsigned char *commit_data;\n ++\tuint32_t lex_index;\n ++\n ++\twhile (pos < g->num_commits_in_base)\n ++\t\tg = g->base_graph;\n ++\n ++\tlex_index = pos - g->num_commits_in_base;\n ++\tcommit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * lex_index;\n + \titem->graph_pos = pos;\n \titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n }\n - \n @@\n \tuint32_t *parent_data_ptr;\n \tuint64_t date_low, date_high;\n \tstruct commit_list **pptr;\n -\tconst unsigned char *commit_data = g->chunk_commit_data + (g->hash_len + 16) * pos;\n +\tconst unsigned char *commit_data;\n ++\tuint32_t lex_index;\n \n -\titem->object.parsed = 1;\n -+\tif (pos < g->num_commits_in_base)\n -+\t\treturn fill_commit_in_graph(r, item, g->base_graph, pos);\n ++\twhile (pos < g->num_commits_in_base)\n ++\t\tg = g->base_graph;\n +\n +\tif (pos >= g->num_commits + g->num_commits_in_base)\n +\t\tBUG(\"position %d is beyond the scope of this commit-graph (%d local + %d base commits)\",\n @@ -75,9 +103,9 @@\n +\t * \"local\" position for the rest of the calculation.\n +\t */\n \titem->graph_pos = pos;\n -+\tpos -= g->num_commits_in_base;\n ++\tlex_index = pos - g->num_commits_in_base;\n +\n -+\tcommit_data = g->chunk_commit_data + (g->hash_len + 16) * pos;\n ++\tcommit_data = g->chunk_commit_data + (g->hash_len + 16) * lex_index;\n +\n +\titem->object.parsed = 1;\n \n @@ -89,13 +117,13 @@\n \t} else {\n -\t\treturn bsearch_graph(g, &(item->object.oid), pos);\n +\t\tstruct commit_graph *cur_g = g;\n -+\t\tuint32_t pos_in_g;\n ++\t\tuint32_t lex_index;\n +\n -+\t\twhile (cur_g && !bsearch_graph(cur_g, &(item->object.oid), &pos_in_g))\n ++\t\twhile (cur_g && !bsearch_graph(cur_g, &(item->object.oid), &lex_index))\n +\t\t\tcur_g = cur_g->base_graph;\n +\n +\t\tif (cur_g) {\n -+\t\t\t*pos = pos_in_g + cur_g->num_commits_in_base;\n ++\t\t\t*pos = lex_index + cur_g->num_commits_in_base;\n +\t\t\treturn 1;\n +\t\t}\n +\n @@ -111,8 +139,8 @@\n -\t\t\t\t\t GRAPH_DATA_WIDTH * (c->graph_pos);\n +\tconst unsigned char *commit_data;\n +\n -+\tif (c->graph_pos < g->num_commits_in_base)\n -+\t\treturn load_tree_for_commit(r, g->base_graph, c);\n ++\twhile (c->graph_pos < g->num_commits_in_base)\n ++\t\tg = g->base_graph;\n +\n +\tcommit_data = g->chunk_commit_data +\n +\t\t\tGRAPH_DATA_WIDTH * (c->graph_pos - g->num_commits_in_base);\n @@ -133,8 +161,3 @@\n \tconst uint32_t *chunk_oid_fanout;\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_commit_data;\n - \tconst unsigned char *chunk_extra_edges;\n -+\tconst unsigned char *chunk_base_graph;\n - };\n - \n - struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st);\n 14: 4436c8f4f1 < -: ---------- commit-graph: load split commit-graph files\n 15: aa4a096813 < -: ---------- commit-graph: write split commit-graph files\n -: ---------- > 3: 809fa7ad80 commit-graph: rename commit_compare to oid_compare\n -: ---------- > 4: a8c0b47c8a commit-graph: load commit-graph chains\n -: ---------- > 5: 4fefd0a654 commit-graph: add base graphs chunk\n -: ---------- > 6: a595a1eb65 commit-graph: rearrange chunk count logic\n -: ---------- > 7: 9cbfb656b3 commit-graph: write commit-graph chains\n 16: 7c5bc06d14 ! 8: 5ad14f574b commit-graph: add --split option\n @@ -1,6 +1,16 @@\n Author: Derrick Stolee \n \n - commit-graph: add --split option\n + commit-graph: add --split option to builtin\n +\n + Add a new \"--split\" option to the 'git commit-graph write' subcommand. This\n + option allows the optional behavior of writing a commit-graph chain.\n +\n + The current behavior will add a tip commit-graph containing any commits that\n + are not in the existing commit-graph or commit-graph chain. Later changes\n + will allow merging the chain and expiring out-dated files.\n +\n + Add a new test script (t5323-split-commit-graph.sh) that demonstrates this\n + behavior.\n \n Signed-off-by: Derrick Stolee \n \n @@ -55,39 +65,130 @@\n \tread_replace_refs = 0;\n \n \n - diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n - --- a/t/t5318-commit-graph.sh\n - +++ b/t/t5318-commit-graph.sh\n + diff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\n + new file mode 100755\n + --- /dev/null\n + +++ b/t/t5323-split-commit-graph.sh\n @@\n - \ttest_cmp_bin commit-graph-after-gc $objdir/info/commit-graph\n - '\n - \n -+test_expect_success 'write split commit-graph' '\n -+\tcd \"$TRASH_DIRECTORY\" &&\n -+\tgit clone full split &&\n -+\tcd split &&\n ++#!/bin/sh\n ++\n ++test_description='split commit graph'\n ++. ./test-lib.sh\n ++\n ++GIT_TEST_COMMIT_GRAPH=0\n ++\n ++test_expect_success 'setup repo' '\n ++\tgit init &&\n +\tgit config core.commitGraph true &&\n -+\tfor i in $(test_seq 1 20); do\n -+\t\ttest_commit padding-$i\n ++\tinfodir=\".git/objects/info\" &&\n ++\tgraphdir=\"$infodir/commit-graphs\" &&\n ++\ttest_oid_init\n ++'\n ++\n ++graph_read_expect() {\n ++\tNUM_BASE=0\n ++\tif test ! -z $2\n ++\tthen\n ++\t\tNUM_BASE=$2\n ++\tfi\n ++\tcat >expect <<- EOF\n ++\theader: 43475048 1 1 3 $NUM_BASE\n ++\tnum_commits: $1\n ++\tchunks: oid_fanout oid_lookup commit_metadata\n ++\tEOF\n ++\tgit commit-graph read >output &&\n ++\ttest_cmp expect output\n ++}\n ++\n ++test_expect_success 'create commits and write commit-graph' '\n ++\tfor i in $(test_seq 3)\n ++\tdo\n ++\t\ttest_commit $i &&\n ++\t\tgit branch commits/$i\n +\tdone &&\n +\tgit commit-graph write --reachable &&\n -+\ttest_commit split-commit &&\n -+\tgit branch -f split-commit &&\n -+\tgit commit-graph write --reachable --split &&\n -+\ttest_path_is_file .git/objects/info/commit-graphs/commit-graph-1\n ++\ttest_path_is_file $infodir/commit-graph &&\n ++\tgraph_read_expect 3\n +'\n +\n -+graph_git_behavior 'split graph, split-commit vs merge 1' bare split-commit merge/1\n ++graph_git_two_modes() {\n ++\tgit -c core.commitGraph=true $1 >output\n ++\tgit -c core.commitGraph=false $1 >expect\n ++\ttest_cmp expect output\n ++}\n ++\n ++graph_git_behavior() {\n ++\tMSG=$1\n ++\tBRANCH=$2\n ++\tCOMPARE=$3\n ++\ttest_expect_success \"check normal git operations: $MSG\" '\n ++\t\tgraph_git_two_modes \"log --oneline $BRANCH\" &&\n ++\t\tgraph_git_two_modes \"log --topo-order $BRANCH\" &&\n ++\t\tgraph_git_two_modes \"log --graph $COMPARE..$BRANCH\" &&\n ++\t\tgraph_git_two_modes \"branch -vv\" &&\n ++\t\tgraph_git_two_modes \"merge-base -a $BRANCH $COMPARE\"\n ++\t'\n ++}\n +\n -+test_expect_success 'collapse split commit-graph' '\n -+\tcd \"$TRASH_DIRECTORY/split\" &&\n ++graph_git_behavior 'graph exists' commits/3 commits/1\n ++\n ++verify_chain_files_exist() {\n ++\tfor hash in $(cat $1/commit-graph-chain)\n ++\tdo\n ++\t\ttest_path_is_file $1/graph-$hash.graph\n ++\tdone\n ++}\n ++\n ++test_expect_success 'add more commits, and write a new base graph' '\n ++\tgit reset --hard commits/1 &&\n ++\tfor i in $(test_seq 4 5)\n ++\tdo\n ++\t\ttest_commit $i &&\n ++\t\tgit branch commits/$i\n ++\tdone &&\n ++\tgit reset --hard commits/2 &&\n ++\tfor i in $(test_seq 6 10)\n ++\tdo\n ++\t\ttest_commit $i &&\n ++\t\tgit branch commits/$i\n ++\tdone &&\n ++\tgit reset --hard commits/2 &&\n ++\tgit merge commits/4 &&\n ++\tgit branch merge/1 &&\n ++\tgit reset --hard commits/4 &&\n ++\tgit merge commits/6 &&\n ++\tgit branch merge/2 &&\n +\tgit commit-graph write --reachable &&\n -+\ttest_path_is_missing .git/objects/info/commit-graphs/commit-graph-1 &&\n -+\ttest_path_is_file .git/objects/info/commit-graph\n ++\tgraph_read_expect 12\n ++'\n ++\n ++test_expect_success 'add three more commits, write a tip graph' '\n ++\tgit reset --hard commits/3 &&\n ++\tgit merge merge/1 &&\n ++\tgit merge commits/5 &&\n ++\tgit merge merge/2 &&\n ++\tgit branch merge/3 &&\n ++\tgit commit-graph write --reachable --split &&\n ++\ttest_path_is_missing $infodir/commit-graph &&\n ++\ttest_path_is_file $graphdir/commit-graph-chain &&\n ++\tls $graphdir/graph-*.graph >graph-files &&\n ++\ttest_line_count = 2 graph-files &&\n ++\tverify_chain_files_exist $graphdir\n ++'\n ++\n ++graph_git_behavior 'split commit-graph: merge 3 vs 2' merge/3 merge/2\n ++\n ++test_expect_success 'add one commit, write a tip graph' '\n ++\ttest_commit 11 &&\n ++\tgit branch commits/11 &&\n ++\tgit commit-graph write --reachable --split &&\n ++\ttest_path_is_missing $infodir/commit-graph &&\n ++\ttest_path_is_file $graphdir/commit-graph-chain &&\n ++\tls $graphdir/graph-*.graph >graph-files &&\n ++\ttest_line_count = 3 graph-files &&\n ++\tverify_chain_files_exist $graphdir\n +'\n +\n -+graph_git_behavior 'collapsed graph, split-commit vs merge 1' bare split-commit merge/1\n ++graph_git_behavior 'three-layer commit-graph: commit 11 vs 6' commits/11 commits/6\n +\n - test_expect_success 'replace-objects invalidates commit-graph' '\n - \tcd \"$TRASH_DIRECTORY\" &&\n - \ttest_when_finished rm -rf replace &&\n ++test_done\n 17: 3c52385e56 < -: ---------- fetch: add fetch.writeCommitGraph config setting\n -: ---------- > 9: 9567daa0b8 commit-graph: merge commit-graph chains\n -: ---------- > 10: 4cfe19a933 commit-graph: allow cross-alternate chains\n -: ---------- > 11: 72fc0a1f17 commit-graph: expire commit-graph files\n\n-- \ngitgitgadget\n"},{"id":"376078","messageId":"72fc0a1f17e006fa4de4dd56209364d7430b02d9.1558554800.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 11/11] commit-graph: expire commit-graph files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-05-22T19:53:30Z","receivedAt":"2019-05-22T19:53:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we merge commit-graph files in a commit-graph chain, we should clean\nup the files that are no longer used.\n\nThis change introduces an 'expiry_window' value to the context, which is\nalways zero (for now). We then check the modified time of each\ngraph-{hash}.graph file in the $OBJDIR/info/commit-graphs folder and\nunlink the files that are older than the expiry_window.\n\nSince this is always zero, this immediately clears all unused graph\nfiles. We will update the value to match a config setting in a future\nchange.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 15 +++++\n commit-graph.c | 71 ++++++++++++++++++++++++\n t/t5323-split-commit-graph.sh | 2 +-\n 3 files changed, 87 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex b6fe8b2321..4ecec54148 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -267,6 +267,21 @@ The merge strategy values (2 for the size multiple, 64,000 for the maximum\n number of commits) could be extracted into config settings for full\n flexibility.\n \n+## Deleting graph-{hash} files\n+\n+After a new tip file is written, some `graph-{hash}` files may no longer\n+be part of a chain. It is important to remove these files from disk, eventually.\n+The main reason to delay removal is that another process could read the\n+`commit-graph-chain` file before it is rewritten, but then look for the\n+`graph-{hash}` files after they are deleted.\n+\n+To allow holding old split commit-graphs for a while after they are unreferenced,\n+we update the modified times of the files when they become unreferenced. Then,\n+we scan the `$OBJDIR/info/commit-graphs/` directory for `graph-{hash}`\n+files whose modified times are older than a given expiry window. This window\n+defaults to zero, but can be changed using command-line arguments or a config\n+setting.\n+\n ## Chains across multiple object directories\n \n In a repo with alternates, we look for the `commit-graph-chain` file starting\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0d8b942e2b..69c4f83c2a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -739,6 +739,8 @@ struct write_commit_graph_context {\n \tunsigned append:1,\n \t\t report_progress:1,\n \t\t split:1;\n+\n+\ttime_t expire_window;\n };\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n@@ -1629,6 +1631,70 @@ static void merge_commit_graphs(struct write_commit_graph_context *ctx)\n \tdeduplicate_commits(ctx);\n }\n \n+static void mark_commit_graphs(struct write_commit_graph_context *ctx)\n+{\n+\tuint32_t i;\n+\ttime_t now = time(NULL);\n+\n+\tfor (i = ctx->num_commit_graphs_after - 1; i < ctx->num_commit_graphs_before; i++) {\n+\t\tstruct stat st;\n+\t\tstruct utimbuf updated_time;\n+\n+\t\tstat(ctx->commit_graph_filenames_before[i], &st);\n+\n+\t\tupdated_time.actime = st.st_atime;\n+\t\tupdated_time.modtime = now;\n+\t\tutime(ctx->commit_graph_filenames_before[i], &updated_time);\n+\t}\n+}\n+\n+static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tDIR *dir;\n+\tstruct dirent *de;\n+\tsize_t dirnamelen;\n+\ttime_t expire_time = time(NULL) - ctx->expire_window;\n+\n+\tstrbuf_addstr(&path, ctx->obj_dir);\n+\tstrbuf_addstr(&path, \"/info/commit-graphs\");\n+\tdir = opendir(path.buf);\n+\n+\tif (!dir) {\n+\t\tstrbuf_release(&path);\n+\t\treturn;\n+\t}\n+\n+\tstrbuf_addch(&path, '/');\n+\tdirnamelen = path.len;\n+\twhile ((de = readdir(dir)) != NULL) {\n+\t\tstruct stat st;\n+\t\tuint32_t i, found = 0;\n+\n+\t\tstrbuf_setlen(&path, dirnamelen);\n+\t\tstrbuf_addstr(&path, de->d_name);\n+\n+\t\tstat(path.buf, &st);\n+\n+\t\tif (st.st_mtime > expire_time)\n+\t\t\tcontinue;\n+\t\tif (path.len < 6 || strcmp(path.buf + path.len - 6, \".graph\"))\n+\t\t\tcontinue;\n+\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++) {\n+\t\t\tif (!strcmp(ctx->commit_graph_filenames_after[i],\n+\t\t\t\t path.buf)) {\n+\t\t\t\tfound = 1;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (!found)\n+\t\t\tunlink(path.buf);\n+\n+\t}\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -1741,6 +1807,11 @@ int write_commit_graph(const char *obj_dir,\n \n \tres = write_commit_graph_file(ctx);\n \n+\tif (ctx->split) {\n+\t\tmark_commit_graphs(ctx);\n+\t\texpire_commit_graphs(ctx);\n+\t}\n+\n cleanup:\n \tfree(ctx->graph_name);\n \tfree(ctx->commits.list);\ndiff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\nindex ed5cac8617..6215efcac5 100755\n--- a/t/t5323-split-commit-graph.sh\n+++ b/t/t5323-split-commit-graph.sh\n@@ -141,7 +141,7 @@ test_expect_success 'add one commit, write a merged graph' '\n \ttest_path_is_file $graphdir/commit-graph-chain &&\n \ttest_line_count = 2 $graphdir/commit-graph-chain &&\n \tls $graphdir/graph-*.graph >graph-files &&\n-\ttest_line_count = 4 graph-files &&\n+\ttest_line_count = 2 graph-files &&\n \tverify_chain_files_exist $graphdir\n '\n \n-- \ngitgitgadget\n"},{"id":"376090","messageId":"87zhnet2zp.fsf@evledraar.gmail.com","threadId":"51052","inReplyTo":"9567daa0b88e9fa2e755d9060341c7a39629ea86.1558554800.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 09/11] commit-graph: merge commit-graph chains","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-05-23T00:43:06Z","receivedAt":"2019-05-23T00:44:21Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, May 22 2019, Derrick Stolee via GitGitGadget wrote:\n\n> To keep lookups fast, but also keep most incremental writes fast, create\n> a strategy for merging levels of the commit-graph chain. The strategy is\n> detailed in the commit-graph design document, but is summarized by these\n> two conditions:\n>\n> 1. If the number of commits we are adding is more than half the number\n> of commits in the graph below, then merge with that graph.\n>\n> 2. If we are writing more than 64,000 commits into a single graph,\n> then merge with all lower graphs.\n>\n> The numeric values in the conditions above are currently constant, but\n> can become config options in a future update.\n> [...]\n> +## Merge Strategy\n> +\n> +When writing a set of commits that do not exist in the commit-graph stack of\n> +height N, we default to creating a new file at level N + 1. We then decide to\n> +merge with the Nth level if one of two conditions hold:\n> +\n> + 1. The expected file size for level N + 1 is at least half the file size for\n> + level N.\n> +\n> + 2. Level N + 1 contains more than MAX_SPLIT_COMMITS commits (64,0000\n> + commits).\n> +\n> +This decision cascades down the levels: when we merge a level we create a new\n> +set of commits that then compares to the next level.\n> +\n> +The first condition bounds the number of levels to be logarithmic in the total\n> +number of commits. The second condition bounds the total number of commits in\n> +a `graph-{hashN}` file and not in the `commit-graph` file, preventing\n> +significant performance issues when the stack merges and another process only\n> +partially reads the previous stack.\n> +\n> +The merge strategy values (2 for the size multiple, 64,000 for the maximum\n> +number of commits) could be extracted into config settings for full\n> +flexibility.\n\nAs noted this can become configurable, so it's no big deal. But is there\nany reason for ths 64K limit anymore?\n\nWhile with the default expiry of 0sec we can still get that race, it\nseems unlikely in practice, as the \"commit-graph write\" process would\nwrite a new manifest at the end, then go and unlink() the old files.\n\nSo maybe at this point we could make this even dumber with something\nthat behaves like gc.autoPackLimit? I.e. keep writing new graphs, and\nthen coalesce them all (or maybe not the \"base\" graph, like\ngc.bigPackThreshold)?\n\nAlso: These docs refer to MAX_SPLIT_COMMITS, but in v2 it's now a\n\"split_strategy_max_commits\" variable instead.\n"},{"id":"376104","messageId":"ae1189cf-d9a6-b5d9-6b0b-e460d59dc9a0@gmail.com","threadId":"51052","inReplyTo":"87zhnet2zp.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 09/11] commit-graph: merge commit-graph chains","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-05-23T13:00:56Z","receivedAt":"2019-05-23T13:01:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/22/2019 8:43 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, May 22 2019, Derrick Stolee via GitGitGadget wrote:\n> \n>> To keep lookups fast, but also keep most incremental writes fast, create\n>> a strategy for merging levels of the commit-graph chain. The strategy is\n>> detailed in the commit-graph design document, but is summarized by these\n>> two conditions:\n>>\n>> 1. If the number of commits we are adding is more than half the number\n>> of commits in the graph below, then merge with that graph.\n>>\n>> 2. If we are writing more than 64,000 commits into a single graph,\n>> then merge with all lower graphs.\n>>\n>> The numeric values in the conditions above are currently constant, but\n>> can become config options in a future update.\n>> [...]\n>> +## Merge Strategy\n>> +\n>> +When writing a set of commits that do not exist in the commit-graph stack of\n>> +height N, we default to creating a new file at level N + 1. We then decide to\n>> +merge with the Nth level if one of two conditions hold:\n>> +\n>> + 1. The expected file size for level N + 1 is at least half the file size for\n>> + level N.\n>> +\n>> + 2. Level N + 1 contains more than MAX_SPLIT_COMMITS commits (64,0000\n>> + commits).\n>> +\n>> +This decision cascades down the levels: when we merge a level we create a new\n>> +set of commits that then compares to the next level.\n>> +\n>> +The first condition bounds the number of levels to be logarithmic in the total\n>> +number of commits. The second condition bounds the total number of commits in\n>> +a `graph-{hashN}` file and not in the `commit-graph` file, preventing\n>> +significant performance issues when the stack merges and another process only\n>> +partially reads the previous stack.\n>> +\n>> +The merge strategy values (2 for the size multiple, 64,000 for the maximum\n>> +number of commits) could be extracted into config settings for full\n>> +flexibility.\n> \n> As noted this can become configurable, so it's no big deal. But is there\n> any reason for ths 64K limit anymore?\n\nThere may not be an important reason to include it by default. Whatever config\noption we use could have special values:\n\n * -1: No maximum commit limit (default)\n * 0: Never write more than one level.\n\nI would personally set a limit somewhere around 64,000 to prevent long chains\nand having some processes hit a concurrency issue where their run-time\ncommit-graph is effectively 100,000 commits behind.\n\n> While with the default expiry of 0sec we can still get that race, it\n> seems unlikely in practice, as the \"commit-graph write\" process would\n> write a new manifest at the end, then go and unlink() the old files.\n\nThe process reading an old manifest can still only succeed partially as it\nreads the chain from bottom-to-top. Succeeding gracefully in this case is\nbuilt-in, but should have a test! (noted)\n\nBut I don't think the concurrency window is any more lenient than before.\nWe now have the flexibility to set a non-zero expiry window. That was\nimpossible in the other model.\n\n> So maybe at this point we could make this even dumber with something\n> that behaves like gc.autoPackLimit? I.e. keep writing new graphs, and\n> then coalesce them all (or maybe not the \"base\" graph, like\n> gc.bigPackThreshold)?\n\nI will look into these existing settings and try to include similar options\nhere. Better to stay consistent when we can.\n\n> Also: These docs refer to MAX_SPLIT_COMMITS, but in v2 it's now a\n> \"split_strategy_max_commits\" variable instead.\n\nThanks for catching my out-of-date documentation.\n\n-Stolee\n\n"},{"id":"376236","messageId":"20190527112828.GK951@szeder.dev","threadId":"51052","inReplyTo":"5ad14f574b6ffa5e233d61098dbcfc29c3632be5.1558554800.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 08/11] commit-graph: add --split option to builtin","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-05-27T11:28:28Z","receivedAt":"2019-05-27T11:28:35Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, May 22, 2019 at 12:53:27PM -0700, Derrick Stolee via GitGitGadget wrote:\n> diff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\n> new file mode 100755\n> index 0000000000..96704b9f5b\n> --- /dev/null\n> +++ b/t/t5323-split-commit-graph.sh\n> @@ -0,0 +1,122 @@\n> +#!/bin/sh\n> +\n> +test_description='split commit graph'\n> +. ./test-lib.sh\n> +\n> +GIT_TEST_COMMIT_GRAPH=0\n> +\n> +test_expect_success 'setup repo' '\n> +\tgit init &&\n> +\tgit config core.commitGraph true &&\n> +\tinfodir=\".git/objects/info\" &&\n> +\tgraphdir=\"$infodir/commit-graphs\" &&\n> +\ttest_oid_init\n> +'\n> +\n> +graph_read_expect() {\n> +\tNUM_BASE=0\n> +\tif test ! -z $2\n> +\tthen\n> +\t\tNUM_BASE=$2\n> +\tfi\n> +\tcat >expect <<- EOF\n> +\theader: 43475048 1 1 3 $NUM_BASE\n> +\tnum_commits: $1\n> +\tchunks: oid_fanout oid_lookup commit_metadata\n> +\tEOF\n> +\tgit commit-graph read >output &&\n> +\ttest_cmp expect output\n> +}\n> +\n> +test_expect_success 'create commits and write commit-graph' '\n> +\tfor i in $(test_seq 3)\n> +\tdo\n> +\t\ttest_commit $i &&\n> +\t\tgit branch commits/$i\n> +\tdone &&\n\nPlease add a \"|| return 1\" at the end of the for loop's body, i.e.\n\n for ....\n do\n this &&\n that || return 1\n done\n\nbecause for loops continue iteration even when the commands in their\nbody failed, potentially hinding errors.\n\nThis applies to the other three for loops below as well.\n\n> +\tgit commit-graph write --reachable &&\n> +\ttest_path_is_file $infodir/commit-graph &&\n> +\tgraph_read_expect 3\n> +'\n> +\n> +graph_git_two_modes() {\n> +\tgit -c core.commitGraph=true $1 >output\n> +\tgit -c core.commitGraph=false $1 >expect\n> +\ttest_cmp expect output\n> +}\n> +\n> +graph_git_behavior() {\n> +\tMSG=$1\n> +\tBRANCH=$2\n> +\tCOMPARE=$3\n> +\ttest_expect_success \"check normal git operations: $MSG\" '\n> +\t\tgraph_git_two_modes \"log --oneline $BRANCH\" &&\n> +\t\tgraph_git_two_modes \"log --topo-order $BRANCH\" &&\n> +\t\tgraph_git_two_modes \"log --graph $COMPARE..$BRANCH\" &&\n> +\t\tgraph_git_two_modes \"branch -vv\" &&\n> +\t\tgraph_git_two_modes \"merge-base -a $BRANCH $COMPARE\"\n> +\t'\n> +}\n> +\n> +graph_git_behavior 'graph exists' commits/3 commits/1\n> +\n> +verify_chain_files_exist() {\n> +\tfor hash in $(cat $1/commit-graph-chain)\n> +\tdo\n> +\t\ttest_path_is_file $1/graph-$hash.graph\n> +\tdone\n> +}\n> +\n> +test_expect_success 'add more commits, and write a new base graph' '\n> +\tgit reset --hard commits/1 &&\n> +\tfor i in $(test_seq 4 5)\n> +\tdo\n> +\t\ttest_commit $i &&\n> +\t\tgit branch commits/$i\n> +\tdone &&\n> +\tgit reset --hard commits/2 &&\n> +\tfor i in $(test_seq 6 10)\n> +\tdo\n> +\t\ttest_commit $i &&\n> +\t\tgit branch commits/$i\n> +\tdone &&\n> +\tgit reset --hard commits/2 &&\n> +\tgit merge commits/4 &&\n> +\tgit branch merge/1 &&\n> +\tgit reset --hard commits/4 &&\n> +\tgit merge commits/6 &&\n> +\tgit branch merge/2 &&\n> +\tgit commit-graph write --reachable &&\n> +\tgraph_read_expect 12\n> +'\n> +\n> +test_expect_success 'add three more commits, write a tip graph' '\n> +\tgit reset --hard commits/3 &&\n> +\tgit merge merge/1 &&\n> +\tgit merge commits/5 &&\n> +\tgit merge merge/2 &&\n> +\tgit branch merge/3 &&\n> +\tgit commit-graph write --reachable --split &&\n> +\ttest_path_is_missing $infodir/commit-graph &&\n> +\ttest_path_is_file $graphdir/commit-graph-chain &&\n> +\tls $graphdir/graph-*.graph >graph-files &&\n> +\ttest_line_count = 2 graph-files &&\n> +\tverify_chain_files_exist $graphdir\n> +'\n> +\n> +graph_git_behavior 'split commit-graph: merge 3 vs 2' merge/3 merge/2\n> +\n> +test_expect_success 'add one commit, write a tip graph' '\n> +\ttest_commit 11 &&\n> +\tgit branch commits/11 &&\n> +\tgit commit-graph write --reachable --split &&\n> +\ttest_path_is_missing $infodir/commit-graph &&\n> +\ttest_path_is_file $graphdir/commit-graph-chain &&\n> +\tls $graphdir/graph-*.graph >graph-files &&\n> +\ttest_line_count = 3 graph-files &&\n> +\tverify_chain_files_exist $graphdir\n> +'\n> +\n> +graph_git_behavior 'three-layer commit-graph: commit 11 vs 6' commits/11 commits/6\n> +\n> +test_done\n> -- \n> gitgitgadget\n> \n"},{"id":"376575","messageId":"b184919255bfc9e0d57e1a5ca12fb76957879c61.1559577826.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 01/14] commit-graph: document commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-03T16:03:48Z","receivedAt":"2019-06-03T16:03:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd a basic description of commit-graph chains. More details about the\nfeature will be added as we add functionality. This introduction gives a\nhigh-level overview to the goals of the feature and the basic layout of\ncommit-graph chains.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 59 ++++++++++++++++++++++++\n 1 file changed, 59 insertions(+)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex fb53341d5e..1dca3bd8fe 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -127,6 +127,65 @@ Design Details\n helpful for these clones, anyway. The commit-graph will not be read or\n written when shallow commits are present.\n \n+Commit Graphs Chains\n+--------------------\n+\n+Typically, repos grow with near-constant velocity (commits per day). Over time,\n+the number of commits added by a fetch operation is much smaller than the\n+number of commits in the full history. By creating a \"chain\" of commit-graphs,\n+we enable fast writes of new commit data without rewriting the entire commit\n+history -- at least, most of the time.\n+\n+## File Layout\n+\n+A commit-graph chain uses multiple files, and we use a fixed naming convention\n+to organize these files. Each commit-graph file has a name\n+`$OBJDIR/info/commit-graphs/graph-{hash}.graph` where `{hash}` is the hex-\n+valued hash stored in the footer of that file (which is a hash of the file's\n+contents before that hash). For a chain of commit-graph files, a plain-text\n+file at `$OBJDIR/info/commit-graphs/commit-graph-chain` contains the\n+hashes for the files in order from \"lowest\" to \"highest\".\n+\n+For example, if the `commit-graph-chain` file contains the lines\n+\n+```\n+\t{hash0}\n+\t{hash1}\n+\t{hash2}\n+```\n+\n+then the commit-graph chain looks like the following diagram:\n+\n+ +-----------------------+\n+ | graph-{hash2}.graph |\n+ +-----------------------+\n+\t |\n+ +-----------------------+\n+ | |\n+ | graph-{hash1}.graph |\n+ | |\n+ +-----------------------+\n+\t |\n+ +-----------------------+\n+ | |\n+ | |\n+ | |\n+ | graph-{hash0}.graph |\n+ | |\n+ | |\n+ | |\n+ +-----------------------+\n+\n+Let X0 be the number of commits in `graph-{hash0}.graph`, X1 be the number of\n+commits in `graph-{hash1}.graph`, and X2 be the number of commits in\n+`graph-{hash2}.graph`. If a commit appears in position i in `graph-{hash2}.graph`,\n+then we interpret this as being the commit in position (X0 + X1 + i), and that\n+will be used as its \"graph position\". The commits in `graph-{hash2}.graph` use these\n+positions to refer to their parents, which may be in `graph-{hash1}.graph` or\n+`graph-{hash0}.graph`. We can navigate to an arbitrary commit in position j by checking\n+its containment in the intervals [0, X0), [X0, X0 + X1), [X0 + X1, X0 + X1 +\n+X2).\n+\n Related Links\n -------------\n [0] https://bugs.chromium.org/p/git/issues/detail?id=8\n-- \ngitgitgadget\n\n"},{"id":"376576","messageId":"pull.184.v3.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v3 00/14] Commit-graph: Write incremental files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-03T16:03:47Z","receivedAt":"2019-06-03T16:03:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This version is now ready for review.\n\nThe commit-graph is a valuable performance feature for repos with large\ncommit histories, but suffers from the same problem as git repack: it\nrewrites the entire file every time. This can be slow when there are\nmillions of commits, especially after we stopped reading from the\ncommit-graph file during a write in 43d3561 (commit-graph write: don't die\nif the existing graph is corrupt).\n\nInstead, create a \"chain\" of commit-graphs in the\n.git/objects/info/commit-graphs folder with name graph-{hash}.graph. The\nlist of hashes is given by the commit-graph-chain file, and also in a \"base\ngraph chunk\" in the commit-graph format. As we read a chain, we can verify\nthat the hashes match the trailing hash of each commit-graph we read along\nthe way and each hash below a level is expected by that graph file.\n\nWhen writing, we don't always want to add a new level to the stack. This\nwould eventually result in performance degradation, especially when\nsearching for a commit (before we know its graph position). We decide to\nmerge levels of the stack when the new commits we will write satisfy two\nconditions:\n\n 1. The expected size of the new file is more than half the size of the tip\n of the stack.\n 2. The new file contains more than 64,000 commits.\n\nThe first condition alone would prevent more than a logarithmic number of\nlevels. The second condition is a stop-gap to prevent performance issues\nwhen another process starts reading the commit-graph stack as we are merging\na large stack of commit-graph files. The reading process could be in a state\nwhere the new file is not ready, but the levels above the new file were\nalready deleted. Thus, the commits that were merged down must be parsed from\npack-files.\n\nThe performance is necessarily amortized across multiple writes, so I tested\nby writing commit-graphs from the (non-rc) tags in the Linux repo. My test\nincluded 72 tags, and wrote everything reachable from the tag using \n--stdin-commits. Here are the overall perf numbers:\n\nwrite --stdin-commits: 8m 12s\nwrite --stdin-commits --split: 28s\nwrite --split && verify --shallow: 60s\n\nUpdates in V3:\n\n * git commit-graph verify now works on commit-graph chains. We do a simple\n test to check the behavior of a new --shallow option.\n \n \n * When someone writes a flat commit-graph, we now expire the old chain\n according to the expire time.\n \n \n * The \"max commits\" limit is no longer enabled by default, but instead is\n enabled by a --max-commits= option. Ignored if n=0.\n \n \n\nThis is based on ds/commit-graph-write-refactor.\n\nThanks, -Stolee\n\n[1] \nhttps://github.com/git/git/commit/43d356180556180b4ef6ac232a14498a5bb2b446\ncommit-graph write: don't die if the existing graph is corrupt\n\nDerrick Stolee (14):\n commit-graph: document commit-graph chains\n commit-graph: prepare for commit-graph chains\n commit-graph: rename commit_compare to oid_compare\n commit-graph: load commit-graph chains\n commit-graph: add base graphs chunk\n commit-graph: rearrange chunk count logic\n commit-graph: write commit-graph chains\n commit-graph: add --split option to builtin\n commit-graph: merge commit-graph chains\n commit-graph: allow cross-alternate chains\n commit-graph: expire commit-graph files\n commit-graph: create options for split files\n commit-graph: verify chains with --shallow mode\n commit-graph: clean up chains after flattened write\n\n Documentation/git-commit-graph.txt | 26 +-\n .../technical/commit-graph-format.txt | 11 +-\n Documentation/technical/commit-graph.txt | 195 +++++\n builtin/commit-graph.c | 53 +-\n builtin/commit.c | 2 +-\n builtin/gc.c | 3 +-\n commit-graph.c | 780 +++++++++++++++++-\n commit-graph.h | 25 +-\n t/t5318-commit-graph.sh | 2 +-\n t/t5323-split-commit-graph.sh | 240 ++++++\n 10 files changed, 1270 insertions(+), 67 deletions(-)\n create mode 100755 t/t5323-split-commit-graph.sh\n\n\nbase-commit: 8520d7fc7c6edd4d71582c69a873436029b6cb1b\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-184%2Fderrickstolee%2Fgraph%2Fincremental-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-184/derrickstolee/graph/incremental-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/184\n\nRange-diff vs v2:\n\n 1: a423afbfdd = 1: b184919255 commit-graph: document commit-graph chains\n 2: 249668fc92 = 2: d0dc154a27 commit-graph: prepare for commit-graph chains\n 3: 809fa7ad80 = 3: f35b04224a commit-graph: rename commit_compare to oid_compare\n 4: a8c0b47c8a ! 4: ca670536df commit-graph: load commit-graph chains\n @@ -47,13 +47,13 @@\n \treturn load_commit_graph_one_fd_st(fd, &st);\n }\n \n -+static int prepare_commit_graph_v1(struct repository *r, const char *obj_dir)\n ++static struct commit_graph *load_commit_graph_v1(struct repository *r, const char *obj_dir)\n +{\n +\tchar *graph_name = get_commit_graph_filename(obj_dir);\n -+\tr->objects->commit_graph = load_commit_graph_one(graph_name);\n ++\tstruct commit_graph *g = load_commit_graph_one(graph_name);\n +\tfree(graph_name);\n +\n -+\treturn r->objects->commit_graph ? 0 : -1;\n ++\treturn g;\n +}\n +\n +static int add_graph_to_chain(struct commit_graph *g,\n @@ -76,8 +76,9 @@\n +\treturn 1;\n +}\n +\n -+static void prepare_commit_graph_chain(struct repository *r, const char *obj_dir)\n ++static struct commit_graph *load_commit_graph_chain(struct repository *r, const char *obj_dir)\n +{\n ++\tstruct commit_graph *graph_chain = NULL;\n +\tstruct strbuf line = STRBUF_INIT;\n +\tstruct stat st;\n +\tstruct object_id *oids;\n @@ -85,17 +86,21 @@\n +\tchar *chain_name = get_chain_filename(obj_dir);\n +\tFILE *fp;\n +\n -+\tif (stat(chain_name, &st))\n -+\t\treturn;\n ++\tif (stat(chain_name, &st)) {\n ++\t\tfree(chain_name);\n ++\t\treturn NULL;\n ++\t}\n +\n -+\tif (st.st_size <= the_hash_algo->hexsz)\n -+\t\treturn;\n ++\tif (st.st_size <= the_hash_algo->hexsz) {\n ++\t\tfree(chain_name);\n ++\t\treturn NULL;\n ++\t}\n +\n +\tfp = fopen(chain_name, \"r\");\n +\tfree(chain_name);\n +\n +\tif (!fp)\n -+\t\treturn;\n ++\t\treturn NULL;\n +\n +\toids = xcalloc(st.st_size / (the_hash_algo->hexsz + 1), sizeof(struct object_id));\n +\n @@ -114,14 +119,26 @@\n +\t\tg = load_commit_graph_one(graph_name);\n +\t\tfree(graph_name);\n +\n -+\t\tif (g && add_graph_to_chain(g, r->objects->commit_graph, oids, i))\n -+\t\t\tr->objects->commit_graph = g;\n ++\t\tif (g && add_graph_to_chain(g, graph_chain, oids, i))\n ++\t\t\tgraph_chain = g;\n +\t\telse\n +\t\t\tvalid = 0;\n +\t}\n +\n +\tfree(oids);\n +\tfclose(fp);\n ++\n ++\treturn graph_chain;\n ++}\n ++\n ++static struct commit_graph *read_commit_graph_one(struct repository *r, const char *obj_dir)\n ++{\n ++\tstruct commit_graph *g = load_commit_graph_v1(r, obj_dir);\n ++\n ++\tif (!g)\n ++\t\tg = load_commit_graph_chain(r, obj_dir);\n ++\n ++\treturn g;\n +}\n +\n static void prepare_commit_graph_one(struct repository *r, const char *obj_dir)\n @@ -134,11 +151,9 @@\n -\tgraph_name = get_commit_graph_filename(obj_dir);\n -\tr->objects->commit_graph =\n -\t\tload_commit_graph_one(graph_name);\n -+\tif (!prepare_commit_graph_v1(r, obj_dir))\n -+\t\treturn;\n - \n +-\n -\tFREE_AND_NULL(graph_name);\n -+\tprepare_commit_graph_chain(r, obj_dir);\n ++\tr->objects->commit_graph = read_commit_graph_one(r, obj_dir);\n }\n \n /*\n 5: 4fefd0a654 = 5: df44cbc1bf commit-graph: add base graphs chunk\n 6: a595a1eb65 = 6: e65f9e841d commit-graph: rearrange chunk count logic\n 7: 9cbfb656b3 ! 7: fe0aa343cd commit-graph: write commit-graph chains\n @@ -51,7 +51,7 @@\n +\treturn g;\n }\n \n - static int prepare_commit_graph_v1(struct repository *r, const char *obj_dir)\n + static struct commit_graph *load_commit_graph_v1(struct repository *r, const char *obj_dir)\n @@\n \tstruct progress *progress;\n \tint progress_done;\n 8: 5ad14f574b ! 8: 4f4ccc8062 commit-graph: add --split option to builtin\n @@ -104,7 +104,7 @@\n +\tfor i in $(test_seq 3)\n +\tdo\n +\t\ttest_commit $i &&\n -+\t\tgit branch commits/$i\n ++\t\tgit branch commits/$i || return 1\n +\tdone &&\n +\tgit commit-graph write --reachable &&\n +\ttest_path_is_file $infodir/commit-graph &&\n @@ -135,7 +135,7 @@\n +verify_chain_files_exist() {\n +\tfor hash in $(cat $1/commit-graph-chain)\n +\tdo\n -+\t\ttest_path_is_file $1/graph-$hash.graph\n ++\t\ttest_path_is_file $1/graph-$hash.graph || return 1\n +\tdone\n +}\n +\n @@ -144,13 +144,13 @@\n +\tfor i in $(test_seq 4 5)\n +\tdo\n +\t\ttest_commit $i &&\n -+\t\tgit branch commits/$i\n ++\t\tgit branch commits/$i || return 1\n +\tdone &&\n +\tgit reset --hard commits/2 &&\n +\tfor i in $(test_seq 6 10)\n +\tdo\n +\t\ttest_commit $i &&\n -+\t\tgit branch commits/$i\n ++\t\tgit branch commits/$i || return 1\n +\tdone &&\n +\tgit reset --hard commits/2 &&\n +\tgit merge commits/4 &&\n 9: 9567daa0b8 ! 9: 87fb895fe4 commit-graph: merge commit-graph chains\n @@ -105,8 +105,7 @@\n + 1. The expected file size for level N + 1 is at least half the file size for\n + level N.\n +\n -+ 2. Level N + 1 contains more than MAX_SPLIT_COMMITS commits (64,0000\n -+ commits).\n ++ 2. Level N + 1 contains more than 64,0000 commits.\n +\n +This decision cascades down the levels: when we merge a level we create a new\n +set of commits that then compares to the next level.\n @@ -290,13 +289,7 @@\n +\t\tcurrent_graph_number--;\n +\n +\t\tif (ctx->report_progress) {\n -+\t\t\tif (current_graph_number)\n -+\t\t\t\tstrbuf_addf(&progress_title,\n -+\t\t\t\t\t _(\"Merging commit-graph-%d\"),\n -+\t\t\t\t\t current_graph_number);\n -+\t\t\telse\n -+\t\t\t\tstrbuf_addstr(&progress_title,\n -+\t\t\t\t\t _(\"Merging commit-graph\"));\n ++\t\t\tstrbuf_addstr(&progress_title, _(\"Merging commit-graph\"));\n +\t\t\tctx->progress = start_delayed_progress(progress_title.buf, 0);\n +\t\t}\n +\n 10: 4cfe19a933 ! 10: 5cfd653d24 commit-graph: allow cross-alternate chains\n @@ -81,13 +81,13 @@\n --- a/commit-graph.c\n +++ b/commit-graph.c\n @@\n - \tr->objects->commit_graph = load_commit_graph_one(graph_name);\n + \tstruct commit_graph *g = load_commit_graph_one(graph_name);\n \tfree(graph_name);\n \n -+\tif (r->objects->commit_graph)\n -+\t\tr->objects->commit_graph->obj_dir = obj_dir;\n ++\tif (g)\n ++\t\tg->obj_dir = obj_dir;\n +\n - \treturn r->objects->commit_graph ? 0 : -1;\n + \treturn g;\n }\n \n @@\n @@ -111,8 +111,8 @@\n +\t\t\tchar *graph_name = get_split_graph_filename(odb->path, line.buf);\n +\t\t\tstruct commit_graph *g = load_commit_graph_one(graph_name);\n \n --\t\tif (g && add_graph_to_chain(g, r->objects->commit_graph, oids, i))\n --\t\t\tr->objects->commit_graph = g;\n +-\t\tif (g && add_graph_to_chain(g, graph_chain, oids, i))\n +-\t\t\tgraph_chain = g;\n -\t\telse\n -\t\t\tvalid = 0;\n +\t\t\tfree(graph_name);\n @@ -120,8 +120,8 @@\n +\t\t\tif (g) {\n +\t\t\t\tg->obj_dir = odb->path;\n +\n -+\t\t\t\tif (add_graph_to_chain(g, r->objects->commit_graph, oids, i))\n -+\t\t\t\t\tr->objects->commit_graph = g;\n ++\t\t\t\tif (add_graph_to_chain(g, graph_chain, oids, i))\n ++\t\t\t\t\tgraph_chain = g;\n +\t\t\t\telse\n +\t\t\t\t\tvalid = 0;\n +\n 11: 72fc0a1f17 ! 11: 18d612be9e commit-graph: expire commit-graph files\n @@ -45,15 +45,6 @@\n diff --git a/commit-graph.c b/commit-graph.c\n --- a/commit-graph.c\n +++ b/commit-graph.c\n -@@\n - \tunsigned append:1,\n - \t\t report_progress:1,\n - \t\t split:1;\n -+\n -+\ttime_t expire_window;\n - };\n - \n - static void write_graph_chunk_fanout(struct hashfile *f,\n @@\n \tdeduplicate_commits(ctx);\n }\n @@ -81,7 +72,7 @@\n +\tDIR *dir;\n +\tstruct dirent *de;\n +\tsize_t dirnamelen;\n -+\ttime_t expire_time = time(NULL) - ctx->expire_window;\n ++\ttime_t expire_time = time(NULL);\n +\n +\tstrbuf_addstr(&path, ctx->obj_dir);\n +\tstrbuf_addstr(&path, \"/info/commit-graphs\");\n -: ---------- > 12: 4de4bfba64 commit-graph: create options for split files\n -: ---------- > 13: fe91ff5fca commit-graph: verify chains with --shallow mode\n -: ---------- > 14: ca41bf08d0 commit-graph: clean up chains after flattened write\n\n-- \ngitgitgadget\n"},{"id":"376577","messageId":"d0dc154a27c80a8113ff47eb60a6b6f98fbeec91.1559577826.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 02/14] commit-graph: prepare for commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-03T16:03:50Z","receivedAt":"2019-06-03T16:03:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTo prepare for a chain of commit-graph files, augment the\ncommit_graph struct to point to a base commit_graph. As we load\ncommits from the graph, we may actually want to read from a base\nfile according to the graph position.\n\nThe \"graph position\" of a commit is given by concatenating the\nlexicographic commit orders from each of the commit-graph files in\nthe chain. This means that we must distinguish two values:\n\n * lexicographic index : the position within the lexicographic\n order in a single commit-graph file.\n\n * graph position: the posiiton within the concatenated order\n of multiple commit-graph files\n\nGiven the lexicographic index of a commit in a graph, we can\ncompute the graph position by adding the number of commits in\nthe lower-level graphs. To find the lexicographic index of\na commit, we subtract the number of commits in lower-level graphs.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 74 ++++++++++++++++++++++++++++++++++++++++++++------\n commit-graph.h | 3 ++\n 2 files changed, 69 insertions(+), 8 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 7723156964..3afedcd7f5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -371,6 +371,25 @@ static int bsearch_graph(struct commit_graph *g, struct object_id *oid, uint32_t\n \t\t\t g->chunk_oid_lookup, g->hash_len, pos);\n }\n \n+static void load_oid_from_graph(struct commit_graph *g, int pos, struct object_id *oid)\n+{\n+\tuint32_t lex_index;\n+\n+\tif (!g)\n+\t\tBUG(\"NULL commit-graph\");\n+\n+\twhile (pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tif (pos >= g->num_commits + g->num_commits_in_base)\n+\t\tBUG(\"position %d is beyond the scope of this commit-graph (%d local + %d base commits)\",\n+\t\t pos, g->num_commits, g->num_commits_in_base);\n+\n+\tlex_index = pos - g->num_commits_in_base;\n+\n+\thashcpy(oid->hash, g->chunk_oid_lookup + g->hash_len * lex_index);\n+}\n+\n static struct commit_list **insert_parent_or_die(struct repository *r,\n \t\t\t\t\t\t struct commit_graph *g,\n \t\t\t\t\t\t uint64_t pos,\n@@ -379,10 +398,10 @@ static struct commit_list **insert_parent_or_die(struct repository *r,\n \tstruct commit *c;\n \tstruct object_id oid;\n \n-\tif (pos >= g->num_commits)\n+\tif (pos >= g->num_commits + g->num_commits_in_base)\n \t\tdie(\"invalid parent position %\"PRIu64, pos);\n \n-\thashcpy(oid.hash, g->chunk_oid_lookup + g->hash_len * pos);\n+\tload_oid_from_graph(g, pos, &oid);\n \tc = lookup_commit(r, &oid);\n \tif (!c)\n \t\tdie(_(\"could not find commit %s\"), oid_to_hex(&oid));\n@@ -392,7 +411,14 @@ static struct commit_list **insert_parent_or_die(struct repository *r,\n \n static void fill_commit_graph_info(struct commit *item, struct commit_graph *g, uint32_t pos)\n {\n-\tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n+\tconst unsigned char *commit_data;\n+\tuint32_t lex_index;\n+\n+\twhile (pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tlex_index = pos - g->num_commits_in_base;\n+\tcommit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * lex_index;\n \titem->graph_pos = pos;\n \titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n }\n@@ -405,10 +431,26 @@ static int fill_commit_in_graph(struct repository *r,\n \tuint32_t *parent_data_ptr;\n \tuint64_t date_low, date_high;\n \tstruct commit_list **pptr;\n-\tconst unsigned char *commit_data = g->chunk_commit_data + (g->hash_len + 16) * pos;\n+\tconst unsigned char *commit_data;\n+\tuint32_t lex_index;\n \n-\titem->object.parsed = 1;\n+\twhile (pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tif (pos >= g->num_commits + g->num_commits_in_base)\n+\t\tBUG(\"position %d is beyond the scope of this commit-graph (%d local + %d base commits)\",\n+\t\t pos, g->num_commits, g->num_commits_in_base);\n+\n+\t/*\n+\t * Store the \"full\" position, but then use the\n+\t * \"local\" position for the rest of the calculation.\n+\t */\n \titem->graph_pos = pos;\n+\tlex_index = pos - g->num_commits_in_base;\n+\n+\tcommit_data = g->chunk_commit_data + (g->hash_len + 16) * lex_index;\n+\n+\titem->object.parsed = 1;\n \n \titem->maybe_tree = NULL;\n \n@@ -452,7 +494,18 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \t\t*pos = item->graph_pos;\n \t\treturn 1;\n \t} else {\n-\t\treturn bsearch_graph(g, &(item->object.oid), pos);\n+\t\tstruct commit_graph *cur_g = g;\n+\t\tuint32_t lex_index;\n+\n+\t\twhile (cur_g && !bsearch_graph(cur_g, &(item->object.oid), &lex_index))\n+\t\t\tcur_g = cur_g->base_graph;\n+\n+\t\tif (cur_g) {\n+\t\t\t*pos = lex_index + cur_g->num_commits_in_base;\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\treturn 0;\n \t}\n }\n \n@@ -492,8 +545,13 @@ static struct tree *load_tree_for_commit(struct repository *r,\n \t\t\t\t\t struct commit *c)\n {\n \tstruct object_id oid;\n-\tconst unsigned char *commit_data = g->chunk_commit_data +\n-\t\t\t\t\t GRAPH_DATA_WIDTH * (c->graph_pos);\n+\tconst unsigned char *commit_data;\n+\n+\twhile (c->graph_pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tcommit_data = g->chunk_commit_data +\n+\t\t\tGRAPH_DATA_WIDTH * (c->graph_pos - g->num_commits_in_base);\n \n \thashcpy(oid.hash, commit_data);\n \tc->maybe_tree = lookup_tree(r, &oid);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 70f4caf0c7..f9fe32ebe3 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -48,6 +48,9 @@ struct commit_graph {\n \tuint32_t num_commits;\n \tstruct object_id oid;\n \n+\tuint32_t num_commits_in_base;\n+\tstruct commit_graph *base_graph;\n+\n \tconst uint32_t *chunk_oid_fanout;\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_commit_data;\n-- \ngitgitgadget\n\n"},{"id":"376578","messageId":"ca670536dfe08277e953b2ef5f017dc32c1581d5.1559577826.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 04/14] commit-graph: load commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-03T16:03:53Z","receivedAt":"2019-06-03T16:03:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nPrepare the logic for reading a chain of commit-graphs.\n\nFirst, look for a file at $OBJDIR/info/commit-graph. If it exists,\nthen use that file and stop.\n\nNext, look for the chain file at $OBJDIR/info/commit-graphs/commit-graph-chain.\nIf this file exists, then load the hash values as line-separated values in that\nfile and load $OBJDIR/info/commit-graphs/graph-{hash[i]}.graph for each hash[i]\nin that file. The file is given in order, so the first hash corresponds to the\n\"base\" file and the final hash corresponds to the \"tip\" file.\n\nThis implementation assumes that all of the graph-{hash}.graph files are in\nthe same object directory as the commit-graph-chain file. This will be updated\nin a future change. This change is purposefully simple so we can isolate the\ndifferent concerns.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 108 insertions(+), 6 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex e2f438f6a3..3ed930159e 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -45,6 +45,19 @@ char *get_commit_graph_filename(const char *obj_dir)\n \treturn xstrfmt(\"%s/info/commit-graph\", obj_dir);\n }\n \n+static char *get_split_graph_filename(const char *obj_dir,\n+\t\t\t\t const char *oid_hex)\n+{\n+\treturn xstrfmt(\"%s/info/commit-graphs/graph-%s.graph\",\n+\t\t obj_dir,\n+\t\t oid_hex);\n+}\n+\n+static char *get_chain_filename(const char *obj_dir)\n+{\n+\treturn xstrfmt(\"%s/info/commit-graphs/commit-graph-chain\", obj_dir);\n+}\n+\n static uint8_t oid_version(void)\n {\n \treturn 1;\n@@ -286,18 +299,107 @@ static struct commit_graph *load_commit_graph_one(const char *graph_file)\n \treturn load_commit_graph_one_fd_st(fd, &st);\n }\n \n+static struct commit_graph *load_commit_graph_v1(struct repository *r, const char *obj_dir)\n+{\n+\tchar *graph_name = get_commit_graph_filename(obj_dir);\n+\tstruct commit_graph *g = load_commit_graph_one(graph_name);\n+\tfree(graph_name);\n+\n+\treturn g;\n+}\n+\n+static int add_graph_to_chain(struct commit_graph *g,\n+\t\t\t struct commit_graph *chain,\n+\t\t\t struct object_id *oids,\n+\t\t\t int n)\n+{\n+\tstruct commit_graph *cur_g = chain;\n+\n+\twhile (n) {\n+\t\tn--;\n+\t\tcur_g = cur_g->base_graph;\n+\t}\n+\n+\tg->base_graph = chain;\n+\n+\tif (chain)\n+\t\tg->num_commits_in_base = chain->num_commits + chain->num_commits_in_base;\n+\n+\treturn 1;\n+}\n+\n+static struct commit_graph *load_commit_graph_chain(struct repository *r, const char *obj_dir)\n+{\n+\tstruct commit_graph *graph_chain = NULL;\n+\tstruct strbuf line = STRBUF_INIT;\n+\tstruct stat st;\n+\tstruct object_id *oids;\n+\tint i = 0, valid = 1;\n+\tchar *chain_name = get_chain_filename(obj_dir);\n+\tFILE *fp;\n+\n+\tif (stat(chain_name, &st)) {\n+\t\tfree(chain_name);\n+\t\treturn NULL;\n+\t}\n+\n+\tif (st.st_size <= the_hash_algo->hexsz) {\n+\t\tfree(chain_name);\n+\t\treturn NULL;\n+\t}\n+\n+\tfp = fopen(chain_name, \"r\");\n+\tfree(chain_name);\n+\n+\tif (!fp)\n+\t\treturn NULL;\n+\n+\toids = xcalloc(st.st_size / (the_hash_algo->hexsz + 1), sizeof(struct object_id));\n+\n+\twhile (strbuf_getline_lf(&line, fp) != EOF && valid) {\n+\t\tchar *graph_name;\n+\t\tstruct commit_graph *g;\n+\n+\t\tif (get_oid_hex(line.buf, &oids[i])) {\n+\t\t\twarning(_(\"invalid commit-graph chain: line '%s' not a hash\"),\n+\t\t\t\tline.buf);\n+\t\t\tvalid = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tgraph_name = get_split_graph_filename(obj_dir, line.buf);\n+\t\tg = load_commit_graph_one(graph_name);\n+\t\tfree(graph_name);\n+\n+\t\tif (g && add_graph_to_chain(g, graph_chain, oids, i))\n+\t\t\tgraph_chain = g;\n+\t\telse\n+\t\t\tvalid = 0;\n+\t}\n+\n+\tfree(oids);\n+\tfclose(fp);\n+\n+\treturn graph_chain;\n+}\n+\n+static struct commit_graph *read_commit_graph_one(struct repository *r, const char *obj_dir)\n+{\n+\tstruct commit_graph *g = load_commit_graph_v1(r, obj_dir);\n+\n+\tif (!g)\n+\t\tg = load_commit_graph_chain(r, obj_dir);\n+\n+\treturn g;\n+}\n+\n static void prepare_commit_graph_one(struct repository *r, const char *obj_dir)\n {\n-\tchar *graph_name;\n \n \tif (r->objects->commit_graph)\n \t\treturn;\n \n-\tgraph_name = get_commit_graph_filename(obj_dir);\n-\tr->objects->commit_graph =\n-\t\tload_commit_graph_one(graph_name);\n-\n-\tFREE_AND_NULL(graph_name);\n+\tr->objects->commit_graph = read_commit_graph_one(r, obj_dir);\n }\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"376579","messageId":"f35b04224abf188c265ed0395cd3dfcf6eb6ac5b.1559577826.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 03/14] commit-graph: rename commit_compare to oid_compare","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-03T16:03:52Z","receivedAt":"2019-06-03T16:03:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe helper function commit_compare() actually compares object_id\nstructs, not commits. A future change to commit-graph.c will need\nto sort commit structs, so rename this function in advance.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 3afedcd7f5..e2f438f6a3 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -761,7 +761,7 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n \t}\n }\n \n-static int commit_compare(const void *_a, const void *_b)\n+static int oid_compare(const void *_a, const void *_b)\n {\n \tconst struct object_id *a = (const struct object_id *)_a;\n \tconst struct object_id *b = (const struct object_id *)_b;\n@@ -1030,7 +1030,7 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n \t\t\t_(\"Counting distinct commits in commit graph\"),\n \t\t\tctx->oids.nr);\n \tdisplay_progress(ctx->progress, 0); /* TODO: Measure QSORT() progress */\n-\tQSORT(ctx->oids.list, ctx->oids.nr, commit_compare);\n+\tQSORT(ctx->oids.list, ctx->oids.nr, oid_compare);\n \n \tfor (i = 1; i < ctx->oids.nr; i++) {\n \t\tdisplay_progress(ctx->progress, i + 1);\n-- \ngitgitgadget\n\n"},{"id":"376580","messageId":"4f4ccc806224e4e8fbea796a20c9a4f774b9a4b0.1559577826.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 08/14] commit-graph: add --split option to builtin","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-03T16:03:56Z","receivedAt":"2019-06-03T16:04:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd a new \"--split\" option to the 'git commit-graph write' subcommand. This\noption allows the optional behavior of writing a commit-graph chain.\n\nThe current behavior will add a tip commit-graph containing any commits that\nare not in the existing commit-graph or commit-graph chain. Later changes\nwill allow merging the chain and expiring out-dated files.\n\nAdd a new test script (t5323-split-commit-graph.sh) that demonstrates this\nbehavior.\n\nSigned-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 10 ++-\n t/t5323-split-commit-graph.sh | 122 ++++++++++++++++++++++++++++++++++\n 2 files changed, 129 insertions(+), 3 deletions(-)\n create mode 100755 t/t5323-split-commit-graph.sh\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 828b1a713f..c2c07d3917 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -10,7 +10,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph verify [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -25,7 +25,7 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -35,9 +35,9 @@ static struct opts_commit_graph {\n \tint stdin_packs;\n \tint stdin_commits;\n \tint append;\n+\tint split;\n } opts;\n \n-\n static int graph_verify(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n@@ -156,6 +156,8 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"start walk at commits listed by stdin\")),\n \t\tOPT_BOOL(0, \"append\", &opts.append,\n \t\t\tN_(\"include all commits already in the commit-graph file\")),\n+\t\tOPT_BOOL(0, \"split\", &opts.split,\n+\t\t\tN_(\"allow writing an incremental commit-graph file\")),\n \t\tOPT_END(),\n \t};\n \n@@ -169,6 +171,8 @@ static int graph_write(int argc, const char **argv)\n \t\topts.obj_dir = get_object_directory();\n \tif (opts.append)\n \t\tflags |= COMMIT_GRAPH_APPEND;\n+\tif (opts.split)\n+\t\tflags |= COMMIT_GRAPH_SPLIT;\n \n \tread_replace_refs = 0;\n \ndiff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\nnew file mode 100755\nindex 0000000000..ccd24bd22b\n--- /dev/null\n+++ b/t/t5323-split-commit-graph.sh\n@@ -0,0 +1,122 @@\n+#!/bin/sh\n+\n+test_description='split commit graph'\n+. ./test-lib.sh\n+\n+GIT_TEST_COMMIT_GRAPH=0\n+\n+test_expect_success 'setup repo' '\n+\tgit init &&\n+\tgit config core.commitGraph true &&\n+\tinfodir=\".git/objects/info\" &&\n+\tgraphdir=\"$infodir/commit-graphs\" &&\n+\ttest_oid_init\n+'\n+\n+graph_read_expect() {\n+\tNUM_BASE=0\n+\tif test ! -z $2\n+\tthen\n+\t\tNUM_BASE=$2\n+\tfi\n+\tcat >expect <<- EOF\n+\theader: 43475048 1 1 3 $NUM_BASE\n+\tnum_commits: $1\n+\tchunks: oid_fanout oid_lookup commit_metadata\n+\tEOF\n+\tgit commit-graph read >output &&\n+\ttest_cmp expect output\n+}\n+\n+test_expect_success 'create commits and write commit-graph' '\n+\tfor i in $(test_seq 3)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i || return 1\n+\tdone &&\n+\tgit commit-graph write --reachable &&\n+\ttest_path_is_file $infodir/commit-graph &&\n+\tgraph_read_expect 3\n+'\n+\n+graph_git_two_modes() {\n+\tgit -c core.commitGraph=true $1 >output\n+\tgit -c core.commitGraph=false $1 >expect\n+\ttest_cmp expect output\n+}\n+\n+graph_git_behavior() {\n+\tMSG=$1\n+\tBRANCH=$2\n+\tCOMPARE=$3\n+\ttest_expect_success \"check normal git operations: $MSG\" '\n+\t\tgraph_git_two_modes \"log --oneline $BRANCH\" &&\n+\t\tgraph_git_two_modes \"log --topo-order $BRANCH\" &&\n+\t\tgraph_git_two_modes \"log --graph $COMPARE..$BRANCH\" &&\n+\t\tgraph_git_two_modes \"branch -vv\" &&\n+\t\tgraph_git_two_modes \"merge-base -a $BRANCH $COMPARE\"\n+\t'\n+}\n+\n+graph_git_behavior 'graph exists' commits/3 commits/1\n+\n+verify_chain_files_exist() {\n+\tfor hash in $(cat $1/commit-graph-chain)\n+\tdo\n+\t\ttest_path_is_file $1/graph-$hash.graph || return 1\n+\tdone\n+}\n+\n+test_expect_success 'add more commits, and write a new base graph' '\n+\tgit reset --hard commits/1 &&\n+\tfor i in $(test_seq 4 5)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i || return 1\n+\tdone &&\n+\tgit reset --hard commits/2 &&\n+\tfor i in $(test_seq 6 10)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i || return 1\n+\tdone &&\n+\tgit reset --hard commits/2 &&\n+\tgit merge commits/4 &&\n+\tgit branch merge/1 &&\n+\tgit reset --hard commits/4 &&\n+\tgit merge commits/6 &&\n+\tgit branch merge/2 &&\n+\tgit commit-graph write --reachable &&\n+\tgraph_read_expect 12\n+'\n+\n+test_expect_success 'add three more commits, write a tip graph' '\n+\tgit reset --hard commits/3 &&\n+\tgit merge merge/1 &&\n+\tgit merge commits/5 &&\n+\tgit merge merge/2 &&\n+\tgit branch merge/3 &&\n+\tgit commit-graph write --reachable --split &&\n+\ttest_path_is_missing $infodir/commit-graph &&\n+\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\tls $graphdir/graph-*.graph >graph-files &&\n+\ttest_line_count = 2 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n+graph_git_behavior 'split commit-graph: merge 3 vs 2' merge/3 merge/2\n+\n+test_expect_success 'add one commit, write a tip graph' '\n+\ttest_commit 11 &&\n+\tgit branch commits/11 &&\n+\tgit commit-graph write --reachable --split &&\n+\ttest_path_is_missing $infodir/commit-graph &&\n+\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\tls $graphdir/graph-*.graph >graph-files &&\n+\ttest_line_count = 3 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n+graph_git_behavior 'three-layer commit-graph: commit 11 vs 6' commits/11 commits/6\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"376581","messageId":"e65f9e841d577c13d26c1737ebb8a2cf65713be9.1559577826.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 06/14] commit-graph: rearrange chunk count logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-03T16:03:54Z","receivedAt":"2019-06-03T16:04:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe number of chunks in a commit-graph file can change depending on\nwhether we need the Extra Edges Chunk. We are going to add more optional\nchunks, and it will be helpful to rearrange this logic around the chunk\ncount before doing so.\n\nSpecifically, we need to finalize the number of chunks before writing\nthe commit-graph header. Further, we also need to fill out the chunk\nlookup table dynamically and using \"num_chunks\" as we add optional\nchunks is useful for adding optional chunks in the future.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 35 +++++++++++++++++++++--------------\n 1 file changed, 21 insertions(+), 14 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 909c841db5..80df6d6d9d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1206,7 +1206,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tuint64_t chunk_offsets[5];\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n-\tint num_chunks = ctx->num_extra_edges ? 4 : 3;\n+\tint num_chunks = 3;\n \n \tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n \tif (safe_create_leading_directories(ctx->graph_name)) {\n@@ -1219,27 +1219,34 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \n-\thashwrite_be32(f, GRAPH_SIGNATURE);\n-\n-\thashwrite_u8(f, GRAPH_VERSION);\n-\thashwrite_u8(f, oid_version());\n-\thashwrite_u8(f, num_chunks);\n-\thashwrite_u8(f, 0); /* unused padding byte */\n-\n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n \tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n-\tif (ctx->num_extra_edges)\n-\t\tchunk_ids[3] = GRAPH_CHUNKID_EXTRAEDGES;\n-\telse\n-\t\tchunk_ids[3] = 0;\n-\tchunk_ids[4] = 0;\n+\tif (ctx->num_extra_edges) {\n+\t\tchunk_ids[num_chunks] = GRAPH_CHUNKID_EXTRAEDGES;\n+\t\tnum_chunks++;\n+\t}\n+\n+\tchunk_ids[num_chunks] = 0;\n \n \tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n \tchunk_offsets[2] = chunk_offsets[1] + hashsz * ctx->commits.nr;\n \tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * ctx->commits.nr;\n-\tchunk_offsets[4] = chunk_offsets[3] + 4 * ctx->num_extra_edges;\n+\n+\tnum_chunks = 3;\n+\tif (ctx->num_extra_edges) {\n+\t\tchunk_offsets[num_chunks + 1] = chunk_offsets[num_chunks] +\n+\t\t\t\t\t\t4 * ctx->num_extra_edges;\n+\t\tnum_chunks++;\n+\t}\n+\n+\thashwrite_be32(f, GRAPH_SIGNATURE);\n+\n+\thashwrite_u8(f, GRAPH_VERSION);\n+\thashwrite_u8(f, oid_version());\n+\thashwrite_u8(f, num_chunks);\n+\thashwrite_u8(f, 0);\n \n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tuint32_t chunk_write[3];\n-- \ngitgitgadget\n\n"},{"id":"376582","messageId":"df44cbc1bf6e18c3682652c2de69de7a9caf5fc2.1559577826.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 05/14] commit-graph: add base graphs chunk","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-03T16:03:54Z","receivedAt":"2019-06-03T16:04:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTo quickly verify a commit-graph chain is valid on load, we will\nread from the new \"Base Graphs Chunk\" of each file in the chain.\nThis will prevent accidentally loading incorrect data from manually\nediting the commit-graph-chain file or renaming graph-{hash}.graph\nfiles.\n\nThe commit_graph struct already had an object_id struct \"oid\", but\nit was never initialized or used. Add a line to read the hash from\nthe end of the commit-graph file and into the oid member.\n\nSigned-off-by: Derrick Stolee \n---\n .../technical/commit-graph-format.txt | 11 ++++++++--\n commit-graph.c | 22 +++++++++++++++++++\n commit-graph.h | 1 +\n 3 files changed, 32 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex 16452a0504..a4f17441ae 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -44,8 +44,9 @@ HEADER:\n \n 1-byte number (C) of \"chunks\"\n \n- 1-byte (reserved for later use)\n- Current clients should ignore this value.\n+ 1-byte number (B) of base commit-graphs\n+ We infer the length (H*B) of the Base Graphs chunk\n+ from this value.\n \n CHUNK LOOKUP:\n \n@@ -92,6 +93,12 @@ CHUNK DATA:\n positions for the parents until reaching a value with the most-significant\n bit on. The other bits correspond to the position of the last parent.\n \n+ Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]\n+ This list of H-byte hashes describe a set of B commit-graph files that\n+ form a commit-graph chain. The graph position for the ith commit in this\n+ file's OID Lookup chunk is equal to i plus the number of commits in all\n+ base graphs. If B is non-zero, this chunk must exist.\n+\n TRAILER:\n \n \tH-byte HASH-checksum of all of the above.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 3ed930159e..909c841db5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -22,6 +22,7 @@\n #define GRAPH_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define GRAPH_CHUNKID_DATA 0x43444154 /* \"CDAT\" */\n #define GRAPH_CHUNKID_EXTRAEDGES 0x45444745 /* \"EDGE\" */\n+#define GRAPH_CHUNKID_BASE 0x42415345 /* \"BASE\" */\n \n #define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n \n@@ -262,6 +263,12 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t\t\telse\n \t\t\t\tgraph->chunk_extra_edges = data + chunk_offset;\n \t\t\tbreak;\n+\n+\t\tcase GRAPH_CHUNKID_BASE:\n+\t\t\tif (graph->chunk_base_graphs)\n+\t\t\t\tchunk_repeated = 1;\n+\t\t\telse\n+\t\t\t\tgraph->chunk_base_graphs = data + chunk_offset;\n \t\t}\n \n \t\tif (chunk_repeated) {\n@@ -280,6 +287,8 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t\tlast_chunk_offset = chunk_offset;\n \t}\n \n+\thashcpy(graph->oid.hash, graph->data + graph->data_len - graph->hash_len);\n+\n \tif (verify_commit_graph_lite(graph))\n \t\treturn NULL;\n \n@@ -315,8 +324,21 @@ static int add_graph_to_chain(struct commit_graph *g,\n {\n \tstruct commit_graph *cur_g = chain;\n \n+\tif (n && !g->chunk_base_graphs) {\n+\t\twarning(_(\"commit-graph has no base graphs chunk\"));\n+\t\treturn 0;\n+\t}\n+\n \twhile (n) {\n \t\tn--;\n+\n+\t\tif (!oideq(&oids[n], &cur_g->oid) ||\n+\t\t !hasheq(oids[n].hash, g->chunk_base_graphs + g->hash_len * n)) {\n+\t\t\twarning(_(\"commit-graph chain does not match\"));\n+\t\t\treturn 0;\n+\t\t}\n+\n+\n \t\tcur_g = cur_g->base_graph;\n \t}\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex f9fe32ebe3..80f4917ddb 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -55,6 +55,7 @@ struct commit_graph {\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_commit_data;\n \tconst unsigned char *chunk_extra_edges;\n+\tconst unsigned char *chunk_base_graphs;\n };\n \n struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st);\n-- \ngitgitgadget\n\n"},{"id":"376583","messageId":"fe0aa343cdb9c1fd5c087b1e8da70247237de01f.1559577826.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 07/14] commit-graph: write commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-03T16:03:55Z","receivedAt":"2019-06-03T16:04:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nExtend write_commit_graph() to write a commit-graph chain when given the\nCOMMIT_GRAPH_SPLIT flag.\n\nThis implementation is purposefully simplistic in how it creates a new\nchain. The commits not already in the chain are added to a new tip\ncommit-graph file.\n\nMuch of the logic around writing a graph-{hash}.graph file and updating\nthe commit-graph-chain file is the same as the commit-graph file case.\nHowever, there are several places where we need to do some extra logic\nin the split case.\n\nTrack the list of graph filenames before and after the planned write.\nThis will be more important when we start merging graph files, but it\nalso allows us to upgrade our commit-graph file to the appropriate\ngraph-{hash}.graph file when we upgrade to a chain of commit-graphs.\n\nNote that we use the eighth byte of the commit-graph header to store the\nnumber of base graph files. This determines the length of the base\ngraphs chunk.\n\nA subtle change of behavior with the new logic is that we do not write a\ncommit-graph if we our commit list is empty. This extends to the typical\ncase, which is reflected in t5318-commit-graph.sh.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 286 ++++++++++++++++++++++++++++++++++++++--\n commit-graph.h | 2 +\n t/t5318-commit-graph.sh | 2 +-\n 3 files changed, 278 insertions(+), 12 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 80df6d6d9d..7d3e001479 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -300,12 +300,18 @@ static struct commit_graph *load_commit_graph_one(const char *graph_file)\n \n \tstruct stat st;\n \tint fd;\n+\tstruct commit_graph *g;\n \tint open_ok = open_commit_graph(graph_file, &fd, &st);\n \n \tif (!open_ok)\n \t\treturn NULL;\n \n-\treturn load_commit_graph_one_fd_st(fd, &st);\n+\tg = load_commit_graph_one_fd_st(fd, &st);\n+\n+\tif (g)\n+\t\tg->filename = xstrdup(graph_file);\n+\n+\treturn g;\n }\n \n static struct commit_graph *load_commit_graph_v1(struct repository *r, const char *obj_dir)\n@@ -723,8 +729,19 @@ struct write_commit_graph_context {\n \tstruct progress *progress;\n \tint progress_done;\n \tuint64_t progress_cnt;\n+\n+\tchar *base_graph_name;\n+\tint num_commit_graphs_before;\n+\tint num_commit_graphs_after;\n+\tchar **commit_graph_filenames_before;\n+\tchar **commit_graph_filenames_after;\n+\tchar **commit_graph_hash_after;\n+\tuint32_t new_num_commits_in_base;\n+\tstruct commit_graph *new_base_graph;\n+\n \tunsigned append:1,\n-\t\t report_progress:1;\n+\t\t report_progress:1,\n+\t\t split:1;\n };\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n@@ -794,6 +811,16 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t commit_to_sha1);\n \n+\t\t\tif (edge_value >= 0)\n+\t\t\t\tedge_value += ctx->new_num_commits_in_base;\n+\t\t\telse {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (find_commit_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n+\t\t\t\t\tedge_value = pos;\n+\t\t\t}\n+\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n \t\t\t\t oid_to_hex(&parent->item->object.oid),\n@@ -814,6 +841,17 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\t\t\t ctx->commits.list,\n \t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t commit_to_sha1);\n+\n+\t\t\tif (edge_value >= 0)\n+\t\t\t\tedge_value += ctx->new_num_commits_in_base;\n+\t\t\telse {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (find_commit_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n+\t\t\t\t\tedge_value = pos;\n+\t\t\t}\n+\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n \t\t\t\t oid_to_hex(&parent->item->object.oid),\n@@ -871,6 +909,16 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n \t\t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t\t commit_to_sha1);\n \n+\t\t\tif (edge_value >= 0)\n+\t\t\t\tedge_value += ctx->new_num_commits_in_base;\n+\t\t\telse {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (find_commit_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n+\t\t\t\t\tedge_value = pos;\n+\t\t\t}\n+\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n \t\t\t\t oid_to_hex(&parent->item->object.oid),\n@@ -962,7 +1010,13 @@ static void close_reachable(struct write_commit_graph_context *ctx)\n \t\tdisplay_progress(ctx->progress, i + 1);\n \t\tcommit = lookup_commit(ctx->r, &ctx->oids.list[i]);\n \n-\t\tif (commit && !parse_commit_no_graph(commit))\n+\t\tif (!commit)\n+\t\t\tcontinue;\n+\t\tif (ctx->split) {\n+\t\t\tif (!parse_commit(commit) &&\n+\t\t\t commit->graph_pos == COMMIT_NOT_FROM_GRAPH)\n+\t\t\t\tadd_missing_parents(ctx, commit);\n+\t\t} else if (!parse_commit_no_graph(commit))\n \t\t\tadd_missing_parents(ctx, commit);\n \t}\n \tstop_progress(&ctx->progress);\n@@ -1158,8 +1212,16 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n \n \tfor (i = 1; i < ctx->oids.nr; i++) {\n \t\tdisplay_progress(ctx->progress, i + 1);\n-\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n+\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i])) {\n+\t\t\tif (ctx->split) {\n+\t\t\t\tstruct commit *c = lookup_commit(ctx->r, &ctx->oids.list[i]);\n+\n+\t\t\t\tif (!c || c->graph_pos != COMMIT_NOT_FROM_GRAPH)\n+\t\t\t\t\tcontinue;\n+\t\t\t}\n+\n \t\t\tcount_distinct++;\n+\t\t}\n \t}\n \tstop_progress(&ctx->progress);\n \n@@ -1182,7 +1244,13 @@ static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n \t\tif (i > 0 && oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n \t\t\tcontinue;\n \n+\t\tALLOC_GROW(ctx->commits.list, ctx->commits.nr + 1, ctx->commits.alloc);\n \t\tctx->commits.list[ctx->commits.nr] = lookup_commit(ctx->r, &ctx->oids.list[i]);\n+\n+\t\tif (ctx->split &&\n+\t\t ctx->commits.list[ctx->commits.nr]->graph_pos != COMMIT_NOT_FROM_GRAPH)\n+\t\t\tcontinue;\n+\n \t\tparse_commit_no_graph(ctx->commits.list[ctx->commits.nr]);\n \n \t\tfor (parent = ctx->commits.list[ctx->commits.nr]->parents;\n@@ -1197,18 +1265,86 @@ static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n+static int write_graph_chunk_base_1(struct hashfile *f,\n+\t\t\t\t struct commit_graph *g)\n+{\n+\tint num = 0;\n+\n+\tif (!g)\n+\t\treturn 0;\n+\n+\tnum = write_graph_chunk_base_1(f, g->base_graph);\n+\thashwrite(f, g->oid.hash, the_hash_algo->rawsz);\n+\treturn num + 1;\n+}\n+\n+static int write_graph_chunk_base(struct hashfile *f,\n+\t\t\t\t struct write_commit_graph_context *ctx)\n+{\n+\tint num = write_graph_chunk_base_1(f, ctx->new_base_graph);\n+\n+\tif (num != ctx->num_commit_graphs_after - 1) {\n+\t\terror(_(\"failed to write correct number of base graph ids\"));\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void init_commit_graph_chain(struct write_commit_graph_context *ctx)\n+{\n+\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\tuint32_t i;\n+\n+\tctx->new_base_graph = g;\n+\tctx->base_graph_name = xstrdup(g->filename);\n+\tctx->new_num_commits_in_base = g->num_commits + g->num_commits_in_base;\n+\n+\tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n+\n+\tALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);\n+\tALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);\n+\n+\tfor (i = 0; i < ctx->num_commit_graphs_before - 1; i++)\n+\t\tctx->commit_graph_filenames_after[i] = xstrdup(ctx->commit_graph_filenames_before[i]);\n+\n+\tif (ctx->num_commit_graphs_before)\n+\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_before - 1] =\n+\t\t\tget_split_graph_filename(ctx->obj_dir, oid_to_hex(&g->oid));\n+\n+\ti = ctx->num_commit_graphs_before - 1;\n+\n+\twhile (g) {\n+\t\tctx->commit_graph_hash_after[i] = xstrdup(oid_to_hex(&g->oid));\n+\t\ti--;\n+\t\tg = g->base_graph;\n+\t}\n+}\n+\n static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n+\tint fd;\n \tstruct hashfile *f;\n \tstruct lock_file lk = LOCK_INIT;\n-\tuint32_t chunk_ids[5];\n-\tuint64_t chunk_offsets[5];\n+\tuint32_t chunk_ids[6];\n+\tuint64_t chunk_offsets[6];\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tint num_chunks = 3;\n+\tstruct object_id file_hash;\n+\n+\tif (ctx->split) {\n+\t\tstruct strbuf tmp_file = STRBUF_INIT;\n+\n+\t\tstrbuf_addf(&tmp_file,\n+\t\t\t \"%s/info/commit-graphs/tmp_graph_XXXXXX\",\n+\t\t\t ctx->obj_dir);\n+\t\tctx->graph_name = strbuf_detach(&tmp_file, NULL);\n+\t} else {\n+\t\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n+\t}\n \n-\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n \tif (safe_create_leading_directories(ctx->graph_name)) {\n \t\tUNLEAK(ctx->graph_name);\n \t\terror(_(\"unable to create leading directories of %s\"),\n@@ -1216,8 +1352,23 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\treturn errno;\n \t}\n \n-\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n-\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n+\tif (ctx->split) {\n+\t\tchar *lock_name = get_chain_filename(ctx->obj_dir);\n+\n+\t\thold_lock_file_for_update(&lk, lock_name, LOCK_DIE_ON_ERROR);\n+\n+\t\tfd = git_mkstemp_mode(ctx->graph_name, 0444);\n+\t\tif (fd < 0) {\n+\t\t\terror(_(\"unable to create '%s'\"), ctx->graph_name);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tf = hashfd(fd, ctx->graph_name);\n+\t} else {\n+\t\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n+\t\tfd = lk.tempfile->fd;\n+\t\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n+\t}\n \n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n@@ -1226,6 +1377,10 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\tchunk_ids[num_chunks] = GRAPH_CHUNKID_EXTRAEDGES;\n \t\tnum_chunks++;\n \t}\n+\tif (ctx->num_commit_graphs_after > 1) {\n+\t\tchunk_ids[num_chunks] = GRAPH_CHUNKID_BASE;\n+\t\tnum_chunks++;\n+\t}\n \n \tchunk_ids[num_chunks] = 0;\n \n@@ -1240,13 +1395,18 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\t\t\t\t4 * ctx->num_extra_edges;\n \t\tnum_chunks++;\n \t}\n+\tif (ctx->num_commit_graphs_after > 1) {\n+\t\tchunk_offsets[num_chunks + 1] = chunk_offsets[num_chunks] +\n+\t\t\t\t\t\thashsz * (ctx->num_commit_graphs_after - 1);\n+\t\tnum_chunks++;\n+\t}\n \n \thashwrite_be32(f, GRAPH_SIGNATURE);\n \n \thashwrite_u8(f, GRAPH_VERSION);\n \thashwrite_u8(f, oid_version());\n \thashwrite_u8(f, num_chunks);\n-\thashwrite_u8(f, 0);\n+\thashwrite_u8(f, ctx->num_commit_graphs_after - 1);\n \n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tuint32_t chunk_write[3];\n@@ -1272,11 +1432,67 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \twrite_graph_chunk_data(f, hashsz, ctx);\n \tif (ctx->num_extra_edges)\n \t\twrite_graph_chunk_extra_edges(f, ctx);\n+\tif (ctx->num_commit_graphs_after > 1 &&\n+\t write_graph_chunk_base(f, ctx)) {\n+\t\treturn -1;\n+\t}\n \tstop_progress(&ctx->progress);\n \tstrbuf_release(&progress_title);\n \n+\tif (ctx->split && ctx->base_graph_name && ctx->num_commit_graphs_after > 1) {\n+\t\tchar *new_base_hash = xstrdup(oid_to_hex(&ctx->new_base_graph->oid));\n+\t\tchar *new_base_name = get_split_graph_filename(ctx->obj_dir, new_base_hash);\n+\n+\t\tfree(ctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2]);\n+\t\tfree(ctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 2]);\n+\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2] = new_base_name;\n+\t\tctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 2] = new_base_hash;\n+\t}\n+\n \tclose_commit_graph(ctx->r);\n-\tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n+\tfinalize_hashfile(f, file_hash.hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n+\n+\tif (ctx->split) {\n+\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n+\t\tchar *final_graph_name;\n+\t\tint result;\n+\n+\t\tclose(fd);\n+\n+\t\tif (!chainf) {\n+\t\t\terror(_(\"unable to open commit-graph chain file\"));\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (ctx->base_graph_name) {\n+\t\t\tresult = rename(ctx->base_graph_name,\n+\t\t\t\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2]);\n+\n+\t\t\tif (result) {\n+\t\t\t\terror(_(\"failed to rename base commit-graph file\"));\n+\t\t\t\treturn -1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tchar *graph_name = get_commit_graph_filename(ctx->obj_dir);\n+\t\t\tunlink(graph_name);\n+\t\t}\n+\n+\t\tctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 1] = xstrdup(oid_to_hex(&file_hash));\n+\t\tfinal_graph_name = get_split_graph_filename(ctx->obj_dir,\n+\t\t\t\t\tctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 1]);\n+\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 1] = final_graph_name;\n+\n+\t\tresult = rename(ctx->graph_name, final_graph_name);\n+\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++)\n+\t\t\tfprintf(lk.tempfile->fp, \"%s\\n\", ctx->commit_graph_hash_after[i]);\n+\n+\t\tif (result) {\n+\t\t\terror(_(\"failed to rename temporary commit-graph file\"));\n+\t\t\treturn -1;\n+\t\t}\n+\t}\n+\n \tcommit_lock_file(&lk);\n \n \treturn 0;\n@@ -1299,6 +1515,30 @@ int write_commit_graph(const char *obj_dir,\n \tctx->obj_dir = obj_dir;\n \tctx->append = flags & COMMIT_GRAPH_APPEND ? 1 : 0;\n \tctx->report_progress = flags & COMMIT_GRAPH_PROGRESS ? 1 : 0;\n+\tctx->split = flags & COMMIT_GRAPH_SPLIT ? 1 : 0;\n+\n+\tif (ctx->split) {\n+\t\tstruct commit_graph *g;\n+\t\tprepare_commit_graph(ctx->r);\n+\n+\t\tg = ctx->r->objects->commit_graph;\n+\n+\t\twhile (g) {\n+\t\t\tctx->num_commit_graphs_before++;\n+\t\t\tg = g->base_graph;\n+\t\t}\n+\n+\t\tif (ctx->num_commit_graphs_before) {\n+\t\t\tALLOC_ARRAY(ctx->commit_graph_filenames_before, ctx->num_commit_graphs_before);\n+\t\t\ti = ctx->num_commit_graphs_before;\n+\t\t\tg = ctx->r->objects->commit_graph;\n+\n+\t\t\twhile (g) {\n+\t\t\t\tctx->commit_graph_filenames_before[--i] = xstrdup(g->filename);\n+\t\t\t\tg = g->base_graph;\n+\t\t\t}\n+\t\t}\n+\t}\n \n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n@@ -1353,6 +1593,14 @@ int write_commit_graph(const char *obj_dir,\n \t\tgoto cleanup;\n \t}\n \n+\tif (!ctx->commits.nr)\n+\t\tgoto cleanup;\n+\n+\tif (ctx->split)\n+\t\tinit_commit_graph_chain(ctx);\n+\telse\n+\t\tctx->num_commit_graphs_after = 1;\n+\n \tcompute_generation_numbers(ctx);\n \n \tres = write_commit_graph_file(ctx);\n@@ -1361,6 +1609,21 @@ int write_commit_graph(const char *obj_dir,\n \tfree(ctx->graph_name);\n \tfree(ctx->commits.list);\n \tfree(ctx->oids.list);\n+\n+\tif (ctx->commit_graph_filenames_after) {\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++) {\n+\t\t\tfree(ctx->commit_graph_filenames_after[i]);\n+\t\t\tfree(ctx->commit_graph_hash_after[i]);\n+\t\t}\n+\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_before; i++)\n+\t\t\tfree(ctx->commit_graph_filenames_before[i]);\n+\n+\t\tfree(ctx->commit_graph_filenames_after);\n+\t\tfree(ctx->commit_graph_filenames_before);\n+\t\tfree(ctx->commit_graph_hash_after);\n+\t}\n+\n \tfree(ctx);\n \n \treturn res;\n@@ -1548,5 +1811,6 @@ void free_commit_graph(struct commit_graph *g)\n \t\tg->data = NULL;\n \t\tclose(g->graph_fd);\n \t}\n+\tfree(g->filename);\n \tfree(g);\n }\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 80f4917ddb..5c48c4f66a 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -47,6 +47,7 @@ struct commit_graph {\n \tunsigned char num_chunks;\n \tuint32_t num_commits;\n \tstruct object_id oid;\n+\tchar *filename;\n \n \tuint32_t num_commits_in_base;\n \tstruct commit_graph *base_graph;\n@@ -71,6 +72,7 @@ int generation_numbers_enabled(struct repository *r);\n \n #define COMMIT_GRAPH_APPEND (1 << 0)\n #define COMMIT_GRAPH_PROGRESS (1 << 1)\n+#define COMMIT_GRAPH_SPLIT (1 << 2)\n \n int write_commit_graph_reachable(const char *obj_dir, unsigned int flags);\n int write_commit_graph(const char *obj_dir,\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 3b6fd0d728..063f906b3e 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -20,7 +20,7 @@ test_expect_success 'verify graph with no graph file' '\n test_expect_success 'write graph with no packs' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write --object-dir . &&\n-\ttest_path_is_file info/commit-graph\n+\ttest_path_is_missing info/commit-graph\n '\n \n test_expect_success 'close with correct error on bad input' '\n-- \ngitgitgadget\n\n"},{"id":"376584","messageId":"5cfd653d24edbb5320ecde53271fa957a773a729.1559577826.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 10/14] commit-graph: allow cross-alternate chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-03T16:03:58Z","receivedAt":"2019-06-03T16:04:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an environment like a fork network, it is helpful to have a\ncommit-graph chain that spans both the base repo and the fork repo. The\nfork is usually a small set of data on top of the large repo, but\nsometimes the fork is much larger. For example, git-for-windows/git has\nalmost double the number of commits as git/git because it rebases its\ncommits on every major version update.\n\nTo allow cross-alternate commit-graph chains, we need a few pieces:\n\n1. When looking for a graph-{hash}.graph file, check all alternates.\n\n2. When merging commit-graph chains, do not merge across alternates.\n\n3. When writing a new commit-graph chain based on a commit-graph file\n in another object directory, do not allow success if the base file\n has of the name \"commit-graph\" instead of\n \"commit-graphs/graoh-{hash}.graph\".\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 40 +++++++++++++++++++++\n commit-graph.c | 46 ++++++++++++++++++------\n commit-graph.h | 1 +\n t/t5323-split-commit-graph.sh | 37 +++++++++++++++++++\n 4 files changed, 114 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex d9c6253b0a..473032e476 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -266,6 +266,42 @@ The merge strategy values (2 for the size multiple, 64,000 for the maximum\n number of commits) could be extracted into config settings for full\n flexibility.\n \n+## Chains across multiple object directories\n+\n+In a repo with alternates, we look for the `commit-graph-chain` file starting\n+in the local object directory and then in each alternate. The first file that\n+exists defines our chain. As we look for the `graph-{hash}` files for\n+each `{hash}` in the chain file, we follow the same pattern for the host\n+directories.\n+\n+This allows commit-graphs to be split across multiple forks in a fork network.\n+The typical case is a large \"base\" repo with many smaller forks.\n+\n+As the base repo advances, it will likely update and merge its commit-graph\n+chain more frequently than the forks. If a fork updates their commit-graph after\n+the base repo, then it should \"reparent\" the commit-graph chain onto the new\n+chain in the base repo. When reading each `graph-{hash}` file, we track\n+the object directory containing it. During a write of a new commit-graph file,\n+we check for any changes in the source object directory and read the\n+`commit-graph-chain` file for that source and create a new file based on those\n+files. During this \"reparent\" operation, we necessarily need to collapse all\n+levels in the fork, as all of the files are invalid against the new base file.\n+\n+It is crucial to be careful when cleaning up \"unreferenced\" `graph-{hash}.graph`\n+files in this scenario. It falls to the user to define the proper settings for\n+their custom environment:\n+\n+ 1. When merging levels in the base repo, the unreferenced files may still be\n+ referenced by chains from fork repos.\n+\n+ 2. The expiry time should be set to a length of time such that every fork has\n+ time to recompute their commit-graph chain to \"reparent\" onto the new base\n+ file(s).\n+\n+ 3. If the commit-graph chain is updated in the base, the fork will not have\n+ access to the new chain until its chain is updated to reference those files.\n+ (This may change in the future [5].)\n+\n Related Links\n -------------\n [0] https://bugs.chromium.org/p/git/issues/detail?id=8\n@@ -292,3 +328,7 @@ Related Links\n \n [4] https://public-inbox.org/git/20180108154822.54829-1-git@jeffhostetler.com/T/#u\n A patch to remove the ahead-behind calculation from 'status'.\n+\n+[5] https://public-inbox.org/git/f27db281-abad-5043-6d71-cbb083b1c877@gmail.com/\n+ A discussion of a \"two-dimensional graph position\" that can allow reading\n+ multiple commit-graph chains at the same time.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex bfef1b6960..0b7c186a5d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -320,6 +320,9 @@ static struct commit_graph *load_commit_graph_v1(struct repository *r, const cha\n \tstruct commit_graph *g = load_commit_graph_one(graph_name);\n \tfree(graph_name);\n \n+\tif (g)\n+\t\tg->obj_dir = obj_dir;\n+\n \treturn g;\n }\n \n@@ -385,8 +388,7 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r, const\n \toids = xcalloc(st.st_size / (the_hash_algo->hexsz + 1), sizeof(struct object_id));\n \n \twhile (strbuf_getline_lf(&line, fp) != EOF && valid) {\n-\t\tchar *graph_name;\n-\t\tstruct commit_graph *g;\n+\t\tstruct object_directory *odb;\n \n \t\tif (get_oid_hex(line.buf, &oids[i])) {\n \t\t\twarning(_(\"invalid commit-graph chain: line '%s' not a hash\"),\n@@ -395,14 +397,23 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r, const\n \t\t\tbreak;\n \t\t}\n \n-\t\tgraph_name = get_split_graph_filename(obj_dir, line.buf);\n-\t\tg = load_commit_graph_one(graph_name);\n-\t\tfree(graph_name);\n+\t\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n+\t\t\tchar *graph_name = get_split_graph_filename(odb->path, line.buf);\n+\t\t\tstruct commit_graph *g = load_commit_graph_one(graph_name);\n \n-\t\tif (g && add_graph_to_chain(g, graph_chain, oids, i))\n-\t\t\tgraph_chain = g;\n-\t\telse\n-\t\t\tvalid = 0;\n+\t\t\tfree(graph_name);\n+\n+\t\t\tif (g) {\n+\t\t\t\tg->obj_dir = odb->path;\n+\n+\t\t\t\tif (add_graph_to_chain(g, graph_chain, oids, i))\n+\t\t\t\t\tgraph_chain = g;\n+\t\t\t\telse\n+\t\t\t\t\tvalid = 0;\n+\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n \t}\n \n \tfree(oids);\n@@ -1411,7 +1422,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \n \tif (ctx->split && ctx->base_graph_name && ctx->num_commit_graphs_after > 1) {\n \t\tchar *new_base_hash = xstrdup(oid_to_hex(&ctx->new_base_graph->oid));\n-\t\tchar *new_base_name = get_split_graph_filename(ctx->obj_dir, new_base_hash);\n+\t\tchar *new_base_name = get_split_graph_filename(ctx->new_base_graph->obj_dir, new_base_hash);\n \n \t\tfree(ctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2]);\n \t\tfree(ctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 2]);\n@@ -1482,6 +1493,9 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \n \twhile (g && (g->num_commits <= split_strategy_size_mult * num_commits ||\n \t\t num_commits > split_strategy_max_commits)) {\n+\t\tif (strcmp(g->obj_dir, ctx->obj_dir))\n+\t\t\tbreak;\n+\n \t\tnum_commits += g->num_commits;\n \t\tg = g->base_graph;\n \n@@ -1490,6 +1504,18 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \n \tctx->new_base_graph = g;\n \n+\tif (ctx->num_commit_graphs_after == 2) {\n+\t\tchar *old_graph_name = get_commit_graph_filename(g->obj_dir);\n+\n+\t\tif (!strcmp(g->filename, old_graph_name) &&\n+\t\t strcmp(g->obj_dir, ctx->obj_dir)) {\n+\t\t\tctx->num_commit_graphs_after = 1;\n+\t\t\tctx->new_base_graph = NULL;\n+\t\t}\n+\n+\t\tfree(old_graph_name);\n+\t}\n+\n \tALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);\n \tALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex 5c48c4f66a..10466bc064 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -48,6 +48,7 @@ struct commit_graph {\n \tuint32_t num_commits;\n \tstruct object_id oid;\n \tchar *filename;\n+\tconst char *obj_dir;\n \n \tuint32_t num_commits_in_base;\n \tstruct commit_graph *base_graph;\ndiff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\nindex 5cb5663a30..fad27dc6e3 100755\n--- a/t/t5323-split-commit-graph.sh\n+++ b/t/t5323-split-commit-graph.sh\n@@ -90,6 +90,21 @@ test_expect_success 'add more commits, and write a new base graph' '\n \tgraph_read_expect 12\n '\n \n+test_expect_success 'fork and fail to base a chain on a commit-graph file' '\n+\ttest_when_finished rm -rf fork &&\n+\tgit clone . fork &&\n+\t(\n+\t\tcd fork &&\n+\t\trm .git/objects/info/commit-graph &&\n+\t\techo \"$TRASH_DIRECTORY/.git/objects\" >.git/objects/info/alternates &&\n+\t\ttest_commit new-commit &&\n+\t\tgit commit-graph write --reachable --split &&\n+\t\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n+\t\tverify_chain_files_exist $graphdir\n+\t)\n+'\n+\n test_expect_success 'add three more commits, write a tip graph' '\n \tgit reset --hard commits/3 &&\n \tgit merge merge/1 &&\n@@ -132,4 +147,26 @@ test_expect_success 'add one commit, write a merged graph' '\n \n graph_git_behavior 'merged commit-graph: commit 12 vs 6' commits/12 commits/6\n \n+test_expect_success 'create fork and chain across alternate' '\n+\tgit clone . fork &&\n+\t(\n+\t\tcd fork &&\n+\t\tgit config core.commitGraph true &&\n+\t\trm -rf $graphdir &&\n+\t\techo \"$TRASH_DIRECTORY/.git/objects\" >.git/objects/info/alternates &&\n+\t\ttest_commit 13 &&\n+\t\tgit branch commits/13 &&\n+\t\tgit commit-graph write --reachable --split &&\n+\t\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\t\ttest_line_count = 3 $graphdir/commit-graph-chain &&\n+\t\tls $graphdir/graph-*.graph >graph-files &&\n+\t\ttest_line_count = 1 graph-files &&\n+\t\tgit -c core.commitGraph=true rev-list HEAD >expect &&\n+\t\tgit -c core.commitGraph=false rev-list HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+graph_git_behavior 'alternate: commit 13 vs 6' commits/13 commits/6\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"376585","messageId":"18d612be9e68a4f406a3c4c3f597024cd973097e.1559577826.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 11/14] commit-graph: expire commit-graph files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-03T16:03:59Z","receivedAt":"2019-06-03T16:04:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we merge commit-graph files in a commit-graph chain, we should clean\nup the files that are no longer used.\n\nThis change introduces an 'expiry_window' value to the context, which is\nalways zero (for now). We then check the modified time of each\ngraph-{hash}.graph file in the $OBJDIR/info/commit-graphs folder and\nunlink the files that are older than the expiry_window.\n\nSince this is always zero, this immediately clears all unused graph\nfiles. We will update the value to match a config setting in a future\nchange.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 15 ++++++\n commit-graph.c | 69 ++++++++++++++++++++++++\n t/t5323-split-commit-graph.sh | 2 +-\n 3 files changed, 85 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex 473032e476..aed4350a59 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -266,6 +266,21 @@ The merge strategy values (2 for the size multiple, 64,000 for the maximum\n number of commits) could be extracted into config settings for full\n flexibility.\n \n+## Deleting graph-{hash} files\n+\n+After a new tip file is written, some `graph-{hash}` files may no longer\n+be part of a chain. It is important to remove these files from disk, eventually.\n+The main reason to delay removal is that another process could read the\n+`commit-graph-chain` file before it is rewritten, but then look for the\n+`graph-{hash}` files after they are deleted.\n+\n+To allow holding old split commit-graphs for a while after they are unreferenced,\n+we update the modified times of the files when they become unreferenced. Then,\n+we scan the `$OBJDIR/info/commit-graphs/` directory for `graph-{hash}`\n+files whose modified times are older than a given expiry window. This window\n+defaults to zero, but can be changed using command-line arguments or a config\n+setting.\n+\n ## Chains across multiple object directories\n \n In a repo with alternates, we look for the `commit-graph-chain` file starting\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0b7c186a5d..4cbced7ff4 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1637,6 +1637,70 @@ static void merge_commit_graphs(struct write_commit_graph_context *ctx)\n \tdeduplicate_commits(ctx);\n }\n \n+static void mark_commit_graphs(struct write_commit_graph_context *ctx)\n+{\n+\tuint32_t i;\n+\ttime_t now = time(NULL);\n+\n+\tfor (i = ctx->num_commit_graphs_after - 1; i < ctx->num_commit_graphs_before; i++) {\n+\t\tstruct stat st;\n+\t\tstruct utimbuf updated_time;\n+\n+\t\tstat(ctx->commit_graph_filenames_before[i], &st);\n+\n+\t\tupdated_time.actime = st.st_atime;\n+\t\tupdated_time.modtime = now;\n+\t\tutime(ctx->commit_graph_filenames_before[i], &updated_time);\n+\t}\n+}\n+\n+static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tDIR *dir;\n+\tstruct dirent *de;\n+\tsize_t dirnamelen;\n+\ttime_t expire_time = time(NULL);\n+\n+\tstrbuf_addstr(&path, ctx->obj_dir);\n+\tstrbuf_addstr(&path, \"/info/commit-graphs\");\n+\tdir = opendir(path.buf);\n+\n+\tif (!dir) {\n+\t\tstrbuf_release(&path);\n+\t\treturn;\n+\t}\n+\n+\tstrbuf_addch(&path, '/');\n+\tdirnamelen = path.len;\n+\twhile ((de = readdir(dir)) != NULL) {\n+\t\tstruct stat st;\n+\t\tuint32_t i, found = 0;\n+\n+\t\tstrbuf_setlen(&path, dirnamelen);\n+\t\tstrbuf_addstr(&path, de->d_name);\n+\n+\t\tstat(path.buf, &st);\n+\n+\t\tif (st.st_mtime > expire_time)\n+\t\t\tcontinue;\n+\t\tif (path.len < 6 || strcmp(path.buf + path.len - 6, \".graph\"))\n+\t\t\tcontinue;\n+\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++) {\n+\t\t\tif (!strcmp(ctx->commit_graph_filenames_after[i],\n+\t\t\t\t path.buf)) {\n+\t\t\t\tfound = 1;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (!found)\n+\t\t\tunlink(path.buf);\n+\n+\t}\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -1749,6 +1813,11 @@ int write_commit_graph(const char *obj_dir,\n \n \tres = write_commit_graph_file(ctx);\n \n+\tif (ctx->split) {\n+\t\tmark_commit_graphs(ctx);\n+\t\texpire_commit_graphs(ctx);\n+\t}\n+\n cleanup:\n \tfree(ctx->graph_name);\n \tfree(ctx->commits.list);\ndiff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\nindex fad27dc6e3..ae45329651 100755\n--- a/t/t5323-split-commit-graph.sh\n+++ b/t/t5323-split-commit-graph.sh\n@@ -141,7 +141,7 @@ test_expect_success 'add one commit, write a merged graph' '\n \ttest_path_is_file $graphdir/commit-graph-chain &&\n \ttest_line_count = 2 $graphdir/commit-graph-chain &&\n \tls $graphdir/graph-*.graph >graph-files &&\n-\ttest_line_count = 4 graph-files &&\n+\ttest_line_count = 2 graph-files &&\n \tverify_chain_files_exist $graphdir\n '\n \n-- \ngitgitgadget\n\n"},{"id":"376586","messageId":"87fb895fe4e9ebc4fd97a6b193d1f99d0f27419c.1559577826.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 09/14] commit-graph: merge commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-03T16:03:57Z","receivedAt":"2019-06-03T16:04:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen searching for a commit in a commit-graph chain of G graphs with N\ncommits, the search takes O(G log N) time. If we always add a new tip\ngraph with every write, the linear G term will start to dominate and\nslow the lookup process.\n\nTo keep lookups fast, but also keep most incremental writes fast, create\na strategy for merging levels of the commit-graph chain. The strategy is\ndetailed in the commit-graph design document, but is summarized by these\ntwo conditions:\n\n 1. If the number of commits we are adding is more than half the number\n of commits in the graph below, then merge with that graph.\n\n 2. If we are writing more than 64,000 commits into a single graph,\n then merge with all lower graphs.\n\nThe numeric values in the conditions above are currently constant, but\ncan become config options in a future update.\n\nAs we merge levels of the commit-graph chain, check that the commits\nstill exist in the repository. A garbage-collection operation may have\nremoved those commits from the object store and we do not want to\npersist them in the commit-graph chain. This is a non-issue if the\n'git gc' process wrote a new, single-level commit-graph file.\n\nAfter we merge levels, the old graph-{hash}.graph files are no longer\nreferenced by the commit-graph-chain file. We will expire these files in\na future change.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 80 ++++++++++\n commit-graph.c | 184 +++++++++++++++++++----\n t/t5323-split-commit-graph.sh | 13 ++\n 3 files changed, 244 insertions(+), 33 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex 1dca3bd8fe..d9c6253b0a 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -186,6 +186,86 @@ positions to refer to their parents, which may be in `graph-{hash1}.graph` or\n its containment in the intervals [0, X0), [X0, X0 + X1), [X0 + X1, X0 + X1 +\n X2).\n \n+Each commit-graph file (except the base, `graph-{hash0}.graph`) contains data\n+specifying the hashes of all files in the lower layers. In the above example,\n+`graph-{hash1}.graph` contains `{hash0}` while `graph-{hash2}.graph` contains\n+`{hash0}` and `{hash1}`.\n+\n+## Merging commit-graph files\n+\n+If we only added a new commit-graph file on every write, we would run into a\n+linear search problem through many commit-graph files. Instead, we use a merge\n+strategy to decide when the stack should collapse some number of levels.\n+\n+The diagram below shows such a collapse. As a set of new commits are added, it\n+is determined by the merge strategy that the files should collapse to\n+`graph-{hash1}`. Thus, the new commits, the commits in `graph-{hash2}` and\n+the commits in `graph-{hash1}` should be combined into a new `graph-{hash3}`\n+file.\n+\n+\t\t\t +---------------------+\n+\t\t\t | |\n+\t\t\t | (new commits) |\n+\t\t\t | |\n+\t\t\t +---------------------+\n+\t\t\t | |\n+ +-----------------------+ +---------------------+\n+ | graph-{hash2} |->| |\n+ +-----------------------+ +---------------------+\n+\t | | |\n+ +-----------------------+ +---------------------+\n+ | | | |\n+ | graph-{hash1} |->| |\n+ | | | |\n+ +-----------------------+ +---------------------+\n+\t | tmp_graphXXX\n+ +-----------------------+\n+ | |\n+ | |\n+ | |\n+ | graph-{hash0} |\n+ | |\n+ | |\n+ | |\n+ +-----------------------+\n+\n+During this process, the commits to write are combined, sorted and we write the\n+contents to a temporary file, all while holding a `commit-graph-chain.lock`\n+lock-file. When the file is flushed, we rename it to `graph-{hash3}`\n+according to the computed `{hash3}`. Finally, we write the new chain data to\n+`commit-graph-chain.lock`:\n+\n+```\n+\t{hash3}\n+\t{hash0}\n+```\n+\n+We then close the lock-file.\n+\n+## Merge Strategy\n+\n+When writing a set of commits that do not exist in the commit-graph stack of\n+height N, we default to creating a new file at level N + 1. We then decide to\n+merge with the Nth level if one of two conditions hold:\n+\n+ 1. The expected file size for level N + 1 is at least half the file size for\n+ level N.\n+\n+ 2. Level N + 1 contains more than 64,0000 commits.\n+\n+This decision cascades down the levels: when we merge a level we create a new\n+set of commits that then compares to the next level.\n+\n+The first condition bounds the number of levels to be logarithmic in the total\n+number of commits. The second condition bounds the total number of commits in\n+a `graph-{hashN}` file and not in the `commit-graph` file, preventing\n+significant performance issues when the stack merges and another process only\n+partially reads the previous stack.\n+\n+The merge strategy values (2 for the size multiple, 64,000 for the maximum\n+number of commits) could be extracted into config settings for full\n+flexibility.\n+\n Related Links\n -------------\n [0] https://bugs.chromium.org/p/git/issues/detail?id=8\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 7d3e001479..bfef1b6960 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1291,36 +1291,6 @@ static int write_graph_chunk_base(struct hashfile *f,\n \treturn 0;\n }\n \n-static void init_commit_graph_chain(struct write_commit_graph_context *ctx)\n-{\n-\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n-\tuint32_t i;\n-\n-\tctx->new_base_graph = g;\n-\tctx->base_graph_name = xstrdup(g->filename);\n-\tctx->new_num_commits_in_base = g->num_commits + g->num_commits_in_base;\n-\n-\tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n-\n-\tALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);\n-\tALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);\n-\n-\tfor (i = 0; i < ctx->num_commit_graphs_before - 1; i++)\n-\t\tctx->commit_graph_filenames_after[i] = xstrdup(ctx->commit_graph_filenames_before[i]);\n-\n-\tif (ctx->num_commit_graphs_before)\n-\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_before - 1] =\n-\t\t\tget_split_graph_filename(ctx->obj_dir, oid_to_hex(&g->oid));\n-\n-\ti = ctx->num_commit_graphs_before - 1;\n-\n-\twhile (g) {\n-\t\tctx->commit_graph_hash_after[i] = xstrdup(oid_to_hex(&g->oid));\n-\t\ti--;\n-\t\tg = g->base_graph;\n-\t}\n-}\n-\n static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n@@ -1498,6 +1468,149 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \treturn 0;\n }\n \n+static int split_strategy_max_commits = 64000;\n+static float split_strategy_size_mult = 2.0f;\n+\n+static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n+{\n+\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\tuint32_t num_commits = ctx->commits.nr;\n+\tuint32_t i;\n+\n+\tg = ctx->r->objects->commit_graph;\n+\tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n+\n+\twhile (g && (g->num_commits <= split_strategy_size_mult * num_commits ||\n+\t\t num_commits > split_strategy_max_commits)) {\n+\t\tnum_commits += g->num_commits;\n+\t\tg = g->base_graph;\n+\n+\t\tctx->num_commit_graphs_after--;\n+\t}\n+\n+\tctx->new_base_graph = g;\n+\n+\tALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);\n+\tALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);\n+\n+\tfor (i = 0; i < ctx->num_commit_graphs_after &&\n+\t\t i < ctx->num_commit_graphs_before; i++)\n+\t\tctx->commit_graph_filenames_after[i] = xstrdup(ctx->commit_graph_filenames_before[i]);\n+\n+\ti = ctx->num_commit_graphs_before - 1;\n+\tg = ctx->r->objects->commit_graph;\n+\n+\twhile (g) {\n+\t\tif (i < ctx->num_commit_graphs_after)\n+\t\t\tctx->commit_graph_hash_after[i] = xstrdup(oid_to_hex(&g->oid));\n+\n+\t\ti--;\n+\t\tg = g->base_graph;\n+\t}\n+}\n+\n+static void merge_commit_graph(struct write_commit_graph_context *ctx,\n+\t\t\t struct commit_graph *g)\n+{\n+\tuint32_t i;\n+\tuint32_t offset = g->num_commits_in_base;\n+\n+\tALLOC_GROW(ctx->commits.list, ctx->commits.nr + g->num_commits, ctx->commits.alloc);\n+\n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct object_id oid;\n+\t\tstruct commit *result;\n+\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\n+\t\tload_oid_from_graph(g, i + offset, &oid);\n+\n+\t\t/* only add commits if they still exist in the repo */\n+\t\tresult = lookup_commit_reference_gently(ctx->r, &oid, 1);\n+\n+\t\tif (result) {\n+\t\t\tctx->commits.list[ctx->commits.nr] = result;\n+\t\t\tctx->commits.nr++;\n+\t\t}\n+\t}\n+}\n+\n+static int commit_compare(const void *_a, const void *_b)\n+{\n+\tconst struct commit *a = *(const struct commit **)_a;\n+\tconst struct commit *b = *(const struct commit **)_b;\n+\treturn oidcmp(&a->object.oid, &b->object.oid);\n+}\n+\n+static void deduplicate_commits(struct write_commit_graph_context *ctx)\n+{\n+\tuint32_t i, num_parents, last_distinct = 0, duplicates = 0;\n+\tstruct commit_list *parent;\n+\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"De-duplicating merged commits\"),\n+\t\t\t\t\tctx->commits.nr);\n+\n+\tQSORT(ctx->commits.list, ctx->commits.nr, commit_compare);\n+\n+\tctx->num_extra_edges = 0;\n+\tfor (i = 1; i < ctx->commits.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i);\n+\n+\t\tif (oideq(&ctx->commits.list[last_distinct]->object.oid,\n+\t\t\t &ctx->commits.list[i]->object.oid)) {\n+\t\t\tduplicates++;\n+\t\t} else {\n+\t\t\tif (duplicates)\n+\t\t\t\tctx->commits.list[last_distinct + 1] = ctx->commits.list[i];\n+\t\t\tlast_distinct++;\n+\n+\t\t\tnum_parents = 0;\n+\t\t\tfor (parent = ctx->commits.list[i]->parents; parent; parent = parent->next)\n+\t\t\t\tnum_parents++;\n+\n+\t\t\tif (num_parents > 2)\n+\t\t\t\tctx->num_extra_edges += num_parents - 2;\n+\t\t}\n+\t}\n+\n+\tctx->commits.nr -= duplicates;\n+\tstop_progress(&ctx->progress);\n+}\n+\n+static void merge_commit_graphs(struct write_commit_graph_context *ctx)\n+{\n+\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\tuint32_t current_graph_number = ctx->num_commit_graphs_before;\n+\tstruct strbuf progress_title = STRBUF_INIT;\n+\n+\twhile (g && current_graph_number >= ctx->num_commit_graphs_after) {\n+\t\tcurrent_graph_number--;\n+\n+\t\tif (ctx->report_progress) {\n+\t\t\tstrbuf_addstr(&progress_title, _(\"Merging commit-graph\"));\n+\t\t\tctx->progress = start_delayed_progress(progress_title.buf, 0);\n+\t\t}\n+\n+\t\tmerge_commit_graph(ctx, g);\n+\t\tstop_progress(&ctx->progress);\n+\t\tstrbuf_release(&progress_title);\n+\n+\t\tg = g->base_graph;\n+\t}\n+\n+\tif (g) {\n+\t\tctx->new_base_graph = g;\n+\t\tctx->new_num_commits_in_base = g->num_commits + g->num_commits_in_base;\n+\t}\n+\n+\tif (ctx->new_base_graph)\n+\t\tctx->base_graph_name = xstrdup(ctx->new_base_graph->filename);\n+\n+\tdeduplicate_commits(ctx);\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -1543,6 +1656,9 @@ int write_commit_graph(const char *obj_dir,\n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n \n+\tif (ctx->split && ctx->oids.alloc > split_strategy_max_commits)\n+\t\tctx->oids.alloc = split_strategy_max_commits;\n+\n \tif (ctx->append) {\n \t\tprepare_commit_graph_one(ctx->r, ctx->obj_dir);\n \t\tif (ctx->r->objects->commit_graph)\n@@ -1596,9 +1712,11 @@ int write_commit_graph(const char *obj_dir,\n \tif (!ctx->commits.nr)\n \t\tgoto cleanup;\n \n-\tif (ctx->split)\n-\t\tinit_commit_graph_chain(ctx);\n-\telse\n+\tif (ctx->split) {\n+\t\tsplit_graph_merge_strategy(ctx);\n+\n+\t\tmerge_commit_graphs(ctx);\n+\t} else\n \t\tctx->num_commit_graphs_after = 1;\n \n \tcompute_generation_numbers(ctx);\ndiff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\nindex ccd24bd22b..5cb5663a30 100755\n--- a/t/t5323-split-commit-graph.sh\n+++ b/t/t5323-split-commit-graph.sh\n@@ -119,4 +119,17 @@ test_expect_success 'add one commit, write a tip graph' '\n \n graph_git_behavior 'three-layer commit-graph: commit 11 vs 6' commits/11 commits/6\n \n+test_expect_success 'add one commit, write a merged graph' '\n+\ttest_commit 12 &&\n+\tgit branch commits/12 &&\n+\tgit commit-graph write --reachable --split &&\n+\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\tls $graphdir/graph-*.graph >graph-files &&\n+\ttest_line_count = 4 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n+graph_git_behavior 'merged commit-graph: commit 12 vs 6' commits/12 commits/6\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"376587","messageId":"ca41bf08d0f8a0db6067f09d18fdc48f94a3a68d.1559577826.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 14/14] commit-graph: clean up chains after flattened write","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-03T16:04:01Z","receivedAt":"2019-06-03T16:04:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf we write a commit-graph file without the split option, then\nwe write to $OBJDIR/info/commit-graph and start to ignore\nthe chains in $OBJDIR/info/commit-graphs/.\n\nUnlink the commit-graph-chain file and expire the graph-{hash}.graph\nfiles in $OBJDIR/info/commit-graphs/ during every write.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 12 +++++++++---\n t/t5323-split-commit-graph.sh | 12 ++++++++++++\n 2 files changed, 21 insertions(+), 3 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f7459d40f2..291a3c60ee 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1672,6 +1672,12 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \n \tif (ctx->split_opts && ctx->split_opts->expire_time)\n \t\texpire_time -= ctx->split_opts->expire_time;\n+\tif (!ctx->split) {\n+\t\tchar *chain_file_name = get_chain_filename(ctx->obj_dir);\n+\t\tunlink(chain_file_name);\n+\t\tfree(chain_file_name);\n+\t\tctx->num_commit_graphs_after = 0;\n+\t}\n \n \tstrbuf_addstr(&path, ctx->obj_dir);\n \tstrbuf_addstr(&path, \"/info/commit-graphs\");\n@@ -1826,10 +1832,10 @@ int write_commit_graph(const char *obj_dir,\n \n \tres = write_commit_graph_file(ctx);\n \n-\tif (ctx->split) {\n+\tif (ctx->split)\n \t\tmark_commit_graphs(ctx);\n-\t\texpire_commit_graphs(ctx);\n-\t}\n+\n+\texpire_commit_graphs(ctx);\n \n cleanup:\n \tfree(ctx->graph_name);\ndiff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\nindex b70dc90706..9a19de9e80 100755\n--- a/t/t5323-split-commit-graph.sh\n+++ b/t/t5323-split-commit-graph.sh\n@@ -204,6 +204,18 @@ test_expect_success 'test merge stragety constants' '\n \t)\n '\n \n+test_expect_success 'remove commit-graph-chain file after flattening' '\n+\tgit clone . flatten &&\n+\t(\n+\t\tcd flatten &&\n+\t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\t\tgit commit-graph write --reachable &&\n+\t\ttest_path_is_missing $graphdir/commit-graph-chain &&\n+\t\tls $graphdir >graph-files &&\n+\t\ttest_line_count = 0 graph-files\n+\t)\n+'\n+\n corrupt_file() {\n \tfile=$1\n \tpos=$2\n-- \ngitgitgadget\n"},{"id":"376588","messageId":"fe91ff5fca48e54cd24f840f6bca9cead4eb3718.1559577826.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 13/14] commit-graph: verify chains with --shallow mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-03T16:04:00Z","receivedAt":"2019-06-03T16:04:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf we wrote a commit-graph chain, we only modified the tip file in\nthe chain. It is valuable to verify what we wrote, but not waste\ntime checking files we did not write.\n\nAdd a '--shallow' option to the 'git commit-graph verify' subcommand\nand check that it does not read the base graph in a two-file chain.\n\nMaking the verify subcommand read from a chain of commit-graphs takes\nsome rearranging of the builtin code.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 5 ++++-\n builtin/commit-graph.c | 27 +++++++++++++++++++--------\n commit-graph.c | 15 ++++++++++++---\n commit-graph.h | 6 ++++--\n t/t5323-split-commit-graph.sh | 21 +++++++++++++++++++++\n 5 files changed, 60 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 365e145e82..eb5e7865f0 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -10,7 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git commit-graph read' [--object-dir ]\n-'git commit-graph verify' [--object-dir ]\n+'git commit-graph verify' [--object-dir ] [--shallow]\n 'git commit-graph write' [--object-dir ]\n \n \n@@ -80,6 +80,9 @@ Used for debugging purposes.\n \n Read the commit-graph file and verify its contents against the object\n database. Used to check for corrupted data.\n++\n+With the `--shallow` option, only check the tip commit-graph file in\n+a chain of split commit-graphs.\n \n \n EXAMPLES\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 18e3b61fb6..7cde1e1aaa 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -5,17 +5,18 @@\n #include \"parse-options.h\"\n #include \"repository.h\"\n #include \"commit-graph.h\"\n+#include \"object-store.h\"\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n-\tN_(\"git commit-graph verify [--object-dir ]\"),\n+\tN_(\"git commit-graph verify [--object-dir ] [--shallow]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] \"),\n \tNULL\n };\n \n static const char * const builtin_commit_graph_verify_usage[] = {\n-\tN_(\"git commit-graph verify [--object-dir ]\"),\n+\tN_(\"git commit-graph verify [--object-dir ] [--shallow]\"),\n \tNULL\n };\n \n@@ -36,6 +37,7 @@ static struct opts_commit_graph {\n \tint stdin_commits;\n \tint append;\n \tint split;\n+\tint shallow;\n } opts;\n \n static int graph_verify(int argc, const char **argv)\n@@ -45,11 +47,14 @@ static int graph_verify(int argc, const char **argv)\n \tint open_ok;\n \tint fd;\n \tstruct stat st;\n+\tint flags = 0;\n \n \tstatic struct option builtin_commit_graph_verify_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n \t\t\t N_(\"dir\"),\n \t\t\t N_(\"The object directory to store the graph\")),\n+\t\tOPT_BOOL(0, \"shallow\", &opts.shallow,\n+\t\t\t N_(\"if the commit-graph is split, only verify the tip file\")),\n \t\tOPT_END(),\n \t};\n \n@@ -59,21 +64,27 @@ static int graph_verify(int argc, const char **argv)\n \n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n+\tif (opts.shallow)\n+\t\tflags |= COMMIT_GRAPH_VERIFY_SHALLOW;\n \n \tgraph_name = get_commit_graph_filename(opts.obj_dir);\n \topen_ok = open_commit_graph(graph_name, &fd, &st);\n-\tif (!open_ok && errno == ENOENT)\n-\t\treturn 0;\n-\tif (!open_ok)\n+\tif (!open_ok && errno != ENOENT)\n \t\tdie_errno(_(\"Could not open commit-graph '%s'\"), graph_name);\n-\tgraph = load_commit_graph_one_fd_st(fd, &st);\n+\n \tFREE_AND_NULL(graph_name);\n \n+\tif (open_ok)\n+\t\tgraph = load_commit_graph_one_fd_st(fd, &st);\n+\t else\n+\t\tgraph = read_commit_graph_one(the_repository, opts.obj_dir);\n+\n+\t/* Return failure if open_ok predicted success */\n \tif (!graph)\n-\t\treturn 1;\n+\t\treturn !!open_ok;\n \n \tUNLEAK(graph);\n-\treturn verify_commit_graph(the_repository, graph);\n+\treturn verify_commit_graph(the_repository, graph, flags);\n }\n \n static int graph_read(int argc, const char **argv)\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 9d1e7393e4..f7459d40f2 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -422,7 +422,7 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r, const\n \treturn graph_chain;\n }\n \n-static struct commit_graph *read_commit_graph_one(struct repository *r, const char *obj_dir)\n+struct commit_graph *read_commit_graph_one(struct repository *r, const char *obj_dir)\n {\n \tstruct commit_graph *g = load_commit_graph_v1(r, obj_dir);\n \n@@ -1872,7 +1872,7 @@ static void graph_report(const char *fmt, ...)\n #define GENERATION_ZERO_EXISTS 1\n #define GENERATION_NUMBER_EXISTS 2\n \n-int verify_commit_graph(struct repository *r, struct commit_graph *g)\n+int verify_commit_graph(struct repository *r, struct commit_graph *g, int flags)\n {\n \tuint32_t i, cur_fanout_pos = 0;\n \tstruct object_id prev_oid, cur_oid, checksum;\n@@ -1880,6 +1880,7 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \tstruct hashfile *f;\n \tint devnull;\n \tstruct progress *progress = NULL;\n+\tint local_error = 0;\n \n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n@@ -1974,6 +1975,9 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\t\t\tbreak;\n \t\t\t}\n \n+\t\t\t/* parse parent in case it is in a base graph */\n+\t\t\tparse_commit_in_graph_one(r, g, graph_parents->item);\n+\n \t\t\tif (!oideq(&graph_parents->item->object.oid, &odb_parents->item->object.oid))\n \t\t\t\tgraph_report(_(\"commit-graph parent for %s is %s != %s\"),\n \t\t\t\t\t oid_to_hex(&cur_oid),\n@@ -2025,7 +2029,12 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t}\n \tstop_progress(&progress);\n \n-\treturn verify_commit_graph_error;\n+\tlocal_error = verify_commit_graph_error;\n+\n+\tif (!(flags & COMMIT_GRAPH_VERIFY_SHALLOW) && g->base_graph)\n+\t\tlocal_error |= verify_commit_graph(r, g->base_graph, flags);\n+\n+\treturn local_error;\n }\n \n void free_commit_graph(struct commit_graph *g)\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 194acab2b7..84e5e91fc6 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -61,7 +61,7 @@ struct commit_graph {\n };\n \n struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st);\n-\n+struct commit_graph *read_commit_graph_one(struct repository *r, const char *obj_dir);\n struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t\t\t\t\tsize_t graph_size);\n \n@@ -89,7 +89,9 @@ int write_commit_graph(const char *obj_dir,\n \t\t unsigned int flags,\n \t\t const struct split_commit_graph_opts *split_opts);\n \n-int verify_commit_graph(struct repository *r, struct commit_graph *g);\n+#define COMMIT_GRAPH_VERIFY_SHALLOW\t(1 << 0)\n+\n+int verify_commit_graph(struct repository *r, struct commit_graph *g, int flags);\n \n void close_commit_graph(struct repository *);\n void free_commit_graph(struct commit_graph *);\ndiff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\nindex 2f9f1bf0dd..b70dc90706 100755\n--- a/t/t5323-split-commit-graph.sh\n+++ b/t/t5323-split-commit-graph.sh\n@@ -204,4 +204,25 @@ test_expect_success 'test merge stragety constants' '\n \t)\n '\n \n+corrupt_file() {\n+\tfile=$1\n+\tpos=$2\n+\tdata=\"${3:-\\0}\"\n+\tprintf \"$data\" | dd of=\"$file\" bs=1 seek=\"$pos\" conv=notrunc\n+}\n+\n+test_expect_success 'verify shallow' '\n+\tgit clone . verify &&\n+\t(\n+\t\tcd verify &&\n+\t\tgit commit-graph verify &&\n+\t\tbase_file=$graphdir/graph-$(head -n 1 $graphdir/commit-graph-chain).graph &&\n+\t\tcorrupt_file \"$base_file\" 1760 \"\\01\" &&\n+\t\tgit commit-graph verify --shallow &&\n+\t\ttest_must_fail git commit-graph verify 2>test_err &&\n+\t\tgrep -v \"^+\" test_err >err &&\n+\t\ttest_i18ngrep \"incorrect checksum\" err\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"376589","messageId":"4de4bfba64fd20ee4c036fd3bd69771b20311b94.1559577826.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 12/14] commit-graph: create options for split files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-03T16:04:00Z","receivedAt":"2019-06-03T16:04:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe split commit-graph feature is now fully implemented, but needs\nsome more run-time configurability. Allow direct callers to 'git\ncommit-graph write --split' to specify the values used in the\nmerge strategy and the expire time.\n\nUpdate the documentation to specify these values.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 21 +++++++++++++-\n Documentation/technical/commit-graph.txt | 7 +++--\n builtin/commit-graph.c | 20 +++++++++++---\n builtin/commit.c | 2 +-\n builtin/gc.c | 3 +-\n commit-graph.c | 35 ++++++++++++++++--------\n commit-graph.h | 12 ++++++--\n t/t5323-split-commit-graph.sh | 35 ++++++++++++++++++++++++\n 8 files changed, 112 insertions(+), 23 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 624470e198..365e145e82 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -26,7 +26,7 @@ OPTIONS\n \tUse given directory for the location of packfiles and commit-graph\n \tfile. This parameter exists to specify the location of an alternate\n \tthat only has the objects directory, not a full `.git` directory. The\n-\tcommit-graph file is expected to be at `/info/commit-graph` and\n+\tcommit-graph file is expected to be in the `/info` directory and\n \tthe packfiles are expected to be in `/pack`.\n \n \n@@ -51,6 +51,25 @@ or `--stdin-packs`.)\n +\n With the `--append` option, include all commits that are present in the\n existing commit-graph file.\n++\n+With the `--split` option, write the commit-graph as a chain of multiple\n+commit-graph files stored in `/info/commit-graphs`. The new commits\n+not already in the commit-graph are added in a new \"tip\" file. This file\n+is merged with the existing file if the following merge conditions are\n+met:\n++\n+* If `--size-multiple=` is not specified, let `X` equal 2. If the new\n+tip file would have `N` commits and the previous tip has `M` commits and\n+`X` times `N` is greater than `M`, instead merge the two files into a\n+single file.\n++\n+* If `--max-commits=` is specified with `M` a positive integer, and the\n+new tip file would have more than `M` commits, then instead merge the new\n+tip with the previous tip.\n++\n+Finally, if `--expire-time=` is not specified, let `datetime`\n+be the current time. After writing the split commit-graph, delete all\n+unused commit-graph whose modified times are older than `datetime`.\n \n 'read'::\n \ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex aed4350a59..729fbcb32f 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -248,10 +248,11 @@ When writing a set of commits that do not exist in the commit-graph stack of\n height N, we default to creating a new file at level N + 1. We then decide to\n merge with the Nth level if one of two conditions hold:\n \n- 1. The expected file size for level N + 1 is at least half the file size for\n- level N.\n+ 1. `--size-multiple=` is specified or X = 2, and the number of commits in\n+ level N is less than X times the number of commits in level N + 1.\n \n- 2. Level N + 1 contains more than 64,0000 commits.\n+ 2. `--max-commits=` is specified with non-zero C and the number of commits\n+ in level N + 1 is more than C commits.\n \n This decision cascades down the levels: when we merge a level we create a new\n set of commits that then compares to the next level.\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex c2c07d3917..18e3b61fb6 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -10,7 +10,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph verify [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] \"),\n \tNULL\n };\n \n@@ -25,7 +25,7 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] \"),\n \tNULL\n };\n \n@@ -135,6 +135,7 @@ static int graph_read(int argc, const char **argv)\n }\n \n extern int read_replace_refs;\n+struct split_commit_graph_opts split_opts;\n \n static int graph_write(int argc, const char **argv)\n {\n@@ -158,9 +159,19 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"include all commits already in the commit-graph file\")),\n \t\tOPT_BOOL(0, \"split\", &opts.split,\n \t\t\tN_(\"allow writing an incremental commit-graph file\")),\n+\t\tOPT_INTEGER(0, \"max-commits\", &split_opts.max_commits,\n+\t\t\tN_(\"maximum number of commits in a non-base split commit-graph\")),\n+\t\tOPT_INTEGER(0, \"size-multiple\", &split_opts.size_multiple,\n+\t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n+\t\tOPT_EXPIRY_DATE(0, \"expire-time\", &split_opts.expire_time,\n+\t\t\tN_(\"maximum number of commits in a non-base split commit-graph\")),\n \t\tOPT_END(),\n \t};\n \n+\tsplit_opts.size_multiple = 2;\n+\tsplit_opts.max_commits = 0;\n+\tsplit_opts.expire_time = 0;\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\n@@ -177,7 +188,7 @@ static int graph_write(int argc, const char **argv)\n \tread_replace_refs = 0;\n \n \tif (opts.reachable)\n-\t\treturn write_commit_graph_reachable(opts.obj_dir, flags);\n+\t\treturn write_commit_graph_reachable(opts.obj_dir, flags, &split_opts);\n \n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n@@ -197,7 +208,8 @@ static int graph_write(int argc, const char **argv)\n \tresult = write_commit_graph(opts.obj_dir,\n \t\t\t\t pack_indexes,\n \t\t\t\t commit_hex,\n-\t\t\t\t flags);\n+\t\t\t\t flags,\n+\t\t\t\t &split_opts);\n \n \tUNLEAK(lines);\n \treturn result;\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex b001ef565d..9216e9c043 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1670,7 +1670,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\t \"not exceeded, and then \\\"git reset HEAD\\\" to recover.\"));\n \n \tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n-\t write_commit_graph_reachable(get_object_directory(), 0))\n+\t write_commit_graph_reachable(get_object_directory(), 0, NULL))\n \t\treturn 1;\n \n \trepo_rerere(the_repository, 0);\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex df2573f124..2ab590ffd4 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -666,7 +666,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \n \tif (gc_write_commit_graph &&\n \t write_commit_graph_reachable(get_object_directory(),\n-\t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_PROGRESS : 0))\n+\t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_PROGRESS : 0,\n+\t\t\t\t\t NULL))\n \t\treturn 1;\n \n \tif (auto_gc && too_many_loose_objects())\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 4cbced7ff4..9d1e7393e4 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -753,6 +753,8 @@ struct write_commit_graph_context {\n \tunsigned append:1,\n \t\t report_progress:1,\n \t\t split:1;\n+\n+\tconst struct split_commit_graph_opts *split_opts;\n };\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n@@ -1101,14 +1103,15 @@ static int add_ref_to_list(const char *refname,\n \treturn 0;\n }\n \n-int write_commit_graph_reachable(const char *obj_dir, unsigned int flags)\n+int write_commit_graph_reachable(const char *obj_dir, unsigned int flags,\n+\t\t\t\t const struct split_commit_graph_opts *split_opts)\n {\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n \tint result;\n \n \tfor_each_ref(add_ref_to_list, &list);\n \tresult = write_commit_graph(obj_dir, NULL, &list,\n-\t\t\t\t flags);\n+\t\t\t\t flags, split_opts);\n \n \tstring_list_clear(&list, 0);\n \treturn result;\n@@ -1479,20 +1482,25 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \treturn 0;\n }\n \n-static int split_strategy_max_commits = 64000;\n-static float split_strategy_size_mult = 2.0f;\n-\n static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n {\n \tstruct commit_graph *g = ctx->r->objects->commit_graph;\n \tuint32_t num_commits = ctx->commits.nr;\n \tuint32_t i;\n \n+\tint max_commits = 0;\n+\tint size_mult = 2;\n+\n+\tif (ctx->split_opts) {\n+\t\tmax_commits = ctx->split_opts->max_commits;\n+\t\tsize_mult = ctx->split_opts->size_multiple;\n+\t}\n+\n \tg = ctx->r->objects->commit_graph;\n \tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n \n-\twhile (g && (g->num_commits <= split_strategy_size_mult * num_commits ||\n-\t\t num_commits > split_strategy_max_commits)) {\n+\twhile (g && (g->num_commits <= size_mult * num_commits ||\n+\t\t (max_commits && num_commits > max_commits))) {\n \t\tif (strcmp(g->obj_dir, ctx->obj_dir))\n \t\t\tbreak;\n \n@@ -1660,7 +1668,10 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \tDIR *dir;\n \tstruct dirent *de;\n \tsize_t dirnamelen;\n-\ttime_t expire_time = time(NULL);\n+\ttimestamp_t expire_time = time(NULL);\n+\n+\tif (ctx->split_opts && ctx->split_opts->expire_time)\n+\t\texpire_time -= ctx->split_opts->expire_time;\n \n \tstrbuf_addstr(&path, ctx->obj_dir);\n \tstrbuf_addstr(&path, \"/info/commit-graphs\");\n@@ -1704,7 +1715,8 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t unsigned int flags)\n+\t\t unsigned int flags,\n+\t\t const struct split_commit_graph_opts *split_opts)\n {\n \tstruct write_commit_graph_context *ctx;\n \tuint32_t i, count_distinct = 0;\n@@ -1719,6 +1731,7 @@ int write_commit_graph(const char *obj_dir,\n \tctx->append = flags & COMMIT_GRAPH_APPEND ? 1 : 0;\n \tctx->report_progress = flags & COMMIT_GRAPH_PROGRESS ? 1 : 0;\n \tctx->split = flags & COMMIT_GRAPH_SPLIT ? 1 : 0;\n+\tctx->split_opts = split_opts;\n \n \tif (ctx->split) {\n \t\tstruct commit_graph *g;\n@@ -1746,8 +1759,8 @@ int write_commit_graph(const char *obj_dir,\n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n \n-\tif (ctx->split && ctx->oids.alloc > split_strategy_max_commits)\n-\t\tctx->oids.alloc = split_strategy_max_commits;\n+\tif (ctx->split && split_opts && ctx->oids.alloc > split_opts->max_commits)\n+\t\tctx->oids.alloc = split_opts->max_commits;\n \n \tif (ctx->append) {\n \t\tprepare_commit_graph_one(ctx->r, ctx->obj_dir);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 10466bc064..194acab2b7 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -75,11 +75,19 @@ int generation_numbers_enabled(struct repository *r);\n #define COMMIT_GRAPH_PROGRESS (1 << 1)\n #define COMMIT_GRAPH_SPLIT (1 << 2)\n \n-int write_commit_graph_reachable(const char *obj_dir, unsigned int flags);\n+struct split_commit_graph_opts {\n+\tint size_multiple;\n+\tint max_commits;\n+\ttimestamp_t expire_time;\n+};\n+\n+int write_commit_graph_reachable(const char *obj_dir, unsigned int flags,\n+\t\t\t\t const struct split_commit_graph_opts *split_opts);\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t unsigned int flags);\n+\t\t unsigned int flags,\n+\t\t const struct split_commit_graph_opts *split_opts);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \ndiff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\nindex ae45329651..2f9f1bf0dd 100755\n--- a/t/t5323-split-commit-graph.sh\n+++ b/t/t5323-split-commit-graph.sh\n@@ -169,4 +169,39 @@ test_expect_success 'create fork and chain across alternate' '\n \n graph_git_behavior 'alternate: commit 13 vs 6' commits/13 commits/6\n \n+test_expect_success 'test merge stragety constants' '\n+\tgit clone . merge-2 &&\n+\t(\n+\t\tcd merge-2 &&\n+\t\tgit config core.commitGraph true &&\n+\t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\t\ttest_commit 14 &&\n+\t\tgit commit-graph write --reachable --split --size-multiple=2 &&\n+\t\ttest_line_count = 3 $graphdir/commit-graph-chain\n+\n+\t) &&\n+\tgit clone . merge-10 &&\n+\t(\n+\t\tcd merge-10 &&\n+\t\tgit config core.commitGraph true &&\n+\t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\t\ttest_commit 14 &&\n+\t\tgit commit-graph write --reachable --split --size-multiple=10 &&\n+\t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n+\t\tls $graphdir/graph-*.graph >graph-files &&\n+\t\ttest_line_count = 1 graph-files\n+\t) &&\n+\tgit clone . merge-10-expire &&\n+\t(\n+\t\tcd merge-10-expire &&\n+\t\tgit config core.commitGraph true &&\n+\t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\t\ttest_commit 15 &&\n+\t\tgit commit-graph write --reachable --split --size-multiple=10 --expire-time=1980-01-01 &&\n+\t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n+\t\tls $graphdir/graph-*.graph >graph-files &&\n+\t\ttest_line_count = 3 graph-files\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"376720","messageId":"xmqqmuiwrlpz.fsf@gitster-ct.c.googlers.com","threadId":"51052","inReplyTo":"b184919255bfc9e0d57e1a5ca12fb76957879c61.1559577826.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 01/14] commit-graph: document commit-graph chains","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-05T17:22:16Z","receivedAt":"2019-06-05T17:22:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> Add a basic description of commit-graph chains. More details about the\n> feature will be added as we add functionality. This introduction gives a\n> high-level overview to the goals of the feature and the basic layout of\n> commit-graph chains.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/technical/commit-graph.txt | 59 ++++++++++++++++++++++++\n> 1 file changed, 59 insertions(+)\n>\n> diff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\n> index fb53341d5e..1dca3bd8fe 100644\n> --- a/Documentation/technical/commit-graph.txt\n> +++ b/Documentation/technical/commit-graph.txt\n> @@ -127,6 +127,65 @@ Design Details\n> helpful for these clones, anyway. The commit-graph will not be read or\n> written when shallow commits are present.\n> \n> +Commit Graphs Chains\n> +--------------------\n> +\n> +Typically, repos grow with near-constant velocity (commits per day). Over time,\n> +the number of commits added by a fetch operation is much smaller than the\n> +number of commits in the full history. By creating a \"chain\" of commit-graphs,\n> +we enable fast writes of new commit data without rewriting the entire commit\n> +history -- at least, most of the time.\n> +\n> +## File Layout\n> +\n> +A commit-graph chain uses multiple files, and we use a fixed naming convention\n> +to organize these files. Each commit-graph file has a name\n> +`$OBJDIR/info/commit-graphs/graph-{hash}.graph` where `{hash}` is the hex-\n> +valued hash stored in the footer of that file (which is a hash of the file's\n> +contents before that hash). For a chain of commit-graph files, a plain-text\n> +file at `$OBJDIR/info/commit-graphs/commit-graph-chain` contains the\n> +hashes for the files in order from \"lowest\" to \"highest\".\n> +\n> +For example, if the `commit-graph-chain` file contains the lines\n> +\n> +```\n> +\t{hash0}\n> +\t{hash1}\n> +\t{hash2}\n> +```\n> +\n> +then the commit-graph chain looks like the following diagram:\n> +\n> + +-----------------------+\n> + | graph-{hash2}.graph |\n> + +-----------------------+\n> +\t |\n> + +-----------------------+\n> + | |\n> + | graph-{hash1}.graph |\n> + | |\n> + +-----------------------+\n> +\t |\n> + +-----------------------+\n> + | |\n> + | |\n> + | |\n> + | graph-{hash0}.graph |\n> + | |\n> + | |\n> + | |\n> + +-----------------------+\n> +\n> +Let X0 be the number of commits in `graph-{hash0}.graph`, X1 be the number of\n> +commits in `graph-{hash1}.graph`, and X2 be the number of commits in\n> +`graph-{hash2}.graph`. If a commit appears in position i in `graph-{hash2}.graph`,\n> +then we interpret this as being the commit in position (X0 + X1 + i), and that\n> +will be used as its \"graph position\". The commits in `graph-{hash2}.graph` use these\n> +positions to refer to their parents, which may be in `graph-{hash1}.graph` or\n> +`graph-{hash0}.graph`. We can navigate to an arbitrary commit in position j by checking\n> +its containment in the intervals [0, X0), [X0, X0 + X1), [X0 + X1, X0 + X1 +\n> +X2).\n\nOne thing that I fail to read from the above is what it means for\ngraphs to be inside a single chain. What is the significance for a\ngraph file graph-{hash1}.graph to be between graph-{hash0}.graph and\ngraph-{hash2}.graph? For example, is any of the following true?\n\n - For a commit in graph-{hash1}.graph file, if graph->{hash0}.graph\n or any other graph files lower in the position in the chain were\n unavailable, information on some ancestor of that commit may not\n be available.\n\n - Even if graph-{hash2}.graph or any other graph files higher in\n the position in the chain gets lost, information on a commit in\n graph-{hash1}.graph file or any of its ancestors is not affected.\n\nAnother thing I've assumed to be true but cannot read from the above\ndescription is that the hashes in `commit-graph-chain` file, other\nthan the newest one, are merely redundant information, and each\ngraph file records the hash of its \"previous\" graph file (i.e. the\none that used to be the youngest before it got created).\n\n\n \n"},{"id":"376721","messageId":"62612a31-7649-5410-3732-ff9b6b61da31@gmail.com","threadId":"51052","inReplyTo":"xmqqmuiwrlpz.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3 01/14] commit-graph: document commit-graph chains","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-06-05T18:09:47Z","receivedAt":"2019-06-05T18:09:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/5/2019 1:22 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> From: Derrick Stolee \n>>\n>> Add a basic description of commit-graph chains. More details about the\n>> feature will be added as we add functionality. This introduction gives a\n>> high-level overview to the goals of the feature and the basic layout of\n>> commit-graph chains.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> Documentation/technical/commit-graph.txt | 59 ++++++++++++++++++++++++\n>> 1 file changed, 59 insertions(+)\n>>\n>> diff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\n>> index fb53341d5e..1dca3bd8fe 100644\n>> --- a/Documentation/technical/commit-graph.txt\n>> +++ b/Documentation/technical/commit-graph.txt\n>> @@ -127,6 +127,65 @@ Design Details\n>> helpful for these clones, anyway. The commit-graph will not be read or\n>> written when shallow commits are present.\n>> \n>> +Commit Graphs Chains\n>> +--------------------\n>> +\n>> +Typically, repos grow with near-constant velocity (commits per day). Over time,\n>> +the number of commits added by a fetch operation is much smaller than the\n>> +number of commits in the full history. By creating a \"chain\" of commit-graphs,\n>> +we enable fast writes of new commit data without rewriting the entire commit\n>> +history -- at least, most of the time.\n>> +\n>> +## File Layout\n>> +\n>> +A commit-graph chain uses multiple files, and we use a fixed naming convention\n>> +to organize these files. Each commit-graph file has a name\n>> +`$OBJDIR/info/commit-graphs/graph-{hash}.graph` where `{hash}` is the hex-\n>> +valued hash stored in the footer of that file (which is a hash of the file's\n>> +contents before that hash). For a chain of commit-graph files, a plain-text\n>> +file at `$OBJDIR/info/commit-graphs/commit-graph-chain` contains the\n>> +hashes for the files in order from \"lowest\" to \"highest\".\n>> +\n>> +For example, if the `commit-graph-chain` file contains the lines\n>> +\n>> +```\n>> +\t{hash0}\n>> +\t{hash1}\n>> +\t{hash2}\n>> +```\n>> +\n>> +then the commit-graph chain looks like the following diagram:\n>> +\n>> + +-----------------------+\n>> + | graph-{hash2}.graph |\n>> + +-----------------------+\n>> +\t |\n>> + +-----------------------+\n>> + | |\n>> + | graph-{hash1}.graph |\n>> + | |\n>> + +-----------------------+\n>> +\t |\n>> + +-----------------------+\n>> + | |\n>> + | |\n>> + | |\n>> + | graph-{hash0}.graph |\n>> + | |\n>> + | |\n>> + | |\n>> + +-----------------------+\n>> +\n>> +Let X0 be the number of commits in `graph-{hash0}.graph`, X1 be the number of\n>> +commits in `graph-{hash1}.graph`, and X2 be the number of commits in\n>> +`graph-{hash2}.graph`. If a commit appears in position i in `graph-{hash2}.graph`,\n>> +then we interpret this as being the commit in position (X0 + X1 + i), and that\n>> +will be used as its \"graph position\". The commits in `graph-{hash2}.graph` use these\n>> +positions to refer to their parents, which may be in `graph-{hash1}.graph` or\n>> +`graph-{hash0}.graph`. We can navigate to an arbitrary commit in position j by checking\n>> +its containment in the intervals [0, X0), [X0, X0 + X1), [X0 + X1, X0 + X1 +\n>> +X2).\n> \n> One thing that I fail to read from the above is what it means for\n> graphs to be inside a single chain. What is the significance for a\n> graph file graph-{hash1}.graph to be between graph-{hash0}.graph and\n> graph-{hash2}.graph? For example, is any of the following true?\n> \n> - For a commit in graph-{hash1}.graph file, if graph->{hash0}.graph\n> or any other graph files lower in the position in the chain were\n> unavailable, information on some ancestor of that commit may not\n> be available.\n\nNot only that, but if graph-{hash0}.graph is unavailable, then the\ngraph-{hash1}.graph file is _unusable_. For example, we don't track\nthe number of commits in the base graph, so we could not accurately\nfind the commit-parent links even within graph-{hash1}.graph.\n\n> - Even if graph-{hash2}.graph or any other graph files higher in\n> the position in the chain gets lost, information on a commit in\n> graph-{hash1}.graph file or any of its ancestors is not affected.\n\nThis is correct. As we \"build from the bottom\" we can stop if we fail\nto find a file, and all the data we already accessed is still valid.\n\n> Another thing I've assumed to be true but cannot read from the above\n> description is that the hashes in `commit-graph-chain` file, other\n> than the newest one, are merely redundant information, and each\n> graph file records the hash of its \"previous\" graph file (i.e. the\n> one that used to be the youngest before it got created).\n\nIf the entire chain is available, then we only really need the tip hash.\nHowever, having the entire chain in this file allows the \"building from\nthe bottom\" pattern so we can get some value even if the tip was removed\nfrom under us. Since we expect the base file to change infrequently, this\nshould cover a large number of commits.\n\nI can try to make this pattern more clear in a future revision, assuming\nwe stick with the pattern. It remains unclear if this strategy as a whole\nhas been accepted as a good direction.\n\nThanks,\n-Stolee\n"},{"id":"376744","messageId":"6a6a0b72-e503-1fdd-e67e-2a84d825f89b@iee.org","threadId":"51052","inReplyTo":"b184919255bfc9e0d57e1a5ca12fb76957879c61.1559577826.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 01/14] commit-graph: document commit-graph chains","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2019-06-06T12:10:31Z","receivedAt":"2019-06-06T12:10:35Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Hi Derrick ,\n\nOn 03/06/2019 17:03, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n>\n> Add a basic description of commit-graph chains.\nNot really your problem, but I did notice that we don't actually explain \nwhat we mean here by a commit graph (before we start chaining them), and \nthe distinction between the generic concept and the specific implementation.\n\nIf I understand it correctly, the regular DAG (directed acyclic graph) \nalready inherently contains the commit graph, showing the parent(s) of \neach commit. Hence, why do we need another? (which then needs explaining \nthe what/why/how)\n\nSo, in one sense, another commit chain is potentially duplicated \nredundant data. What hasn't been surfaced (for the reader coming later) \nis probably that accessing the DAG commit graph can be (a) slow, (b) one \nway (no child relationships), and (c) accesses large amounts of other \ndata that isn't relevant to the task at hand.\n\nSo the commit graph (implementation) is [I think] a fast, compact, \nsorted(?), list of commit oids that provides two way linkage through the \ncommit graph (?) to allow fast queries within the Git codebase.\n\nThe commit graph is normally considered immutable, however the DAG \ncommit graph can be extended by new commits, trimmed by branch deletion, \nrebasing, forced push, etc, or even reorganised via 'replace' or grafts \ncommits, which must then be reflected in the commit graph (implementation).\n\nIt just felt that there is a gap between the high level DAG, explained \nin the glossary, and the commit-graph That perhaps the \ntechnical/commit-graph.txt ought to summarise.\n\n--\nPhilip\n> More details about the\n> feature will be added as we add functionality. This introduction gives a\n> high-level overview to the goals of the feature and the basic layout of\n> commit-graph chains.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/technical/commit-graph.txt | 59 ++++++++++++++++++++++++\n> 1 file changed, 59 insertions(+)\n>\n> diff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\n> index fb53341d5e..1dca3bd8fe 100644\n> --- a/Documentation/technical/commit-graph.txt\n> +++ b/Documentation/technical/commit-graph.txt\n> @@ -127,6 +127,65 @@ Design Details\n> helpful for these clones, anyway. The commit-graph will not be read or\n> written when shallow commits are present.\n> \n> +Commit Graphs Chains\n> +--------------------\n> +\n> +Typically, repos grow with near-constant velocity (commits per day). Over time,\n> +the number of commits added by a fetch operation is much smaller than the\n> +number of commits in the full history. By creating a \"chain\" of commit-graphs,\n> +we enable fast writes of new commit data without rewriting the entire commit\n> +history -- at least, most of the time.\n> +\n> +## File Layout\n> +\n> +A commit-graph chain uses multiple files, and we use a fixed naming convention\n> +to organize these files. Each commit-graph file has a name\n> +`$OBJDIR/info/commit-graphs/graph-{hash}.graph` where `{hash}` is the hex-\n> +valued hash stored in the footer of that file (which is a hash of the file's\n> +contents before that hash). For a chain of commit-graph files, a plain-text\n> +file at `$OBJDIR/info/commit-graphs/commit-graph-chain` contains the\n> +hashes for the files in order from \"lowest\" to \"highest\".\n> +\n> +For example, if the `commit-graph-chain` file contains the lines\n> +\n> +```\n> +\t{hash0}\n> +\t{hash1}\n> +\t{hash2}\n> +```\n> +\n> +then the commit-graph chain looks like the following diagram:\n> +\n> + +-----------------------+\n> + | graph-{hash2}.graph |\n> + +-----------------------+\n> +\t |\n> + +-----------------------+\n> + | |\n> + | graph-{hash1}.graph |\n> + | |\n> + +-----------------------+\n> +\t |\n> + +-----------------------+\n> + | |\n> + | |\n> + | |\n> + | graph-{hash0}.graph |\n> + | |\n> + | |\n> + | |\n> + +-----------------------+\n> +\n> +Let X0 be the number of commits in `graph-{hash0}.graph`, X1 be the number of\n> +commits in `graph-{hash1}.graph`, and X2 be the number of commits in\n> +`graph-{hash2}.graph`. If a commit appears in position i in `graph-{hash2}.graph`,\n> +then we interpret this as being the commit in position (X0 + X1 + i), and that\n> +will be used as its \"graph position\". The commits in `graph-{hash2}.graph` use these\n> +positions to refer to their parents, which may be in `graph-{hash1}.graph` or\n> +`graph-{hash0}.graph`. We can navigate to an arbitrary commit in position j by checking\n> +its containment in the intervals [0, X0), [X0, X0 + X1), [X0 + X1, X0 + X1 +\n> +X2).\n> +\n> Related Links\n> -------------\n> [0] https://bugs.chromium.org/p/git/issues/detail?id=8\n\n"},{"id":"376751","messageId":"pull.184.v4.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v4 00/14] Commit-graph: Write incremental files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-06T14:15:28Z","receivedAt":"2019-06-06T14:15:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This version is now ready for review.\n\nThe commit-graph is a valuable performance feature for repos with large\ncommit histories, but suffers from the same problem as git repack: it\nrewrites the entire file every time. This can be slow when there are\nmillions of commits, especially after we stopped reading from the\ncommit-graph file during a write in 43d3561 (commit-graph write: don't die\nif the existing graph is corrupt).\n\nInstead, create a \"chain\" of commit-graphs in the\n.git/objects/info/commit-graphs folder with name graph-{hash}.graph. The\nlist of hashes is given by the commit-graph-chain file, and also in a \"base\ngraph chunk\" in the commit-graph format. As we read a chain, we can verify\nthat the hashes match the trailing hash of each commit-graph we read along\nthe way and each hash below a level is expected by that graph file.\n\nWhen writing, we don't always want to add a new level to the stack. This\nwould eventually result in performance degradation, especially when\nsearching for a commit (before we know its graph position). We decide to\nmerge levels of the stack when the new commits we will write satisfy two\nconditions:\n\n 1. The expected size of the new file is more than half the size of the tip\n of the stack.\n 2. The new file contains more than 64,000 commits.\n\nThe first condition alone would prevent more than a logarithmic number of\nlevels. The second condition is a stop-gap to prevent performance issues\nwhen another process starts reading the commit-graph stack as we are merging\na large stack of commit-graph files. The reading process could be in a state\nwhere the new file is not ready, but the levels above the new file were\nalready deleted. Thus, the commits that were merged down must be parsed from\npack-files.\n\nThe performance is necessarily amortized across multiple writes, so I tested\nby writing commit-graphs from the (non-rc) tags in the Linux repo. My test\nincluded 72 tags, and wrote everything reachable from the tag using \n--stdin-commits. Here are the overall perf numbers:\n\nwrite --stdin-commits: 8m 12s\nwrite --stdin-commits --split: 28s\nwrite --split && verify --shallow: 60s\n\nUpdates in V3:\n\n * git commit-graph verify now works on commit-graph chains. We do a simple\n test to check the behavior of a new --shallow option.\n \n \n * When someone writes a flat commit-graph, we now expire the old chain\n according to the expire time.\n \n \n * The \"max commits\" limit is no longer enabled by default, but instead is\n enabled by a --max-commits= option. Ignored if n=0.\n \n \n\nUpdates in V4:\n\nJohannes pointed out some test failures on the Windows platform. We found\nthat the tests were not running on Windows in the gitgitgadget PR builds,\nwhich is now resolved.\n\n * We need to close commit-graphs recursively down the chain. This prevented\n an unlink() from working because of an open handle.\n \n \n * Creating the alternates file used a path-specification that didn't work\n on Windows.\n \n \n * Renaming a file to the same name failed, but is probably related to the\n unlink() error mentioned above.\n \n \n\nThis is based on ds/commit-graph-write-refactor.\n\nThanks, -Stolee\n\n[1] \nhttps://github.com/git/git/commit/43d356180556180b4ef6ac232a14498a5bb2b446\ncommit-graph write: don't die if the existing graph is corrupt\n\nDerrick Stolee (14):\n commit-graph: document commit-graph chains\n commit-graph: prepare for commit-graph chains\n commit-graph: rename commit_compare to oid_compare\n commit-graph: load commit-graph chains\n commit-graph: add base graphs chunk\n commit-graph: rearrange chunk count logic\n commit-graph: write commit-graph chains\n commit-graph: add --split option to builtin\n commit-graph: merge commit-graph chains\n commit-graph: allow cross-alternate chains\n commit-graph: expire commit-graph files\n commit-graph: create options for split files\n commit-graph: verify chains with --shallow mode\n commit-graph: clean up chains after flattened write\n\n Documentation/git-commit-graph.txt | 26 +-\n .../technical/commit-graph-format.txt | 11 +-\n Documentation/technical/commit-graph.txt | 195 +++++\n builtin/commit-graph.c | 53 +-\n builtin/commit.c | 2 +-\n builtin/gc.c | 3 +-\n commit-graph.c | 794 +++++++++++++++++-\n commit-graph.h | 25 +-\n t/t5318-commit-graph.sh | 2 +-\n t/t5323-split-commit-graph.sh | 240 ++++++\n 10 files changed, 1283 insertions(+), 68 deletions(-)\n create mode 100755 t/t5323-split-commit-graph.sh\n\n\nbase-commit: 8520d7fc7c6edd4d71582c69a873436029b6cb1b\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-184%2Fderrickstolee%2Fgraph%2Fincremental-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-184/derrickstolee/graph/incremental-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/184\n\nRange-diff vs v3:\n\n 1: b184919255 = 1: b184919255 commit-graph: document commit-graph chains\n 2: d0dc154a27 = 2: d0dc154a27 commit-graph: prepare for commit-graph chains\n 3: f35b04224a = 3: f35b04224a commit-graph: rename commit_compare to oid_compare\n 4: ca670536df = 4: ca670536df commit-graph: load commit-graph chains\n 5: df44cbc1bf = 5: df44cbc1bf commit-graph: add base graphs chunk\n 6: e65f9e841d = 6: e65f9e841d commit-graph: rearrange chunk count logic\n 7: fe0aa343cd = 7: fe0aa343cd commit-graph: write commit-graph chains\n 8: 4f4ccc8062 ! 8: c42e683ef6 commit-graph: add --split option to builtin\n @@ -12,6 +12,7 @@\n Add a new test script (t5323-split-commit-graph.sh) that demonstrates this\n behavior.\n \n + Helped-by: Johannes Schindelin \n Signed-off-by: Derrick Stolee \n \n diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n @@ -65,6 +66,32 @@\n \tread_replace_refs = 0;\n \n \n + diff --git a/commit-graph.c b/commit-graph.c\n + --- a/commit-graph.c\n + +++ b/commit-graph.c\n +@@\n + \t\t}\n + \n + \t\tif (ctx->base_graph_name) {\n +-\t\t\tresult = rename(ctx->base_graph_name,\n +-\t\t\t\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2]);\n ++\t\t\tconst char *dest = ctx->commit_graph_filenames_after[\n ++\t\t\t\t\t\tctx->num_commit_graphs_after - 2];\n + \n +-\t\t\tif (result) {\n +-\t\t\t\terror(_(\"failed to rename base commit-graph file\"));\n +-\t\t\t\treturn -1;\n ++\t\t\tif (strcmp(ctx->base_graph_name, dest)) {\n ++\t\t\t\tresult = rename(ctx->base_graph_name, dest);\n ++\n ++\t\t\t\tif (result) {\n ++\t\t\t\t\terror(_(\"failed to rename base commit-graph file\"));\n ++\t\t\t\t\treturn -1;\n ++\t\t\t\t}\n + \t\t\t}\n + \t\t} else {\n + \t\t\tchar *graph_name = get_commit_graph_filename(ctx->obj_dir);\n +\n diff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\n new file mode 100755\n --- /dev/null\n 9: 87fb895fe4 = 9: d065758454 commit-graph: merge commit-graph chains\n 10: 5cfd653d24 ! 10: 62b3fca582 commit-graph: allow cross-alternate chains\n @@ -195,7 +195,7 @@\n +\t(\n +\t\tcd fork &&\n +\t\trm .git/objects/info/commit-graph &&\n -+\t\techo \"$TRASH_DIRECTORY/.git/objects\" >.git/objects/info/alternates &&\n ++\t\techo \"$(pwd)\"/../.git/objects >.git/objects/info/alternates &&\n +\t\ttest_commit new-commit &&\n +\t\tgit commit-graph write --reachable --split &&\n +\t\ttest_path_is_file $graphdir/commit-graph-chain &&\n @@ -217,7 +217,7 @@\n +\t\tcd fork &&\n +\t\tgit config core.commitGraph true &&\n +\t\trm -rf $graphdir &&\n -+\t\techo \"$TRASH_DIRECTORY/.git/objects\" >.git/objects/info/alternates &&\n ++\t\techo \"$(pwd)\"/../.git/objects >.git/objects/info/alternates &&\n +\t\ttest_commit 13 &&\n +\t\tgit branch commits/13 &&\n +\t\tgit commit-graph write --reachable --split &&\n 11: 18d612be9e ! 11: b5aeeed909 commit-graph: expire commit-graph files\n @@ -45,6 +45,26 @@\n diff --git a/commit-graph.c b/commit-graph.c\n --- a/commit-graph.c\n +++ b/commit-graph.c\n +@@\n + \treturn !!first_generation;\n + }\n + \n ++static void close_commit_graph_one(struct commit_graph *g)\n ++{\n ++\tif (!g)\n ++\t\treturn;\n ++\n ++\tclose_commit_graph_one(g->base_graph);\n ++\tfree_commit_graph(g);\n ++}\n ++\n + void close_commit_graph(struct repository *r)\n + {\n +-\tfree_commit_graph(r->objects->commit_graph);\n ++\tclose_commit_graph_one(r->objects->commit_graph);\n + \tr->objects->commit_graph = NULL;\n + }\n + \n @@\n \tdeduplicate_commits(ctx);\n }\n @@ -109,7 +129,6 @@\n +\n +\t\tif (!found)\n +\t\t\tunlink(path.buf);\n -+\n +\t}\n +}\n +\n 12: 4de4bfba64 = 12: ac5586a20f commit-graph: create options for split files\n 13: fe91ff5fca = 13: 548ec69d01 commit-graph: verify chains with --shallow mode\n 14: ca41bf08d0 = 14: 6084bbd164 commit-graph: clean up chains after flattened write\n\n-- \ngitgitgadget\n"},{"id":"376752","messageId":"d0dc154a27c80a8113ff47eb60a6b6f98fbeec91.1559830527.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 02/14] commit-graph: prepare for commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-06T14:15:30Z","receivedAt":"2019-06-06T14:15:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTo prepare for a chain of commit-graph files, augment the\ncommit_graph struct to point to a base commit_graph. As we load\ncommits from the graph, we may actually want to read from a base\nfile according to the graph position.\n\nThe \"graph position\" of a commit is given by concatenating the\nlexicographic commit orders from each of the commit-graph files in\nthe chain. This means that we must distinguish two values:\n\n * lexicographic index : the position within the lexicographic\n order in a single commit-graph file.\n\n * graph position: the posiiton within the concatenated order\n of multiple commit-graph files\n\nGiven the lexicographic index of a commit in a graph, we can\ncompute the graph position by adding the number of commits in\nthe lower-level graphs. To find the lexicographic index of\na commit, we subtract the number of commits in lower-level graphs.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 74 ++++++++++++++++++++++++++++++++++++++++++++------\n commit-graph.h | 3 ++\n 2 files changed, 69 insertions(+), 8 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 7723156964..3afedcd7f5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -371,6 +371,25 @@ static int bsearch_graph(struct commit_graph *g, struct object_id *oid, uint32_t\n \t\t\t g->chunk_oid_lookup, g->hash_len, pos);\n }\n \n+static void load_oid_from_graph(struct commit_graph *g, int pos, struct object_id *oid)\n+{\n+\tuint32_t lex_index;\n+\n+\tif (!g)\n+\t\tBUG(\"NULL commit-graph\");\n+\n+\twhile (pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tif (pos >= g->num_commits + g->num_commits_in_base)\n+\t\tBUG(\"position %d is beyond the scope of this commit-graph (%d local + %d base commits)\",\n+\t\t pos, g->num_commits, g->num_commits_in_base);\n+\n+\tlex_index = pos - g->num_commits_in_base;\n+\n+\thashcpy(oid->hash, g->chunk_oid_lookup + g->hash_len * lex_index);\n+}\n+\n static struct commit_list **insert_parent_or_die(struct repository *r,\n \t\t\t\t\t\t struct commit_graph *g,\n \t\t\t\t\t\t uint64_t pos,\n@@ -379,10 +398,10 @@ static struct commit_list **insert_parent_or_die(struct repository *r,\n \tstruct commit *c;\n \tstruct object_id oid;\n \n-\tif (pos >= g->num_commits)\n+\tif (pos >= g->num_commits + g->num_commits_in_base)\n \t\tdie(\"invalid parent position %\"PRIu64, pos);\n \n-\thashcpy(oid.hash, g->chunk_oid_lookup + g->hash_len * pos);\n+\tload_oid_from_graph(g, pos, &oid);\n \tc = lookup_commit(r, &oid);\n \tif (!c)\n \t\tdie(_(\"could not find commit %s\"), oid_to_hex(&oid));\n@@ -392,7 +411,14 @@ static struct commit_list **insert_parent_or_die(struct repository *r,\n \n static void fill_commit_graph_info(struct commit *item, struct commit_graph *g, uint32_t pos)\n {\n-\tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n+\tconst unsigned char *commit_data;\n+\tuint32_t lex_index;\n+\n+\twhile (pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tlex_index = pos - g->num_commits_in_base;\n+\tcommit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * lex_index;\n \titem->graph_pos = pos;\n \titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n }\n@@ -405,10 +431,26 @@ static int fill_commit_in_graph(struct repository *r,\n \tuint32_t *parent_data_ptr;\n \tuint64_t date_low, date_high;\n \tstruct commit_list **pptr;\n-\tconst unsigned char *commit_data = g->chunk_commit_data + (g->hash_len + 16) * pos;\n+\tconst unsigned char *commit_data;\n+\tuint32_t lex_index;\n \n-\titem->object.parsed = 1;\n+\twhile (pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tif (pos >= g->num_commits + g->num_commits_in_base)\n+\t\tBUG(\"position %d is beyond the scope of this commit-graph (%d local + %d base commits)\",\n+\t\t pos, g->num_commits, g->num_commits_in_base);\n+\n+\t/*\n+\t * Store the \"full\" position, but then use the\n+\t * \"local\" position for the rest of the calculation.\n+\t */\n \titem->graph_pos = pos;\n+\tlex_index = pos - g->num_commits_in_base;\n+\n+\tcommit_data = g->chunk_commit_data + (g->hash_len + 16) * lex_index;\n+\n+\titem->object.parsed = 1;\n \n \titem->maybe_tree = NULL;\n \n@@ -452,7 +494,18 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \t\t*pos = item->graph_pos;\n \t\treturn 1;\n \t} else {\n-\t\treturn bsearch_graph(g, &(item->object.oid), pos);\n+\t\tstruct commit_graph *cur_g = g;\n+\t\tuint32_t lex_index;\n+\n+\t\twhile (cur_g && !bsearch_graph(cur_g, &(item->object.oid), &lex_index))\n+\t\t\tcur_g = cur_g->base_graph;\n+\n+\t\tif (cur_g) {\n+\t\t\t*pos = lex_index + cur_g->num_commits_in_base;\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\treturn 0;\n \t}\n }\n \n@@ -492,8 +545,13 @@ static struct tree *load_tree_for_commit(struct repository *r,\n \t\t\t\t\t struct commit *c)\n {\n \tstruct object_id oid;\n-\tconst unsigned char *commit_data = g->chunk_commit_data +\n-\t\t\t\t\t GRAPH_DATA_WIDTH * (c->graph_pos);\n+\tconst unsigned char *commit_data;\n+\n+\twhile (c->graph_pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tcommit_data = g->chunk_commit_data +\n+\t\t\tGRAPH_DATA_WIDTH * (c->graph_pos - g->num_commits_in_base);\n \n \thashcpy(oid.hash, commit_data);\n \tc->maybe_tree = lookup_tree(r, &oid);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 70f4caf0c7..f9fe32ebe3 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -48,6 +48,9 @@ struct commit_graph {\n \tuint32_t num_commits;\n \tstruct object_id oid;\n \n+\tuint32_t num_commits_in_base;\n+\tstruct commit_graph *base_graph;\n+\n \tconst uint32_t *chunk_oid_fanout;\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_commit_data;\n-- \ngitgitgadget\n\n"},{"id":"376753","messageId":"f35b04224abf188c265ed0395cd3dfcf6eb6ac5b.1559830527.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 03/14] commit-graph: rename commit_compare to oid_compare","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-06T14:15:31Z","receivedAt":"2019-06-06T14:15:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe helper function commit_compare() actually compares object_id\nstructs, not commits. A future change to commit-graph.c will need\nto sort commit structs, so rename this function in advance.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 3afedcd7f5..e2f438f6a3 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -761,7 +761,7 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n \t}\n }\n \n-static int commit_compare(const void *_a, const void *_b)\n+static int oid_compare(const void *_a, const void *_b)\n {\n \tconst struct object_id *a = (const struct object_id *)_a;\n \tconst struct object_id *b = (const struct object_id *)_b;\n@@ -1030,7 +1030,7 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n \t\t\t_(\"Counting distinct commits in commit graph\"),\n \t\t\tctx->oids.nr);\n \tdisplay_progress(ctx->progress, 0); /* TODO: Measure QSORT() progress */\n-\tQSORT(ctx->oids.list, ctx->oids.nr, commit_compare);\n+\tQSORT(ctx->oids.list, ctx->oids.nr, oid_compare);\n \n \tfor (i = 1; i < ctx->oids.nr; i++) {\n \t\tdisplay_progress(ctx->progress, i + 1);\n-- \ngitgitgadget\n\n"},{"id":"376754","messageId":"b184919255bfc9e0d57e1a5ca12fb76957879c61.1559830527.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 01/14] commit-graph: document commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-06T14:15:29Z","receivedAt":"2019-06-06T14:15:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd a basic description of commit-graph chains. More details about the\nfeature will be added as we add functionality. This introduction gives a\nhigh-level overview to the goals of the feature and the basic layout of\ncommit-graph chains.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 59 ++++++++++++++++++++++++\n 1 file changed, 59 insertions(+)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex fb53341d5e..1dca3bd8fe 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -127,6 +127,65 @@ Design Details\n helpful for these clones, anyway. The commit-graph will not be read or\n written when shallow commits are present.\n \n+Commit Graphs Chains\n+--------------------\n+\n+Typically, repos grow with near-constant velocity (commits per day). Over time,\n+the number of commits added by a fetch operation is much smaller than the\n+number of commits in the full history. By creating a \"chain\" of commit-graphs,\n+we enable fast writes of new commit data without rewriting the entire commit\n+history -- at least, most of the time.\n+\n+## File Layout\n+\n+A commit-graph chain uses multiple files, and we use a fixed naming convention\n+to organize these files. Each commit-graph file has a name\n+`$OBJDIR/info/commit-graphs/graph-{hash}.graph` where `{hash}` is the hex-\n+valued hash stored in the footer of that file (which is a hash of the file's\n+contents before that hash). For a chain of commit-graph files, a plain-text\n+file at `$OBJDIR/info/commit-graphs/commit-graph-chain` contains the\n+hashes for the files in order from \"lowest\" to \"highest\".\n+\n+For example, if the `commit-graph-chain` file contains the lines\n+\n+```\n+\t{hash0}\n+\t{hash1}\n+\t{hash2}\n+```\n+\n+then the commit-graph chain looks like the following diagram:\n+\n+ +-----------------------+\n+ | graph-{hash2}.graph |\n+ +-----------------------+\n+\t |\n+ +-----------------------+\n+ | |\n+ | graph-{hash1}.graph |\n+ | |\n+ +-----------------------+\n+\t |\n+ +-----------------------+\n+ | |\n+ | |\n+ | |\n+ | graph-{hash0}.graph |\n+ | |\n+ | |\n+ | |\n+ +-----------------------+\n+\n+Let X0 be the number of commits in `graph-{hash0}.graph`, X1 be the number of\n+commits in `graph-{hash1}.graph`, and X2 be the number of commits in\n+`graph-{hash2}.graph`. If a commit appears in position i in `graph-{hash2}.graph`,\n+then we interpret this as being the commit in position (X0 + X1 + i), and that\n+will be used as its \"graph position\". The commits in `graph-{hash2}.graph` use these\n+positions to refer to their parents, which may be in `graph-{hash1}.graph` or\n+`graph-{hash0}.graph`. We can navigate to an arbitrary commit in position j by checking\n+its containment in the intervals [0, X0), [X0, X0 + X1), [X0 + X1, X0 + X1 +\n+X2).\n+\n Related Links\n -------------\n [0] https://bugs.chromium.org/p/git/issues/detail?id=8\n-- \ngitgitgadget\n\n"},{"id":"376755","messageId":"e65f9e841d577c13d26c1737ebb8a2cf65713be9.1559830527.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 06/14] commit-graph: rearrange chunk count logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-06T14:15:34Z","receivedAt":"2019-06-06T14:15:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe number of chunks in a commit-graph file can change depending on\nwhether we need the Extra Edges Chunk. We are going to add more optional\nchunks, and it will be helpful to rearrange this logic around the chunk\ncount before doing so.\n\nSpecifically, we need to finalize the number of chunks before writing\nthe commit-graph header. Further, we also need to fill out the chunk\nlookup table dynamically and using \"num_chunks\" as we add optional\nchunks is useful for adding optional chunks in the future.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 35 +++++++++++++++++++++--------------\n 1 file changed, 21 insertions(+), 14 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 909c841db5..80df6d6d9d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1206,7 +1206,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tuint64_t chunk_offsets[5];\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n-\tint num_chunks = ctx->num_extra_edges ? 4 : 3;\n+\tint num_chunks = 3;\n \n \tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n \tif (safe_create_leading_directories(ctx->graph_name)) {\n@@ -1219,27 +1219,34 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \n-\thashwrite_be32(f, GRAPH_SIGNATURE);\n-\n-\thashwrite_u8(f, GRAPH_VERSION);\n-\thashwrite_u8(f, oid_version());\n-\thashwrite_u8(f, num_chunks);\n-\thashwrite_u8(f, 0); /* unused padding byte */\n-\n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n \tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n-\tif (ctx->num_extra_edges)\n-\t\tchunk_ids[3] = GRAPH_CHUNKID_EXTRAEDGES;\n-\telse\n-\t\tchunk_ids[3] = 0;\n-\tchunk_ids[4] = 0;\n+\tif (ctx->num_extra_edges) {\n+\t\tchunk_ids[num_chunks] = GRAPH_CHUNKID_EXTRAEDGES;\n+\t\tnum_chunks++;\n+\t}\n+\n+\tchunk_ids[num_chunks] = 0;\n \n \tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n \tchunk_offsets[2] = chunk_offsets[1] + hashsz * ctx->commits.nr;\n \tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * ctx->commits.nr;\n-\tchunk_offsets[4] = chunk_offsets[3] + 4 * ctx->num_extra_edges;\n+\n+\tnum_chunks = 3;\n+\tif (ctx->num_extra_edges) {\n+\t\tchunk_offsets[num_chunks + 1] = chunk_offsets[num_chunks] +\n+\t\t\t\t\t\t4 * ctx->num_extra_edges;\n+\t\tnum_chunks++;\n+\t}\n+\n+\thashwrite_be32(f, GRAPH_SIGNATURE);\n+\n+\thashwrite_u8(f, GRAPH_VERSION);\n+\thashwrite_u8(f, oid_version());\n+\thashwrite_u8(f, num_chunks);\n+\thashwrite_u8(f, 0);\n \n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tuint32_t chunk_write[3];\n-- \ngitgitgadget\n\n"},{"id":"376756","messageId":"ca670536dfe08277e953b2ef5f017dc32c1581d5.1559830527.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 04/14] commit-graph: load commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-06T14:15:32Z","receivedAt":"2019-06-06T14:15:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nPrepare the logic for reading a chain of commit-graphs.\n\nFirst, look for a file at $OBJDIR/info/commit-graph. If it exists,\nthen use that file and stop.\n\nNext, look for the chain file at $OBJDIR/info/commit-graphs/commit-graph-chain.\nIf this file exists, then load the hash values as line-separated values in that\nfile and load $OBJDIR/info/commit-graphs/graph-{hash[i]}.graph for each hash[i]\nin that file. The file is given in order, so the first hash corresponds to the\n\"base\" file and the final hash corresponds to the \"tip\" file.\n\nThis implementation assumes that all of the graph-{hash}.graph files are in\nthe same object directory as the commit-graph-chain file. This will be updated\nin a future change. This change is purposefully simple so we can isolate the\ndifferent concerns.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 114 ++++++++++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 108 insertions(+), 6 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex e2f438f6a3..3ed930159e 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -45,6 +45,19 @@ char *get_commit_graph_filename(const char *obj_dir)\n \treturn xstrfmt(\"%s/info/commit-graph\", obj_dir);\n }\n \n+static char *get_split_graph_filename(const char *obj_dir,\n+\t\t\t\t const char *oid_hex)\n+{\n+\treturn xstrfmt(\"%s/info/commit-graphs/graph-%s.graph\",\n+\t\t obj_dir,\n+\t\t oid_hex);\n+}\n+\n+static char *get_chain_filename(const char *obj_dir)\n+{\n+\treturn xstrfmt(\"%s/info/commit-graphs/commit-graph-chain\", obj_dir);\n+}\n+\n static uint8_t oid_version(void)\n {\n \treturn 1;\n@@ -286,18 +299,107 @@ static struct commit_graph *load_commit_graph_one(const char *graph_file)\n \treturn load_commit_graph_one_fd_st(fd, &st);\n }\n \n+static struct commit_graph *load_commit_graph_v1(struct repository *r, const char *obj_dir)\n+{\n+\tchar *graph_name = get_commit_graph_filename(obj_dir);\n+\tstruct commit_graph *g = load_commit_graph_one(graph_name);\n+\tfree(graph_name);\n+\n+\treturn g;\n+}\n+\n+static int add_graph_to_chain(struct commit_graph *g,\n+\t\t\t struct commit_graph *chain,\n+\t\t\t struct object_id *oids,\n+\t\t\t int n)\n+{\n+\tstruct commit_graph *cur_g = chain;\n+\n+\twhile (n) {\n+\t\tn--;\n+\t\tcur_g = cur_g->base_graph;\n+\t}\n+\n+\tg->base_graph = chain;\n+\n+\tif (chain)\n+\t\tg->num_commits_in_base = chain->num_commits + chain->num_commits_in_base;\n+\n+\treturn 1;\n+}\n+\n+static struct commit_graph *load_commit_graph_chain(struct repository *r, const char *obj_dir)\n+{\n+\tstruct commit_graph *graph_chain = NULL;\n+\tstruct strbuf line = STRBUF_INIT;\n+\tstruct stat st;\n+\tstruct object_id *oids;\n+\tint i = 0, valid = 1;\n+\tchar *chain_name = get_chain_filename(obj_dir);\n+\tFILE *fp;\n+\n+\tif (stat(chain_name, &st)) {\n+\t\tfree(chain_name);\n+\t\treturn NULL;\n+\t}\n+\n+\tif (st.st_size <= the_hash_algo->hexsz) {\n+\t\tfree(chain_name);\n+\t\treturn NULL;\n+\t}\n+\n+\tfp = fopen(chain_name, \"r\");\n+\tfree(chain_name);\n+\n+\tif (!fp)\n+\t\treturn NULL;\n+\n+\toids = xcalloc(st.st_size / (the_hash_algo->hexsz + 1), sizeof(struct object_id));\n+\n+\twhile (strbuf_getline_lf(&line, fp) != EOF && valid) {\n+\t\tchar *graph_name;\n+\t\tstruct commit_graph *g;\n+\n+\t\tif (get_oid_hex(line.buf, &oids[i])) {\n+\t\t\twarning(_(\"invalid commit-graph chain: line '%s' not a hash\"),\n+\t\t\t\tline.buf);\n+\t\t\tvalid = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tgraph_name = get_split_graph_filename(obj_dir, line.buf);\n+\t\tg = load_commit_graph_one(graph_name);\n+\t\tfree(graph_name);\n+\n+\t\tif (g && add_graph_to_chain(g, graph_chain, oids, i))\n+\t\t\tgraph_chain = g;\n+\t\telse\n+\t\t\tvalid = 0;\n+\t}\n+\n+\tfree(oids);\n+\tfclose(fp);\n+\n+\treturn graph_chain;\n+}\n+\n+static struct commit_graph *read_commit_graph_one(struct repository *r, const char *obj_dir)\n+{\n+\tstruct commit_graph *g = load_commit_graph_v1(r, obj_dir);\n+\n+\tif (!g)\n+\t\tg = load_commit_graph_chain(r, obj_dir);\n+\n+\treturn g;\n+}\n+\n static void prepare_commit_graph_one(struct repository *r, const char *obj_dir)\n {\n-\tchar *graph_name;\n \n \tif (r->objects->commit_graph)\n \t\treturn;\n \n-\tgraph_name = get_commit_graph_filename(obj_dir);\n-\tr->objects->commit_graph =\n-\t\tload_commit_graph_one(graph_name);\n-\n-\tFREE_AND_NULL(graph_name);\n+\tr->objects->commit_graph = read_commit_graph_one(r, obj_dir);\n }\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"376757","messageId":"fe0aa343cdb9c1fd5c087b1e8da70247237de01f.1559830527.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 07/14] commit-graph: write commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-06T14:15:35Z","receivedAt":"2019-06-06T14:15:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nExtend write_commit_graph() to write a commit-graph chain when given the\nCOMMIT_GRAPH_SPLIT flag.\n\nThis implementation is purposefully simplistic in how it creates a new\nchain. The commits not already in the chain are added to a new tip\ncommit-graph file.\n\nMuch of the logic around writing a graph-{hash}.graph file and updating\nthe commit-graph-chain file is the same as the commit-graph file case.\nHowever, there are several places where we need to do some extra logic\nin the split case.\n\nTrack the list of graph filenames before and after the planned write.\nThis will be more important when we start merging graph files, but it\nalso allows us to upgrade our commit-graph file to the appropriate\ngraph-{hash}.graph file when we upgrade to a chain of commit-graphs.\n\nNote that we use the eighth byte of the commit-graph header to store the\nnumber of base graph files. This determines the length of the base\ngraphs chunk.\n\nA subtle change of behavior with the new logic is that we do not write a\ncommit-graph if we our commit list is empty. This extends to the typical\ncase, which is reflected in t5318-commit-graph.sh.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 286 ++++++++++++++++++++++++++++++++++++++--\n commit-graph.h | 2 +\n t/t5318-commit-graph.sh | 2 +-\n 3 files changed, 278 insertions(+), 12 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 80df6d6d9d..7d3e001479 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -300,12 +300,18 @@ static struct commit_graph *load_commit_graph_one(const char *graph_file)\n \n \tstruct stat st;\n \tint fd;\n+\tstruct commit_graph *g;\n \tint open_ok = open_commit_graph(graph_file, &fd, &st);\n \n \tif (!open_ok)\n \t\treturn NULL;\n \n-\treturn load_commit_graph_one_fd_st(fd, &st);\n+\tg = load_commit_graph_one_fd_st(fd, &st);\n+\n+\tif (g)\n+\t\tg->filename = xstrdup(graph_file);\n+\n+\treturn g;\n }\n \n static struct commit_graph *load_commit_graph_v1(struct repository *r, const char *obj_dir)\n@@ -723,8 +729,19 @@ struct write_commit_graph_context {\n \tstruct progress *progress;\n \tint progress_done;\n \tuint64_t progress_cnt;\n+\n+\tchar *base_graph_name;\n+\tint num_commit_graphs_before;\n+\tint num_commit_graphs_after;\n+\tchar **commit_graph_filenames_before;\n+\tchar **commit_graph_filenames_after;\n+\tchar **commit_graph_hash_after;\n+\tuint32_t new_num_commits_in_base;\n+\tstruct commit_graph *new_base_graph;\n+\n \tunsigned append:1,\n-\t\t report_progress:1;\n+\t\t report_progress:1,\n+\t\t split:1;\n };\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n@@ -794,6 +811,16 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t commit_to_sha1);\n \n+\t\t\tif (edge_value >= 0)\n+\t\t\t\tedge_value += ctx->new_num_commits_in_base;\n+\t\t\telse {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (find_commit_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n+\t\t\t\t\tedge_value = pos;\n+\t\t\t}\n+\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n \t\t\t\t oid_to_hex(&parent->item->object.oid),\n@@ -814,6 +841,17 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\t\t\t ctx->commits.list,\n \t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t commit_to_sha1);\n+\n+\t\t\tif (edge_value >= 0)\n+\t\t\t\tedge_value += ctx->new_num_commits_in_base;\n+\t\t\telse {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (find_commit_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n+\t\t\t\t\tedge_value = pos;\n+\t\t\t}\n+\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n \t\t\t\t oid_to_hex(&parent->item->object.oid),\n@@ -871,6 +909,16 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n \t\t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t\t commit_to_sha1);\n \n+\t\t\tif (edge_value >= 0)\n+\t\t\t\tedge_value += ctx->new_num_commits_in_base;\n+\t\t\telse {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (find_commit_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n+\t\t\t\t\tedge_value = pos;\n+\t\t\t}\n+\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n \t\t\t\t oid_to_hex(&parent->item->object.oid),\n@@ -962,7 +1010,13 @@ static void close_reachable(struct write_commit_graph_context *ctx)\n \t\tdisplay_progress(ctx->progress, i + 1);\n \t\tcommit = lookup_commit(ctx->r, &ctx->oids.list[i]);\n \n-\t\tif (commit && !parse_commit_no_graph(commit))\n+\t\tif (!commit)\n+\t\t\tcontinue;\n+\t\tif (ctx->split) {\n+\t\t\tif (!parse_commit(commit) &&\n+\t\t\t commit->graph_pos == COMMIT_NOT_FROM_GRAPH)\n+\t\t\t\tadd_missing_parents(ctx, commit);\n+\t\t} else if (!parse_commit_no_graph(commit))\n \t\t\tadd_missing_parents(ctx, commit);\n \t}\n \tstop_progress(&ctx->progress);\n@@ -1158,8 +1212,16 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n \n \tfor (i = 1; i < ctx->oids.nr; i++) {\n \t\tdisplay_progress(ctx->progress, i + 1);\n-\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n+\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i])) {\n+\t\t\tif (ctx->split) {\n+\t\t\t\tstruct commit *c = lookup_commit(ctx->r, &ctx->oids.list[i]);\n+\n+\t\t\t\tif (!c || c->graph_pos != COMMIT_NOT_FROM_GRAPH)\n+\t\t\t\t\tcontinue;\n+\t\t\t}\n+\n \t\t\tcount_distinct++;\n+\t\t}\n \t}\n \tstop_progress(&ctx->progress);\n \n@@ -1182,7 +1244,13 @@ static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n \t\tif (i > 0 && oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n \t\t\tcontinue;\n \n+\t\tALLOC_GROW(ctx->commits.list, ctx->commits.nr + 1, ctx->commits.alloc);\n \t\tctx->commits.list[ctx->commits.nr] = lookup_commit(ctx->r, &ctx->oids.list[i]);\n+\n+\t\tif (ctx->split &&\n+\t\t ctx->commits.list[ctx->commits.nr]->graph_pos != COMMIT_NOT_FROM_GRAPH)\n+\t\t\tcontinue;\n+\n \t\tparse_commit_no_graph(ctx->commits.list[ctx->commits.nr]);\n \n \t\tfor (parent = ctx->commits.list[ctx->commits.nr]->parents;\n@@ -1197,18 +1265,86 @@ static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n+static int write_graph_chunk_base_1(struct hashfile *f,\n+\t\t\t\t struct commit_graph *g)\n+{\n+\tint num = 0;\n+\n+\tif (!g)\n+\t\treturn 0;\n+\n+\tnum = write_graph_chunk_base_1(f, g->base_graph);\n+\thashwrite(f, g->oid.hash, the_hash_algo->rawsz);\n+\treturn num + 1;\n+}\n+\n+static int write_graph_chunk_base(struct hashfile *f,\n+\t\t\t\t struct write_commit_graph_context *ctx)\n+{\n+\tint num = write_graph_chunk_base_1(f, ctx->new_base_graph);\n+\n+\tif (num != ctx->num_commit_graphs_after - 1) {\n+\t\terror(_(\"failed to write correct number of base graph ids\"));\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void init_commit_graph_chain(struct write_commit_graph_context *ctx)\n+{\n+\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\tuint32_t i;\n+\n+\tctx->new_base_graph = g;\n+\tctx->base_graph_name = xstrdup(g->filename);\n+\tctx->new_num_commits_in_base = g->num_commits + g->num_commits_in_base;\n+\n+\tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n+\n+\tALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);\n+\tALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);\n+\n+\tfor (i = 0; i < ctx->num_commit_graphs_before - 1; i++)\n+\t\tctx->commit_graph_filenames_after[i] = xstrdup(ctx->commit_graph_filenames_before[i]);\n+\n+\tif (ctx->num_commit_graphs_before)\n+\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_before - 1] =\n+\t\t\tget_split_graph_filename(ctx->obj_dir, oid_to_hex(&g->oid));\n+\n+\ti = ctx->num_commit_graphs_before - 1;\n+\n+\twhile (g) {\n+\t\tctx->commit_graph_hash_after[i] = xstrdup(oid_to_hex(&g->oid));\n+\t\ti--;\n+\t\tg = g->base_graph;\n+\t}\n+}\n+\n static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n+\tint fd;\n \tstruct hashfile *f;\n \tstruct lock_file lk = LOCK_INIT;\n-\tuint32_t chunk_ids[5];\n-\tuint64_t chunk_offsets[5];\n+\tuint32_t chunk_ids[6];\n+\tuint64_t chunk_offsets[6];\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tint num_chunks = 3;\n+\tstruct object_id file_hash;\n+\n+\tif (ctx->split) {\n+\t\tstruct strbuf tmp_file = STRBUF_INIT;\n+\n+\t\tstrbuf_addf(&tmp_file,\n+\t\t\t \"%s/info/commit-graphs/tmp_graph_XXXXXX\",\n+\t\t\t ctx->obj_dir);\n+\t\tctx->graph_name = strbuf_detach(&tmp_file, NULL);\n+\t} else {\n+\t\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n+\t}\n \n-\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n \tif (safe_create_leading_directories(ctx->graph_name)) {\n \t\tUNLEAK(ctx->graph_name);\n \t\terror(_(\"unable to create leading directories of %s\"),\n@@ -1216,8 +1352,23 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\treturn errno;\n \t}\n \n-\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n-\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n+\tif (ctx->split) {\n+\t\tchar *lock_name = get_chain_filename(ctx->obj_dir);\n+\n+\t\thold_lock_file_for_update(&lk, lock_name, LOCK_DIE_ON_ERROR);\n+\n+\t\tfd = git_mkstemp_mode(ctx->graph_name, 0444);\n+\t\tif (fd < 0) {\n+\t\t\terror(_(\"unable to create '%s'\"), ctx->graph_name);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tf = hashfd(fd, ctx->graph_name);\n+\t} else {\n+\t\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n+\t\tfd = lk.tempfile->fd;\n+\t\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n+\t}\n \n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n@@ -1226,6 +1377,10 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\tchunk_ids[num_chunks] = GRAPH_CHUNKID_EXTRAEDGES;\n \t\tnum_chunks++;\n \t}\n+\tif (ctx->num_commit_graphs_after > 1) {\n+\t\tchunk_ids[num_chunks] = GRAPH_CHUNKID_BASE;\n+\t\tnum_chunks++;\n+\t}\n \n \tchunk_ids[num_chunks] = 0;\n \n@@ -1240,13 +1395,18 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\t\t\t\t4 * ctx->num_extra_edges;\n \t\tnum_chunks++;\n \t}\n+\tif (ctx->num_commit_graphs_after > 1) {\n+\t\tchunk_offsets[num_chunks + 1] = chunk_offsets[num_chunks] +\n+\t\t\t\t\t\thashsz * (ctx->num_commit_graphs_after - 1);\n+\t\tnum_chunks++;\n+\t}\n \n \thashwrite_be32(f, GRAPH_SIGNATURE);\n \n \thashwrite_u8(f, GRAPH_VERSION);\n \thashwrite_u8(f, oid_version());\n \thashwrite_u8(f, num_chunks);\n-\thashwrite_u8(f, 0);\n+\thashwrite_u8(f, ctx->num_commit_graphs_after - 1);\n \n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tuint32_t chunk_write[3];\n@@ -1272,11 +1432,67 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \twrite_graph_chunk_data(f, hashsz, ctx);\n \tif (ctx->num_extra_edges)\n \t\twrite_graph_chunk_extra_edges(f, ctx);\n+\tif (ctx->num_commit_graphs_after > 1 &&\n+\t write_graph_chunk_base(f, ctx)) {\n+\t\treturn -1;\n+\t}\n \tstop_progress(&ctx->progress);\n \tstrbuf_release(&progress_title);\n \n+\tif (ctx->split && ctx->base_graph_name && ctx->num_commit_graphs_after > 1) {\n+\t\tchar *new_base_hash = xstrdup(oid_to_hex(&ctx->new_base_graph->oid));\n+\t\tchar *new_base_name = get_split_graph_filename(ctx->obj_dir, new_base_hash);\n+\n+\t\tfree(ctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2]);\n+\t\tfree(ctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 2]);\n+\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2] = new_base_name;\n+\t\tctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 2] = new_base_hash;\n+\t}\n+\n \tclose_commit_graph(ctx->r);\n-\tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n+\tfinalize_hashfile(f, file_hash.hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n+\n+\tif (ctx->split) {\n+\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n+\t\tchar *final_graph_name;\n+\t\tint result;\n+\n+\t\tclose(fd);\n+\n+\t\tif (!chainf) {\n+\t\t\terror(_(\"unable to open commit-graph chain file\"));\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (ctx->base_graph_name) {\n+\t\t\tresult = rename(ctx->base_graph_name,\n+\t\t\t\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2]);\n+\n+\t\t\tif (result) {\n+\t\t\t\terror(_(\"failed to rename base commit-graph file\"));\n+\t\t\t\treturn -1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tchar *graph_name = get_commit_graph_filename(ctx->obj_dir);\n+\t\t\tunlink(graph_name);\n+\t\t}\n+\n+\t\tctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 1] = xstrdup(oid_to_hex(&file_hash));\n+\t\tfinal_graph_name = get_split_graph_filename(ctx->obj_dir,\n+\t\t\t\t\tctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 1]);\n+\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 1] = final_graph_name;\n+\n+\t\tresult = rename(ctx->graph_name, final_graph_name);\n+\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++)\n+\t\t\tfprintf(lk.tempfile->fp, \"%s\\n\", ctx->commit_graph_hash_after[i]);\n+\n+\t\tif (result) {\n+\t\t\terror(_(\"failed to rename temporary commit-graph file\"));\n+\t\t\treturn -1;\n+\t\t}\n+\t}\n+\n \tcommit_lock_file(&lk);\n \n \treturn 0;\n@@ -1299,6 +1515,30 @@ int write_commit_graph(const char *obj_dir,\n \tctx->obj_dir = obj_dir;\n \tctx->append = flags & COMMIT_GRAPH_APPEND ? 1 : 0;\n \tctx->report_progress = flags & COMMIT_GRAPH_PROGRESS ? 1 : 0;\n+\tctx->split = flags & COMMIT_GRAPH_SPLIT ? 1 : 0;\n+\n+\tif (ctx->split) {\n+\t\tstruct commit_graph *g;\n+\t\tprepare_commit_graph(ctx->r);\n+\n+\t\tg = ctx->r->objects->commit_graph;\n+\n+\t\twhile (g) {\n+\t\t\tctx->num_commit_graphs_before++;\n+\t\t\tg = g->base_graph;\n+\t\t}\n+\n+\t\tif (ctx->num_commit_graphs_before) {\n+\t\t\tALLOC_ARRAY(ctx->commit_graph_filenames_before, ctx->num_commit_graphs_before);\n+\t\t\ti = ctx->num_commit_graphs_before;\n+\t\t\tg = ctx->r->objects->commit_graph;\n+\n+\t\t\twhile (g) {\n+\t\t\t\tctx->commit_graph_filenames_before[--i] = xstrdup(g->filename);\n+\t\t\t\tg = g->base_graph;\n+\t\t\t}\n+\t\t}\n+\t}\n \n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n@@ -1353,6 +1593,14 @@ int write_commit_graph(const char *obj_dir,\n \t\tgoto cleanup;\n \t}\n \n+\tif (!ctx->commits.nr)\n+\t\tgoto cleanup;\n+\n+\tif (ctx->split)\n+\t\tinit_commit_graph_chain(ctx);\n+\telse\n+\t\tctx->num_commit_graphs_after = 1;\n+\n \tcompute_generation_numbers(ctx);\n \n \tres = write_commit_graph_file(ctx);\n@@ -1361,6 +1609,21 @@ int write_commit_graph(const char *obj_dir,\n \tfree(ctx->graph_name);\n \tfree(ctx->commits.list);\n \tfree(ctx->oids.list);\n+\n+\tif (ctx->commit_graph_filenames_after) {\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++) {\n+\t\t\tfree(ctx->commit_graph_filenames_after[i]);\n+\t\t\tfree(ctx->commit_graph_hash_after[i]);\n+\t\t}\n+\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_before; i++)\n+\t\t\tfree(ctx->commit_graph_filenames_before[i]);\n+\n+\t\tfree(ctx->commit_graph_filenames_after);\n+\t\tfree(ctx->commit_graph_filenames_before);\n+\t\tfree(ctx->commit_graph_hash_after);\n+\t}\n+\n \tfree(ctx);\n \n \treturn res;\n@@ -1548,5 +1811,6 @@ void free_commit_graph(struct commit_graph *g)\n \t\tg->data = NULL;\n \t\tclose(g->graph_fd);\n \t}\n+\tfree(g->filename);\n \tfree(g);\n }\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 80f4917ddb..5c48c4f66a 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -47,6 +47,7 @@ struct commit_graph {\n \tunsigned char num_chunks;\n \tuint32_t num_commits;\n \tstruct object_id oid;\n+\tchar *filename;\n \n \tuint32_t num_commits_in_base;\n \tstruct commit_graph *base_graph;\n@@ -71,6 +72,7 @@ int generation_numbers_enabled(struct repository *r);\n \n #define COMMIT_GRAPH_APPEND (1 << 0)\n #define COMMIT_GRAPH_PROGRESS (1 << 1)\n+#define COMMIT_GRAPH_SPLIT (1 << 2)\n \n int write_commit_graph_reachable(const char *obj_dir, unsigned int flags);\n int write_commit_graph(const char *obj_dir,\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 3b6fd0d728..063f906b3e 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -20,7 +20,7 @@ test_expect_success 'verify graph with no graph file' '\n test_expect_success 'write graph with no packs' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write --object-dir . &&\n-\ttest_path_is_file info/commit-graph\n+\ttest_path_is_missing info/commit-graph\n '\n \n test_expect_success 'close with correct error on bad input' '\n-- \ngitgitgadget\n\n"},{"id":"376758","messageId":"df44cbc1bf6e18c3682652c2de69de7a9caf5fc2.1559830527.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 05/14] commit-graph: add base graphs chunk","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-06T14:15:33Z","receivedAt":"2019-06-06T14:15:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTo quickly verify a commit-graph chain is valid on load, we will\nread from the new \"Base Graphs Chunk\" of each file in the chain.\nThis will prevent accidentally loading incorrect data from manually\nediting the commit-graph-chain file or renaming graph-{hash}.graph\nfiles.\n\nThe commit_graph struct already had an object_id struct \"oid\", but\nit was never initialized or used. Add a line to read the hash from\nthe end of the commit-graph file and into the oid member.\n\nSigned-off-by: Derrick Stolee \n---\n .../technical/commit-graph-format.txt | 11 ++++++++--\n commit-graph.c | 22 +++++++++++++++++++\n commit-graph.h | 1 +\n 3 files changed, 32 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex 16452a0504..a4f17441ae 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -44,8 +44,9 @@ HEADER:\n \n 1-byte number (C) of \"chunks\"\n \n- 1-byte (reserved for later use)\n- Current clients should ignore this value.\n+ 1-byte number (B) of base commit-graphs\n+ We infer the length (H*B) of the Base Graphs chunk\n+ from this value.\n \n CHUNK LOOKUP:\n \n@@ -92,6 +93,12 @@ CHUNK DATA:\n positions for the parents until reaching a value with the most-significant\n bit on. The other bits correspond to the position of the last parent.\n \n+ Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]\n+ This list of H-byte hashes describe a set of B commit-graph files that\n+ form a commit-graph chain. The graph position for the ith commit in this\n+ file's OID Lookup chunk is equal to i plus the number of commits in all\n+ base graphs. If B is non-zero, this chunk must exist.\n+\n TRAILER:\n \n \tH-byte HASH-checksum of all of the above.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 3ed930159e..909c841db5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -22,6 +22,7 @@\n #define GRAPH_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define GRAPH_CHUNKID_DATA 0x43444154 /* \"CDAT\" */\n #define GRAPH_CHUNKID_EXTRAEDGES 0x45444745 /* \"EDGE\" */\n+#define GRAPH_CHUNKID_BASE 0x42415345 /* \"BASE\" */\n \n #define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n \n@@ -262,6 +263,12 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t\t\telse\n \t\t\t\tgraph->chunk_extra_edges = data + chunk_offset;\n \t\t\tbreak;\n+\n+\t\tcase GRAPH_CHUNKID_BASE:\n+\t\t\tif (graph->chunk_base_graphs)\n+\t\t\t\tchunk_repeated = 1;\n+\t\t\telse\n+\t\t\t\tgraph->chunk_base_graphs = data + chunk_offset;\n \t\t}\n \n \t\tif (chunk_repeated) {\n@@ -280,6 +287,8 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t\tlast_chunk_offset = chunk_offset;\n \t}\n \n+\thashcpy(graph->oid.hash, graph->data + graph->data_len - graph->hash_len);\n+\n \tif (verify_commit_graph_lite(graph))\n \t\treturn NULL;\n \n@@ -315,8 +324,21 @@ static int add_graph_to_chain(struct commit_graph *g,\n {\n \tstruct commit_graph *cur_g = chain;\n \n+\tif (n && !g->chunk_base_graphs) {\n+\t\twarning(_(\"commit-graph has no base graphs chunk\"));\n+\t\treturn 0;\n+\t}\n+\n \twhile (n) {\n \t\tn--;\n+\n+\t\tif (!oideq(&oids[n], &cur_g->oid) ||\n+\t\t !hasheq(oids[n].hash, g->chunk_base_graphs + g->hash_len * n)) {\n+\t\t\twarning(_(\"commit-graph chain does not match\"));\n+\t\t\treturn 0;\n+\t\t}\n+\n+\n \t\tcur_g = cur_g->base_graph;\n \t}\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex f9fe32ebe3..80f4917ddb 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -55,6 +55,7 @@ struct commit_graph {\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_commit_data;\n \tconst unsigned char *chunk_extra_edges;\n+\tconst unsigned char *chunk_base_graphs;\n };\n \n struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st);\n-- \ngitgitgadget\n\n"},{"id":"376760","messageId":"d065758454979375aa934cc835f2e48db406c387.1559830527.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 09/14] commit-graph: merge commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-06T14:15:36Z","receivedAt":"2019-06-06T14:15:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen searching for a commit in a commit-graph chain of G graphs with N\ncommits, the search takes O(G log N) time. If we always add a new tip\ngraph with every write, the linear G term will start to dominate and\nslow the lookup process.\n\nTo keep lookups fast, but also keep most incremental writes fast, create\na strategy for merging levels of the commit-graph chain. The strategy is\ndetailed in the commit-graph design document, but is summarized by these\ntwo conditions:\n\n 1. If the number of commits we are adding is more than half the number\n of commits in the graph below, then merge with that graph.\n\n 2. If we are writing more than 64,000 commits into a single graph,\n then merge with all lower graphs.\n\nThe numeric values in the conditions above are currently constant, but\ncan become config options in a future update.\n\nAs we merge levels of the commit-graph chain, check that the commits\nstill exist in the repository. A garbage-collection operation may have\nremoved those commits from the object store and we do not want to\npersist them in the commit-graph chain. This is a non-issue if the\n'git gc' process wrote a new, single-level commit-graph file.\n\nAfter we merge levels, the old graph-{hash}.graph files are no longer\nreferenced by the commit-graph-chain file. We will expire these files in\na future change.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 80 ++++++++++\n commit-graph.c | 184 +++++++++++++++++++----\n t/t5323-split-commit-graph.sh | 13 ++\n 3 files changed, 244 insertions(+), 33 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex 1dca3bd8fe..d9c6253b0a 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -186,6 +186,86 @@ positions to refer to their parents, which may be in `graph-{hash1}.graph` or\n its containment in the intervals [0, X0), [X0, X0 + X1), [X0 + X1, X0 + X1 +\n X2).\n \n+Each commit-graph file (except the base, `graph-{hash0}.graph`) contains data\n+specifying the hashes of all files in the lower layers. In the above example,\n+`graph-{hash1}.graph` contains `{hash0}` while `graph-{hash2}.graph` contains\n+`{hash0}` and `{hash1}`.\n+\n+## Merging commit-graph files\n+\n+If we only added a new commit-graph file on every write, we would run into a\n+linear search problem through many commit-graph files. Instead, we use a merge\n+strategy to decide when the stack should collapse some number of levels.\n+\n+The diagram below shows such a collapse. As a set of new commits are added, it\n+is determined by the merge strategy that the files should collapse to\n+`graph-{hash1}`. Thus, the new commits, the commits in `graph-{hash2}` and\n+the commits in `graph-{hash1}` should be combined into a new `graph-{hash3}`\n+file.\n+\n+\t\t\t +---------------------+\n+\t\t\t | |\n+\t\t\t | (new commits) |\n+\t\t\t | |\n+\t\t\t +---------------------+\n+\t\t\t | |\n+ +-----------------------+ +---------------------+\n+ | graph-{hash2} |->| |\n+ +-----------------------+ +---------------------+\n+\t | | |\n+ +-----------------------+ +---------------------+\n+ | | | |\n+ | graph-{hash1} |->| |\n+ | | | |\n+ +-----------------------+ +---------------------+\n+\t | tmp_graphXXX\n+ +-----------------------+\n+ | |\n+ | |\n+ | |\n+ | graph-{hash0} |\n+ | |\n+ | |\n+ | |\n+ +-----------------------+\n+\n+During this process, the commits to write are combined, sorted and we write the\n+contents to a temporary file, all while holding a `commit-graph-chain.lock`\n+lock-file. When the file is flushed, we rename it to `graph-{hash3}`\n+according to the computed `{hash3}`. Finally, we write the new chain data to\n+`commit-graph-chain.lock`:\n+\n+```\n+\t{hash3}\n+\t{hash0}\n+```\n+\n+We then close the lock-file.\n+\n+## Merge Strategy\n+\n+When writing a set of commits that do not exist in the commit-graph stack of\n+height N, we default to creating a new file at level N + 1. We then decide to\n+merge with the Nth level if one of two conditions hold:\n+\n+ 1. The expected file size for level N + 1 is at least half the file size for\n+ level N.\n+\n+ 2. Level N + 1 contains more than 64,0000 commits.\n+\n+This decision cascades down the levels: when we merge a level we create a new\n+set of commits that then compares to the next level.\n+\n+The first condition bounds the number of levels to be logarithmic in the total\n+number of commits. The second condition bounds the total number of commits in\n+a `graph-{hashN}` file and not in the `commit-graph` file, preventing\n+significant performance issues when the stack merges and another process only\n+partially reads the previous stack.\n+\n+The merge strategy values (2 for the size multiple, 64,000 for the maximum\n+number of commits) could be extracted into config settings for full\n+flexibility.\n+\n Related Links\n -------------\n [0] https://bugs.chromium.org/p/git/issues/detail?id=8\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 9b4acc0ac9..59c1067e70 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1291,36 +1291,6 @@ static int write_graph_chunk_base(struct hashfile *f,\n \treturn 0;\n }\n \n-static void init_commit_graph_chain(struct write_commit_graph_context *ctx)\n-{\n-\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n-\tuint32_t i;\n-\n-\tctx->new_base_graph = g;\n-\tctx->base_graph_name = xstrdup(g->filename);\n-\tctx->new_num_commits_in_base = g->num_commits + g->num_commits_in_base;\n-\n-\tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n-\n-\tALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);\n-\tALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);\n-\n-\tfor (i = 0; i < ctx->num_commit_graphs_before - 1; i++)\n-\t\tctx->commit_graph_filenames_after[i] = xstrdup(ctx->commit_graph_filenames_before[i]);\n-\n-\tif (ctx->num_commit_graphs_before)\n-\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_before - 1] =\n-\t\t\tget_split_graph_filename(ctx->obj_dir, oid_to_hex(&g->oid));\n-\n-\ti = ctx->num_commit_graphs_before - 1;\n-\n-\twhile (g) {\n-\t\tctx->commit_graph_hash_after[i] = xstrdup(oid_to_hex(&g->oid));\n-\t\ti--;\n-\t\tg = g->base_graph;\n-\t}\n-}\n-\n static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n@@ -1502,6 +1472,149 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \treturn 0;\n }\n \n+static int split_strategy_max_commits = 64000;\n+static float split_strategy_size_mult = 2.0f;\n+\n+static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n+{\n+\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\tuint32_t num_commits = ctx->commits.nr;\n+\tuint32_t i;\n+\n+\tg = ctx->r->objects->commit_graph;\n+\tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n+\n+\twhile (g && (g->num_commits <= split_strategy_size_mult * num_commits ||\n+\t\t num_commits > split_strategy_max_commits)) {\n+\t\tnum_commits += g->num_commits;\n+\t\tg = g->base_graph;\n+\n+\t\tctx->num_commit_graphs_after--;\n+\t}\n+\n+\tctx->new_base_graph = g;\n+\n+\tALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);\n+\tALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);\n+\n+\tfor (i = 0; i < ctx->num_commit_graphs_after &&\n+\t\t i < ctx->num_commit_graphs_before; i++)\n+\t\tctx->commit_graph_filenames_after[i] = xstrdup(ctx->commit_graph_filenames_before[i]);\n+\n+\ti = ctx->num_commit_graphs_before - 1;\n+\tg = ctx->r->objects->commit_graph;\n+\n+\twhile (g) {\n+\t\tif (i < ctx->num_commit_graphs_after)\n+\t\t\tctx->commit_graph_hash_after[i] = xstrdup(oid_to_hex(&g->oid));\n+\n+\t\ti--;\n+\t\tg = g->base_graph;\n+\t}\n+}\n+\n+static void merge_commit_graph(struct write_commit_graph_context *ctx,\n+\t\t\t struct commit_graph *g)\n+{\n+\tuint32_t i;\n+\tuint32_t offset = g->num_commits_in_base;\n+\n+\tALLOC_GROW(ctx->commits.list, ctx->commits.nr + g->num_commits, ctx->commits.alloc);\n+\n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct object_id oid;\n+\t\tstruct commit *result;\n+\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\n+\t\tload_oid_from_graph(g, i + offset, &oid);\n+\n+\t\t/* only add commits if they still exist in the repo */\n+\t\tresult = lookup_commit_reference_gently(ctx->r, &oid, 1);\n+\n+\t\tif (result) {\n+\t\t\tctx->commits.list[ctx->commits.nr] = result;\n+\t\t\tctx->commits.nr++;\n+\t\t}\n+\t}\n+}\n+\n+static int commit_compare(const void *_a, const void *_b)\n+{\n+\tconst struct commit *a = *(const struct commit **)_a;\n+\tconst struct commit *b = *(const struct commit **)_b;\n+\treturn oidcmp(&a->object.oid, &b->object.oid);\n+}\n+\n+static void deduplicate_commits(struct write_commit_graph_context *ctx)\n+{\n+\tuint32_t i, num_parents, last_distinct = 0, duplicates = 0;\n+\tstruct commit_list *parent;\n+\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"De-duplicating merged commits\"),\n+\t\t\t\t\tctx->commits.nr);\n+\n+\tQSORT(ctx->commits.list, ctx->commits.nr, commit_compare);\n+\n+\tctx->num_extra_edges = 0;\n+\tfor (i = 1; i < ctx->commits.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i);\n+\n+\t\tif (oideq(&ctx->commits.list[last_distinct]->object.oid,\n+\t\t\t &ctx->commits.list[i]->object.oid)) {\n+\t\t\tduplicates++;\n+\t\t} else {\n+\t\t\tif (duplicates)\n+\t\t\t\tctx->commits.list[last_distinct + 1] = ctx->commits.list[i];\n+\t\t\tlast_distinct++;\n+\n+\t\t\tnum_parents = 0;\n+\t\t\tfor (parent = ctx->commits.list[i]->parents; parent; parent = parent->next)\n+\t\t\t\tnum_parents++;\n+\n+\t\t\tif (num_parents > 2)\n+\t\t\t\tctx->num_extra_edges += num_parents - 2;\n+\t\t}\n+\t}\n+\n+\tctx->commits.nr -= duplicates;\n+\tstop_progress(&ctx->progress);\n+}\n+\n+static void merge_commit_graphs(struct write_commit_graph_context *ctx)\n+{\n+\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\tuint32_t current_graph_number = ctx->num_commit_graphs_before;\n+\tstruct strbuf progress_title = STRBUF_INIT;\n+\n+\twhile (g && current_graph_number >= ctx->num_commit_graphs_after) {\n+\t\tcurrent_graph_number--;\n+\n+\t\tif (ctx->report_progress) {\n+\t\t\tstrbuf_addstr(&progress_title, _(\"Merging commit-graph\"));\n+\t\t\tctx->progress = start_delayed_progress(progress_title.buf, 0);\n+\t\t}\n+\n+\t\tmerge_commit_graph(ctx, g);\n+\t\tstop_progress(&ctx->progress);\n+\t\tstrbuf_release(&progress_title);\n+\n+\t\tg = g->base_graph;\n+\t}\n+\n+\tif (g) {\n+\t\tctx->new_base_graph = g;\n+\t\tctx->new_num_commits_in_base = g->num_commits + g->num_commits_in_base;\n+\t}\n+\n+\tif (ctx->new_base_graph)\n+\t\tctx->base_graph_name = xstrdup(ctx->new_base_graph->filename);\n+\n+\tdeduplicate_commits(ctx);\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -1547,6 +1660,9 @@ int write_commit_graph(const char *obj_dir,\n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n \n+\tif (ctx->split && ctx->oids.alloc > split_strategy_max_commits)\n+\t\tctx->oids.alloc = split_strategy_max_commits;\n+\n \tif (ctx->append) {\n \t\tprepare_commit_graph_one(ctx->r, ctx->obj_dir);\n \t\tif (ctx->r->objects->commit_graph)\n@@ -1600,9 +1716,11 @@ int write_commit_graph(const char *obj_dir,\n \tif (!ctx->commits.nr)\n \t\tgoto cleanup;\n \n-\tif (ctx->split)\n-\t\tinit_commit_graph_chain(ctx);\n-\telse\n+\tif (ctx->split) {\n+\t\tsplit_graph_merge_strategy(ctx);\n+\n+\t\tmerge_commit_graphs(ctx);\n+\t} else\n \t\tctx->num_commit_graphs_after = 1;\n \n \tcompute_generation_numbers(ctx);\ndiff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\nindex ccd24bd22b..5cb5663a30 100755\n--- a/t/t5323-split-commit-graph.sh\n+++ b/t/t5323-split-commit-graph.sh\n@@ -119,4 +119,17 @@ test_expect_success 'add one commit, write a tip graph' '\n \n graph_git_behavior 'three-layer commit-graph: commit 11 vs 6' commits/11 commits/6\n \n+test_expect_success 'add one commit, write a merged graph' '\n+\ttest_commit 12 &&\n+\tgit branch commits/12 &&\n+\tgit commit-graph write --reachable --split &&\n+\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\tls $graphdir/graph-*.graph >graph-files &&\n+\ttest_line_count = 4 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n+graph_git_behavior 'merged commit-graph: commit 12 vs 6' commits/12 commits/6\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"376759","messageId":"b5aeeed909c1f02086b7be1149d92ca1f961e358.1559830527.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 11/14] commit-graph: expire commit-graph files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-06T14:15:38Z","receivedAt":"2019-06-06T14:15:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we merge commit-graph files in a commit-graph chain, we should clean\nup the files that are no longer used.\n\nThis change introduces an 'expiry_window' value to the context, which is\nalways zero (for now). We then check the modified time of each\ngraph-{hash}.graph file in the $OBJDIR/info/commit-graphs folder and\nunlink the files that are older than the expiry_window.\n\nSince this is always zero, this immediately clears all unused graph\nfiles. We will update the value to match a config setting in a future\nchange.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 15 +++++\n commit-graph.c | 79 +++++++++++++++++++++++-\n t/t5323-split-commit-graph.sh | 2 +-\n 3 files changed, 94 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex 473032e476..aed4350a59 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -266,6 +266,21 @@ The merge strategy values (2 for the size multiple, 64,000 for the maximum\n number of commits) could be extracted into config settings for full\n flexibility.\n \n+## Deleting graph-{hash} files\n+\n+After a new tip file is written, some `graph-{hash}` files may no longer\n+be part of a chain. It is important to remove these files from disk, eventually.\n+The main reason to delay removal is that another process could read the\n+`commit-graph-chain` file before it is rewritten, but then look for the\n+`graph-{hash}` files after they are deleted.\n+\n+To allow holding old split commit-graphs for a while after they are unreferenced,\n+we update the modified times of the files when they become unreferenced. Then,\n+we scan the `$OBJDIR/info/commit-graphs/` directory for `graph-{hash}`\n+files whose modified times are older than a given expiry window. This window\n+defaults to zero, but can be changed using command-line arguments or a config\n+setting.\n+\n ## Chains across multiple object directories\n \n In a repo with alternates, we look for the `commit-graph-chain` file starting\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 39d986bb29..6409e5ed8d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -500,9 +500,18 @@ int generation_numbers_enabled(struct repository *r)\n \treturn !!first_generation;\n }\n \n+static void close_commit_graph_one(struct commit_graph *g)\n+{\n+\tif (!g)\n+\t\treturn;\n+\n+\tclose_commit_graph_one(g->base_graph);\n+\tfree_commit_graph(g);\n+}\n+\n void close_commit_graph(struct repository *r)\n {\n-\tfree_commit_graph(r->objects->commit_graph);\n+\tclose_commit_graph_one(r->objects->commit_graph);\n \tr->objects->commit_graph = NULL;\n }\n \n@@ -1641,6 +1650,69 @@ static void merge_commit_graphs(struct write_commit_graph_context *ctx)\n \tdeduplicate_commits(ctx);\n }\n \n+static void mark_commit_graphs(struct write_commit_graph_context *ctx)\n+{\n+\tuint32_t i;\n+\ttime_t now = time(NULL);\n+\n+\tfor (i = ctx->num_commit_graphs_after - 1; i < ctx->num_commit_graphs_before; i++) {\n+\t\tstruct stat st;\n+\t\tstruct utimbuf updated_time;\n+\n+\t\tstat(ctx->commit_graph_filenames_before[i], &st);\n+\n+\t\tupdated_time.actime = st.st_atime;\n+\t\tupdated_time.modtime = now;\n+\t\tutime(ctx->commit_graph_filenames_before[i], &updated_time);\n+\t}\n+}\n+\n+static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tDIR *dir;\n+\tstruct dirent *de;\n+\tsize_t dirnamelen;\n+\ttime_t expire_time = time(NULL);\n+\n+\tstrbuf_addstr(&path, ctx->obj_dir);\n+\tstrbuf_addstr(&path, \"/info/commit-graphs\");\n+\tdir = opendir(path.buf);\n+\n+\tif (!dir) {\n+\t\tstrbuf_release(&path);\n+\t\treturn;\n+\t}\n+\n+\tstrbuf_addch(&path, '/');\n+\tdirnamelen = path.len;\n+\twhile ((de = readdir(dir)) != NULL) {\n+\t\tstruct stat st;\n+\t\tuint32_t i, found = 0;\n+\n+\t\tstrbuf_setlen(&path, dirnamelen);\n+\t\tstrbuf_addstr(&path, de->d_name);\n+\n+\t\tstat(path.buf, &st);\n+\n+\t\tif (st.st_mtime > expire_time)\n+\t\t\tcontinue;\n+\t\tif (path.len < 6 || strcmp(path.buf + path.len - 6, \".graph\"))\n+\t\t\tcontinue;\n+\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++) {\n+\t\t\tif (!strcmp(ctx->commit_graph_filenames_after[i],\n+\t\t\t\t path.buf)) {\n+\t\t\t\tfound = 1;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (!found)\n+\t\t\tunlink(path.buf);\n+\t}\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -1753,6 +1825,11 @@ int write_commit_graph(const char *obj_dir,\n \n \tres = write_commit_graph_file(ctx);\n \n+\tif (ctx->split) {\n+\t\tmark_commit_graphs(ctx);\n+\t\texpire_commit_graphs(ctx);\n+\t}\n+\n cleanup:\n \tfree(ctx->graph_name);\n \tfree(ctx->commits.list);\ndiff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\nindex cd4d5f05b6..c6bb685eb9 100755\n--- a/t/t5323-split-commit-graph.sh\n+++ b/t/t5323-split-commit-graph.sh\n@@ -141,7 +141,7 @@ test_expect_success 'add one commit, write a merged graph' '\n \ttest_path_is_file $graphdir/commit-graph-chain &&\n \ttest_line_count = 2 $graphdir/commit-graph-chain &&\n \tls $graphdir/graph-*.graph >graph-files &&\n-\ttest_line_count = 4 graph-files &&\n+\ttest_line_count = 2 graph-files &&\n \tverify_chain_files_exist $graphdir\n '\n \n-- \ngitgitgadget\n\n"},{"id":"376761","messageId":"62b3fca582814c2cee050bab07533409e22e2c3c.1559830527.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 10/14] commit-graph: allow cross-alternate chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-06T14:15:37Z","receivedAt":"2019-06-06T14:15:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an environment like a fork network, it is helpful to have a\ncommit-graph chain that spans both the base repo and the fork repo. The\nfork is usually a small set of data on top of the large repo, but\nsometimes the fork is much larger. For example, git-for-windows/git has\nalmost double the number of commits as git/git because it rebases its\ncommits on every major version update.\n\nTo allow cross-alternate commit-graph chains, we need a few pieces:\n\n1. When looking for a graph-{hash}.graph file, check all alternates.\n\n2. When merging commit-graph chains, do not merge across alternates.\n\n3. When writing a new commit-graph chain based on a commit-graph file\n in another object directory, do not allow success if the base file\n has of the name \"commit-graph\" instead of\n \"commit-graphs/graoh-{hash}.graph\".\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 40 +++++++++++++++++++++\n commit-graph.c | 46 ++++++++++++++++++------\n commit-graph.h | 1 +\n t/t5323-split-commit-graph.sh | 37 +++++++++++++++++++\n 4 files changed, 114 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex d9c6253b0a..473032e476 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -266,6 +266,42 @@ The merge strategy values (2 for the size multiple, 64,000 for the maximum\n number of commits) could be extracted into config settings for full\n flexibility.\n \n+## Chains across multiple object directories\n+\n+In a repo with alternates, we look for the `commit-graph-chain` file starting\n+in the local object directory and then in each alternate. The first file that\n+exists defines our chain. As we look for the `graph-{hash}` files for\n+each `{hash}` in the chain file, we follow the same pattern for the host\n+directories.\n+\n+This allows commit-graphs to be split across multiple forks in a fork network.\n+The typical case is a large \"base\" repo with many smaller forks.\n+\n+As the base repo advances, it will likely update and merge its commit-graph\n+chain more frequently than the forks. If a fork updates their commit-graph after\n+the base repo, then it should \"reparent\" the commit-graph chain onto the new\n+chain in the base repo. When reading each `graph-{hash}` file, we track\n+the object directory containing it. During a write of a new commit-graph file,\n+we check for any changes in the source object directory and read the\n+`commit-graph-chain` file for that source and create a new file based on those\n+files. During this \"reparent\" operation, we necessarily need to collapse all\n+levels in the fork, as all of the files are invalid against the new base file.\n+\n+It is crucial to be careful when cleaning up \"unreferenced\" `graph-{hash}.graph`\n+files in this scenario. It falls to the user to define the proper settings for\n+their custom environment:\n+\n+ 1. When merging levels in the base repo, the unreferenced files may still be\n+ referenced by chains from fork repos.\n+\n+ 2. The expiry time should be set to a length of time such that every fork has\n+ time to recompute their commit-graph chain to \"reparent\" onto the new base\n+ file(s).\n+\n+ 3. If the commit-graph chain is updated in the base, the fork will not have\n+ access to the new chain until its chain is updated to reference those files.\n+ (This may change in the future [5].)\n+\n Related Links\n -------------\n [0] https://bugs.chromium.org/p/git/issues/detail?id=8\n@@ -292,3 +328,7 @@ Related Links\n \n [4] https://public-inbox.org/git/20180108154822.54829-1-git@jeffhostetler.com/T/#u\n A patch to remove the ahead-behind calculation from 'status'.\n+\n+[5] https://public-inbox.org/git/f27db281-abad-5043-6d71-cbb083b1c877@gmail.com/\n+ A discussion of a \"two-dimensional graph position\" that can allow reading\n+ multiple commit-graph chains at the same time.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 59c1067e70..39d986bb29 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -320,6 +320,9 @@ static struct commit_graph *load_commit_graph_v1(struct repository *r, const cha\n \tstruct commit_graph *g = load_commit_graph_one(graph_name);\n \tfree(graph_name);\n \n+\tif (g)\n+\t\tg->obj_dir = obj_dir;\n+\n \treturn g;\n }\n \n@@ -385,8 +388,7 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r, const\n \toids = xcalloc(st.st_size / (the_hash_algo->hexsz + 1), sizeof(struct object_id));\n \n \twhile (strbuf_getline_lf(&line, fp) != EOF && valid) {\n-\t\tchar *graph_name;\n-\t\tstruct commit_graph *g;\n+\t\tstruct object_directory *odb;\n \n \t\tif (get_oid_hex(line.buf, &oids[i])) {\n \t\t\twarning(_(\"invalid commit-graph chain: line '%s' not a hash\"),\n@@ -395,14 +397,23 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r, const\n \t\t\tbreak;\n \t\t}\n \n-\t\tgraph_name = get_split_graph_filename(obj_dir, line.buf);\n-\t\tg = load_commit_graph_one(graph_name);\n-\t\tfree(graph_name);\n+\t\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n+\t\t\tchar *graph_name = get_split_graph_filename(odb->path, line.buf);\n+\t\t\tstruct commit_graph *g = load_commit_graph_one(graph_name);\n \n-\t\tif (g && add_graph_to_chain(g, graph_chain, oids, i))\n-\t\t\tgraph_chain = g;\n-\t\telse\n-\t\t\tvalid = 0;\n+\t\t\tfree(graph_name);\n+\n+\t\t\tif (g) {\n+\t\t\t\tg->obj_dir = odb->path;\n+\n+\t\t\t\tif (add_graph_to_chain(g, graph_chain, oids, i))\n+\t\t\t\t\tgraph_chain = g;\n+\t\t\t\telse\n+\t\t\t\t\tvalid = 0;\n+\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n \t}\n \n \tfree(oids);\n@@ -1411,7 +1422,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \n \tif (ctx->split && ctx->base_graph_name && ctx->num_commit_graphs_after > 1) {\n \t\tchar *new_base_hash = xstrdup(oid_to_hex(&ctx->new_base_graph->oid));\n-\t\tchar *new_base_name = get_split_graph_filename(ctx->obj_dir, new_base_hash);\n+\t\tchar *new_base_name = get_split_graph_filename(ctx->new_base_graph->obj_dir, new_base_hash);\n \n \t\tfree(ctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2]);\n \t\tfree(ctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 2]);\n@@ -1486,6 +1497,9 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \n \twhile (g && (g->num_commits <= split_strategy_size_mult * num_commits ||\n \t\t num_commits > split_strategy_max_commits)) {\n+\t\tif (strcmp(g->obj_dir, ctx->obj_dir))\n+\t\t\tbreak;\n+\n \t\tnum_commits += g->num_commits;\n \t\tg = g->base_graph;\n \n@@ -1494,6 +1508,18 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \n \tctx->new_base_graph = g;\n \n+\tif (ctx->num_commit_graphs_after == 2) {\n+\t\tchar *old_graph_name = get_commit_graph_filename(g->obj_dir);\n+\n+\t\tif (!strcmp(g->filename, old_graph_name) &&\n+\t\t strcmp(g->obj_dir, ctx->obj_dir)) {\n+\t\t\tctx->num_commit_graphs_after = 1;\n+\t\t\tctx->new_base_graph = NULL;\n+\t\t}\n+\n+\t\tfree(old_graph_name);\n+\t}\n+\n \tALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);\n \tALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex 5c48c4f66a..10466bc064 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -48,6 +48,7 @@ struct commit_graph {\n \tuint32_t num_commits;\n \tstruct object_id oid;\n \tchar *filename;\n+\tconst char *obj_dir;\n \n \tuint32_t num_commits_in_base;\n \tstruct commit_graph *base_graph;\ndiff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\nindex 5cb5663a30..cd4d5f05b6 100755\n--- a/t/t5323-split-commit-graph.sh\n+++ b/t/t5323-split-commit-graph.sh\n@@ -90,6 +90,21 @@ test_expect_success 'add more commits, and write a new base graph' '\n \tgraph_read_expect 12\n '\n \n+test_expect_success 'fork and fail to base a chain on a commit-graph file' '\n+\ttest_when_finished rm -rf fork &&\n+\tgit clone . fork &&\n+\t(\n+\t\tcd fork &&\n+\t\trm .git/objects/info/commit-graph &&\n+\t\techo \"$(pwd)\"/../.git/objects >.git/objects/info/alternates &&\n+\t\ttest_commit new-commit &&\n+\t\tgit commit-graph write --reachable --split &&\n+\t\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n+\t\tverify_chain_files_exist $graphdir\n+\t)\n+'\n+\n test_expect_success 'add three more commits, write a tip graph' '\n \tgit reset --hard commits/3 &&\n \tgit merge merge/1 &&\n@@ -132,4 +147,26 @@ test_expect_success 'add one commit, write a merged graph' '\n \n graph_git_behavior 'merged commit-graph: commit 12 vs 6' commits/12 commits/6\n \n+test_expect_success 'create fork and chain across alternate' '\n+\tgit clone . fork &&\n+\t(\n+\t\tcd fork &&\n+\t\tgit config core.commitGraph true &&\n+\t\trm -rf $graphdir &&\n+\t\techo \"$(pwd)\"/../.git/objects >.git/objects/info/alternates &&\n+\t\ttest_commit 13 &&\n+\t\tgit branch commits/13 &&\n+\t\tgit commit-graph write --reachable --split &&\n+\t\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\t\ttest_line_count = 3 $graphdir/commit-graph-chain &&\n+\t\tls $graphdir/graph-*.graph >graph-files &&\n+\t\ttest_line_count = 1 graph-files &&\n+\t\tgit -c core.commitGraph=true rev-list HEAD >expect &&\n+\t\tgit -c core.commitGraph=false rev-list HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+graph_git_behavior 'alternate: commit 13 vs 6' commits/13 commits/6\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"376762","messageId":"c42e683ef644e7d34614eb74156e5a2b31392758.1559830527.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 08/14] commit-graph: add --split option to builtin","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-06T14:15:35Z","receivedAt":"2019-06-06T14:15:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd a new \"--split\" option to the 'git commit-graph write' subcommand. This\noption allows the optional behavior of writing a commit-graph chain.\n\nThe current behavior will add a tip commit-graph containing any commits that\nare not in the existing commit-graph or commit-graph chain. Later changes\nwill allow merging the chain and expiring out-dated files.\n\nAdd a new test script (t5323-split-commit-graph.sh) that demonstrates this\nbehavior.\n\nHelped-by: Johannes Schindelin \nSigned-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 10 ++-\n commit-graph.c | 14 ++--\n t/t5323-split-commit-graph.sh | 122 ++++++++++++++++++++++++++++++++++\n 3 files changed, 138 insertions(+), 8 deletions(-)\n create mode 100755 t/t5323-split-commit-graph.sh\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 828b1a713f..c2c07d3917 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -10,7 +10,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph verify [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -25,7 +25,7 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -35,9 +35,9 @@ static struct opts_commit_graph {\n \tint stdin_packs;\n \tint stdin_commits;\n \tint append;\n+\tint split;\n } opts;\n \n-\n static int graph_verify(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n@@ -156,6 +156,8 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"start walk at commits listed by stdin\")),\n \t\tOPT_BOOL(0, \"append\", &opts.append,\n \t\t\tN_(\"include all commits already in the commit-graph file\")),\n+\t\tOPT_BOOL(0, \"split\", &opts.split,\n+\t\t\tN_(\"allow writing an incremental commit-graph file\")),\n \t\tOPT_END(),\n \t};\n \n@@ -169,6 +171,8 @@ static int graph_write(int argc, const char **argv)\n \t\topts.obj_dir = get_object_directory();\n \tif (opts.append)\n \t\tflags |= COMMIT_GRAPH_APPEND;\n+\tif (opts.split)\n+\t\tflags |= COMMIT_GRAPH_SPLIT;\n \n \tread_replace_refs = 0;\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 7d3e001479..9b4acc0ac9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1465,12 +1465,16 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t}\n \n \t\tif (ctx->base_graph_name) {\n-\t\t\tresult = rename(ctx->base_graph_name,\n-\t\t\t\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2]);\n+\t\t\tconst char *dest = ctx->commit_graph_filenames_after[\n+\t\t\t\t\t\tctx->num_commit_graphs_after - 2];\n \n-\t\t\tif (result) {\n-\t\t\t\terror(_(\"failed to rename base commit-graph file\"));\n-\t\t\t\treturn -1;\n+\t\t\tif (strcmp(ctx->base_graph_name, dest)) {\n+\t\t\t\tresult = rename(ctx->base_graph_name, dest);\n+\n+\t\t\t\tif (result) {\n+\t\t\t\t\terror(_(\"failed to rename base commit-graph file\"));\n+\t\t\t\t\treturn -1;\n+\t\t\t\t}\n \t\t\t}\n \t\t} else {\n \t\t\tchar *graph_name = get_commit_graph_filename(ctx->obj_dir);\ndiff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\nnew file mode 100755\nindex 0000000000..ccd24bd22b\n--- /dev/null\n+++ b/t/t5323-split-commit-graph.sh\n@@ -0,0 +1,122 @@\n+#!/bin/sh\n+\n+test_description='split commit graph'\n+. ./test-lib.sh\n+\n+GIT_TEST_COMMIT_GRAPH=0\n+\n+test_expect_success 'setup repo' '\n+\tgit init &&\n+\tgit config core.commitGraph true &&\n+\tinfodir=\".git/objects/info\" &&\n+\tgraphdir=\"$infodir/commit-graphs\" &&\n+\ttest_oid_init\n+'\n+\n+graph_read_expect() {\n+\tNUM_BASE=0\n+\tif test ! -z $2\n+\tthen\n+\t\tNUM_BASE=$2\n+\tfi\n+\tcat >expect <<- EOF\n+\theader: 43475048 1 1 3 $NUM_BASE\n+\tnum_commits: $1\n+\tchunks: oid_fanout oid_lookup commit_metadata\n+\tEOF\n+\tgit commit-graph read >output &&\n+\ttest_cmp expect output\n+}\n+\n+test_expect_success 'create commits and write commit-graph' '\n+\tfor i in $(test_seq 3)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i || return 1\n+\tdone &&\n+\tgit commit-graph write --reachable &&\n+\ttest_path_is_file $infodir/commit-graph &&\n+\tgraph_read_expect 3\n+'\n+\n+graph_git_two_modes() {\n+\tgit -c core.commitGraph=true $1 >output\n+\tgit -c core.commitGraph=false $1 >expect\n+\ttest_cmp expect output\n+}\n+\n+graph_git_behavior() {\n+\tMSG=$1\n+\tBRANCH=$2\n+\tCOMPARE=$3\n+\ttest_expect_success \"check normal git operations: $MSG\" '\n+\t\tgraph_git_two_modes \"log --oneline $BRANCH\" &&\n+\t\tgraph_git_two_modes \"log --topo-order $BRANCH\" &&\n+\t\tgraph_git_two_modes \"log --graph $COMPARE..$BRANCH\" &&\n+\t\tgraph_git_two_modes \"branch -vv\" &&\n+\t\tgraph_git_two_modes \"merge-base -a $BRANCH $COMPARE\"\n+\t'\n+}\n+\n+graph_git_behavior 'graph exists' commits/3 commits/1\n+\n+verify_chain_files_exist() {\n+\tfor hash in $(cat $1/commit-graph-chain)\n+\tdo\n+\t\ttest_path_is_file $1/graph-$hash.graph || return 1\n+\tdone\n+}\n+\n+test_expect_success 'add more commits, and write a new base graph' '\n+\tgit reset --hard commits/1 &&\n+\tfor i in $(test_seq 4 5)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i || return 1\n+\tdone &&\n+\tgit reset --hard commits/2 &&\n+\tfor i in $(test_seq 6 10)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i || return 1\n+\tdone &&\n+\tgit reset --hard commits/2 &&\n+\tgit merge commits/4 &&\n+\tgit branch merge/1 &&\n+\tgit reset --hard commits/4 &&\n+\tgit merge commits/6 &&\n+\tgit branch merge/2 &&\n+\tgit commit-graph write --reachable &&\n+\tgraph_read_expect 12\n+'\n+\n+test_expect_success 'add three more commits, write a tip graph' '\n+\tgit reset --hard commits/3 &&\n+\tgit merge merge/1 &&\n+\tgit merge commits/5 &&\n+\tgit merge merge/2 &&\n+\tgit branch merge/3 &&\n+\tgit commit-graph write --reachable --split &&\n+\ttest_path_is_missing $infodir/commit-graph &&\n+\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\tls $graphdir/graph-*.graph >graph-files &&\n+\ttest_line_count = 2 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n+graph_git_behavior 'split commit-graph: merge 3 vs 2' merge/3 merge/2\n+\n+test_expect_success 'add one commit, write a tip graph' '\n+\ttest_commit 11 &&\n+\tgit branch commits/11 &&\n+\tgit commit-graph write --reachable --split &&\n+\ttest_path_is_missing $infodir/commit-graph &&\n+\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\tls $graphdir/graph-*.graph >graph-files &&\n+\ttest_line_count = 3 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n+graph_git_behavior 'three-layer commit-graph: commit 11 vs 6' commits/11 commits/6\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"376763","messageId":"ac5586a20f7ed600f8912b1aed07dea5098ea046.1559830527.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 12/14] commit-graph: create options for split files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-06T14:15:39Z","receivedAt":"2019-06-06T14:15:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe split commit-graph feature is now fully implemented, but needs\nsome more run-time configurability. Allow direct callers to 'git\ncommit-graph write --split' to specify the values used in the\nmerge strategy and the expire time.\n\nUpdate the documentation to specify these values.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 21 +++++++++++++-\n Documentation/technical/commit-graph.txt | 7 +++--\n builtin/commit-graph.c | 20 +++++++++++---\n builtin/commit.c | 2 +-\n builtin/gc.c | 3 +-\n commit-graph.c | 35 ++++++++++++++++--------\n commit-graph.h | 12 ++++++--\n t/t5323-split-commit-graph.sh | 35 ++++++++++++++++++++++++\n 8 files changed, 112 insertions(+), 23 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 624470e198..365e145e82 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -26,7 +26,7 @@ OPTIONS\n \tUse given directory for the location of packfiles and commit-graph\n \tfile. This parameter exists to specify the location of an alternate\n \tthat only has the objects directory, not a full `.git` directory. The\n-\tcommit-graph file is expected to be at `/info/commit-graph` and\n+\tcommit-graph file is expected to be in the `/info` directory and\n \tthe packfiles are expected to be in `/pack`.\n \n \n@@ -51,6 +51,25 @@ or `--stdin-packs`.)\n +\n With the `--append` option, include all commits that are present in the\n existing commit-graph file.\n++\n+With the `--split` option, write the commit-graph as a chain of multiple\n+commit-graph files stored in `/info/commit-graphs`. The new commits\n+not already in the commit-graph are added in a new \"tip\" file. This file\n+is merged with the existing file if the following merge conditions are\n+met:\n++\n+* If `--size-multiple=` is not specified, let `X` equal 2. If the new\n+tip file would have `N` commits and the previous tip has `M` commits and\n+`X` times `N` is greater than `M`, instead merge the two files into a\n+single file.\n++\n+* If `--max-commits=` is specified with `M` a positive integer, and the\n+new tip file would have more than `M` commits, then instead merge the new\n+tip with the previous tip.\n++\n+Finally, if `--expire-time=` is not specified, let `datetime`\n+be the current time. After writing the split commit-graph, delete all\n+unused commit-graph whose modified times are older than `datetime`.\n \n 'read'::\n \ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex aed4350a59..729fbcb32f 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -248,10 +248,11 @@ When writing a set of commits that do not exist in the commit-graph stack of\n height N, we default to creating a new file at level N + 1. We then decide to\n merge with the Nth level if one of two conditions hold:\n \n- 1. The expected file size for level N + 1 is at least half the file size for\n- level N.\n+ 1. `--size-multiple=` is specified or X = 2, and the number of commits in\n+ level N is less than X times the number of commits in level N + 1.\n \n- 2. Level N + 1 contains more than 64,0000 commits.\n+ 2. `--max-commits=` is specified with non-zero C and the number of commits\n+ in level N + 1 is more than C commits.\n \n This decision cascades down the levels: when we merge a level we create a new\n set of commits that then compares to the next level.\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex c2c07d3917..18e3b61fb6 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -10,7 +10,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph verify [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] \"),\n \tNULL\n };\n \n@@ -25,7 +25,7 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] \"),\n \tNULL\n };\n \n@@ -135,6 +135,7 @@ static int graph_read(int argc, const char **argv)\n }\n \n extern int read_replace_refs;\n+struct split_commit_graph_opts split_opts;\n \n static int graph_write(int argc, const char **argv)\n {\n@@ -158,9 +159,19 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"include all commits already in the commit-graph file\")),\n \t\tOPT_BOOL(0, \"split\", &opts.split,\n \t\t\tN_(\"allow writing an incremental commit-graph file\")),\n+\t\tOPT_INTEGER(0, \"max-commits\", &split_opts.max_commits,\n+\t\t\tN_(\"maximum number of commits in a non-base split commit-graph\")),\n+\t\tOPT_INTEGER(0, \"size-multiple\", &split_opts.size_multiple,\n+\t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n+\t\tOPT_EXPIRY_DATE(0, \"expire-time\", &split_opts.expire_time,\n+\t\t\tN_(\"maximum number of commits in a non-base split commit-graph\")),\n \t\tOPT_END(),\n \t};\n \n+\tsplit_opts.size_multiple = 2;\n+\tsplit_opts.max_commits = 0;\n+\tsplit_opts.expire_time = 0;\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\n@@ -177,7 +188,7 @@ static int graph_write(int argc, const char **argv)\n \tread_replace_refs = 0;\n \n \tif (opts.reachable)\n-\t\treturn write_commit_graph_reachable(opts.obj_dir, flags);\n+\t\treturn write_commit_graph_reachable(opts.obj_dir, flags, &split_opts);\n \n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n@@ -197,7 +208,8 @@ static int graph_write(int argc, const char **argv)\n \tresult = write_commit_graph(opts.obj_dir,\n \t\t\t\t pack_indexes,\n \t\t\t\t commit_hex,\n-\t\t\t\t flags);\n+\t\t\t\t flags,\n+\t\t\t\t &split_opts);\n \n \tUNLEAK(lines);\n \treturn result;\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex b001ef565d..9216e9c043 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1670,7 +1670,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\t \"not exceeded, and then \\\"git reset HEAD\\\" to recover.\"));\n \n \tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n-\t write_commit_graph_reachable(get_object_directory(), 0))\n+\t write_commit_graph_reachable(get_object_directory(), 0, NULL))\n \t\treturn 1;\n \n \trepo_rerere(the_repository, 0);\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex df2573f124..2ab590ffd4 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -666,7 +666,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \n \tif (gc_write_commit_graph &&\n \t write_commit_graph_reachable(get_object_directory(),\n-\t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_PROGRESS : 0))\n+\t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_PROGRESS : 0,\n+\t\t\t\t\t NULL))\n \t\treturn 1;\n \n \tif (auto_gc && too_many_loose_objects())\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 6409e5ed8d..abbfc12f1f 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -762,6 +762,8 @@ struct write_commit_graph_context {\n \tunsigned append:1,\n \t\t report_progress:1,\n \t\t split:1;\n+\n+\tconst struct split_commit_graph_opts *split_opts;\n };\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n@@ -1110,14 +1112,15 @@ static int add_ref_to_list(const char *refname,\n \treturn 0;\n }\n \n-int write_commit_graph_reachable(const char *obj_dir, unsigned int flags)\n+int write_commit_graph_reachable(const char *obj_dir, unsigned int flags,\n+\t\t\t\t const struct split_commit_graph_opts *split_opts)\n {\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n \tint result;\n \n \tfor_each_ref(add_ref_to_list, &list);\n \tresult = write_commit_graph(obj_dir, NULL, &list,\n-\t\t\t\t flags);\n+\t\t\t\t flags, split_opts);\n \n \tstring_list_clear(&list, 0);\n \treturn result;\n@@ -1492,20 +1495,25 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \treturn 0;\n }\n \n-static int split_strategy_max_commits = 64000;\n-static float split_strategy_size_mult = 2.0f;\n-\n static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n {\n \tstruct commit_graph *g = ctx->r->objects->commit_graph;\n \tuint32_t num_commits = ctx->commits.nr;\n \tuint32_t i;\n \n+\tint max_commits = 0;\n+\tint size_mult = 2;\n+\n+\tif (ctx->split_opts) {\n+\t\tmax_commits = ctx->split_opts->max_commits;\n+\t\tsize_mult = ctx->split_opts->size_multiple;\n+\t}\n+\n \tg = ctx->r->objects->commit_graph;\n \tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n \n-\twhile (g && (g->num_commits <= split_strategy_size_mult * num_commits ||\n-\t\t num_commits > split_strategy_max_commits)) {\n+\twhile (g && (g->num_commits <= size_mult * num_commits ||\n+\t\t (max_commits && num_commits > max_commits))) {\n \t\tif (strcmp(g->obj_dir, ctx->obj_dir))\n \t\t\tbreak;\n \n@@ -1673,7 +1681,10 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \tDIR *dir;\n \tstruct dirent *de;\n \tsize_t dirnamelen;\n-\ttime_t expire_time = time(NULL);\n+\ttimestamp_t expire_time = time(NULL);\n+\n+\tif (ctx->split_opts && ctx->split_opts->expire_time)\n+\t\texpire_time -= ctx->split_opts->expire_time;\n \n \tstrbuf_addstr(&path, ctx->obj_dir);\n \tstrbuf_addstr(&path, \"/info/commit-graphs\");\n@@ -1716,7 +1727,8 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t unsigned int flags)\n+\t\t unsigned int flags,\n+\t\t const struct split_commit_graph_opts *split_opts)\n {\n \tstruct write_commit_graph_context *ctx;\n \tuint32_t i, count_distinct = 0;\n@@ -1731,6 +1743,7 @@ int write_commit_graph(const char *obj_dir,\n \tctx->append = flags & COMMIT_GRAPH_APPEND ? 1 : 0;\n \tctx->report_progress = flags & COMMIT_GRAPH_PROGRESS ? 1 : 0;\n \tctx->split = flags & COMMIT_GRAPH_SPLIT ? 1 : 0;\n+\tctx->split_opts = split_opts;\n \n \tif (ctx->split) {\n \t\tstruct commit_graph *g;\n@@ -1758,8 +1771,8 @@ int write_commit_graph(const char *obj_dir,\n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n \n-\tif (ctx->split && ctx->oids.alloc > split_strategy_max_commits)\n-\t\tctx->oids.alloc = split_strategy_max_commits;\n+\tif (ctx->split && split_opts && ctx->oids.alloc > split_opts->max_commits)\n+\t\tctx->oids.alloc = split_opts->max_commits;\n \n \tif (ctx->append) {\n \t\tprepare_commit_graph_one(ctx->r, ctx->obj_dir);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 10466bc064..194acab2b7 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -75,11 +75,19 @@ int generation_numbers_enabled(struct repository *r);\n #define COMMIT_GRAPH_PROGRESS (1 << 1)\n #define COMMIT_GRAPH_SPLIT (1 << 2)\n \n-int write_commit_graph_reachable(const char *obj_dir, unsigned int flags);\n+struct split_commit_graph_opts {\n+\tint size_multiple;\n+\tint max_commits;\n+\ttimestamp_t expire_time;\n+};\n+\n+int write_commit_graph_reachable(const char *obj_dir, unsigned int flags,\n+\t\t\t\t const struct split_commit_graph_opts *split_opts);\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t unsigned int flags);\n+\t\t unsigned int flags,\n+\t\t const struct split_commit_graph_opts *split_opts);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \ndiff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\nindex c6bb685eb9..a915ef388c 100755\n--- a/t/t5323-split-commit-graph.sh\n+++ b/t/t5323-split-commit-graph.sh\n@@ -169,4 +169,39 @@ test_expect_success 'create fork and chain across alternate' '\n \n graph_git_behavior 'alternate: commit 13 vs 6' commits/13 commits/6\n \n+test_expect_success 'test merge stragety constants' '\n+\tgit clone . merge-2 &&\n+\t(\n+\t\tcd merge-2 &&\n+\t\tgit config core.commitGraph true &&\n+\t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\t\ttest_commit 14 &&\n+\t\tgit commit-graph write --reachable --split --size-multiple=2 &&\n+\t\ttest_line_count = 3 $graphdir/commit-graph-chain\n+\n+\t) &&\n+\tgit clone . merge-10 &&\n+\t(\n+\t\tcd merge-10 &&\n+\t\tgit config core.commitGraph true &&\n+\t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\t\ttest_commit 14 &&\n+\t\tgit commit-graph write --reachable --split --size-multiple=10 &&\n+\t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n+\t\tls $graphdir/graph-*.graph >graph-files &&\n+\t\ttest_line_count = 1 graph-files\n+\t) &&\n+\tgit clone . merge-10-expire &&\n+\t(\n+\t\tcd merge-10-expire &&\n+\t\tgit config core.commitGraph true &&\n+\t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\t\ttest_commit 15 &&\n+\t\tgit commit-graph write --reachable --split --size-multiple=10 --expire-time=1980-01-01 &&\n+\t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n+\t\tls $graphdir/graph-*.graph >graph-files &&\n+\t\ttest_line_count = 3 graph-files\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"376764","messageId":"548ec69d017e10a90c4ce6b3be24729418cc002f.1559830527.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 13/14] commit-graph: verify chains with --shallow mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-06T14:15:40Z","receivedAt":"2019-06-06T14:15:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf we wrote a commit-graph chain, we only modified the tip file in\nthe chain. It is valuable to verify what we wrote, but not waste\ntime checking files we did not write.\n\nAdd a '--shallow' option to the 'git commit-graph verify' subcommand\nand check that it does not read the base graph in a two-file chain.\n\nMaking the verify subcommand read from a chain of commit-graphs takes\nsome rearranging of the builtin code.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 5 ++++-\n builtin/commit-graph.c | 27 +++++++++++++++++++--------\n commit-graph.c | 15 ++++++++++++---\n commit-graph.h | 6 ++++--\n t/t5323-split-commit-graph.sh | 21 +++++++++++++++++++++\n 5 files changed, 60 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 365e145e82..eb5e7865f0 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -10,7 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git commit-graph read' [--object-dir ]\n-'git commit-graph verify' [--object-dir ]\n+'git commit-graph verify' [--object-dir ] [--shallow]\n 'git commit-graph write' [--object-dir ]\n \n \n@@ -80,6 +80,9 @@ Used for debugging purposes.\n \n Read the commit-graph file and verify its contents against the object\n database. Used to check for corrupted data.\n++\n+With the `--shallow` option, only check the tip commit-graph file in\n+a chain of split commit-graphs.\n \n \n EXAMPLES\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 18e3b61fb6..7cde1e1aaa 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -5,17 +5,18 @@\n #include \"parse-options.h\"\n #include \"repository.h\"\n #include \"commit-graph.h\"\n+#include \"object-store.h\"\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n-\tN_(\"git commit-graph verify [--object-dir ]\"),\n+\tN_(\"git commit-graph verify [--object-dir ] [--shallow]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] \"),\n \tNULL\n };\n \n static const char * const builtin_commit_graph_verify_usage[] = {\n-\tN_(\"git commit-graph verify [--object-dir ]\"),\n+\tN_(\"git commit-graph verify [--object-dir ] [--shallow]\"),\n \tNULL\n };\n \n@@ -36,6 +37,7 @@ static struct opts_commit_graph {\n \tint stdin_commits;\n \tint append;\n \tint split;\n+\tint shallow;\n } opts;\n \n static int graph_verify(int argc, const char **argv)\n@@ -45,11 +47,14 @@ static int graph_verify(int argc, const char **argv)\n \tint open_ok;\n \tint fd;\n \tstruct stat st;\n+\tint flags = 0;\n \n \tstatic struct option builtin_commit_graph_verify_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n \t\t\t N_(\"dir\"),\n \t\t\t N_(\"The object directory to store the graph\")),\n+\t\tOPT_BOOL(0, \"shallow\", &opts.shallow,\n+\t\t\t N_(\"if the commit-graph is split, only verify the tip file\")),\n \t\tOPT_END(),\n \t};\n \n@@ -59,21 +64,27 @@ static int graph_verify(int argc, const char **argv)\n \n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n+\tif (opts.shallow)\n+\t\tflags |= COMMIT_GRAPH_VERIFY_SHALLOW;\n \n \tgraph_name = get_commit_graph_filename(opts.obj_dir);\n \topen_ok = open_commit_graph(graph_name, &fd, &st);\n-\tif (!open_ok && errno == ENOENT)\n-\t\treturn 0;\n-\tif (!open_ok)\n+\tif (!open_ok && errno != ENOENT)\n \t\tdie_errno(_(\"Could not open commit-graph '%s'\"), graph_name);\n-\tgraph = load_commit_graph_one_fd_st(fd, &st);\n+\n \tFREE_AND_NULL(graph_name);\n \n+\tif (open_ok)\n+\t\tgraph = load_commit_graph_one_fd_st(fd, &st);\n+\t else\n+\t\tgraph = read_commit_graph_one(the_repository, opts.obj_dir);\n+\n+\t/* Return failure if open_ok predicted success */\n \tif (!graph)\n-\t\treturn 1;\n+\t\treturn !!open_ok;\n \n \tUNLEAK(graph);\n-\treturn verify_commit_graph(the_repository, graph);\n+\treturn verify_commit_graph(the_repository, graph, flags);\n }\n \n static int graph_read(int argc, const char **argv)\ndiff --git a/commit-graph.c b/commit-graph.c\nindex abbfc12f1f..07856959c1 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -422,7 +422,7 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r, const\n \treturn graph_chain;\n }\n \n-static struct commit_graph *read_commit_graph_one(struct repository *r, const char *obj_dir)\n+struct commit_graph *read_commit_graph_one(struct repository *r, const char *obj_dir)\n {\n \tstruct commit_graph *g = load_commit_graph_v1(r, obj_dir);\n \n@@ -1884,7 +1884,7 @@ static void graph_report(const char *fmt, ...)\n #define GENERATION_ZERO_EXISTS 1\n #define GENERATION_NUMBER_EXISTS 2\n \n-int verify_commit_graph(struct repository *r, struct commit_graph *g)\n+int verify_commit_graph(struct repository *r, struct commit_graph *g, int flags)\n {\n \tuint32_t i, cur_fanout_pos = 0;\n \tstruct object_id prev_oid, cur_oid, checksum;\n@@ -1892,6 +1892,7 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \tstruct hashfile *f;\n \tint devnull;\n \tstruct progress *progress = NULL;\n+\tint local_error = 0;\n \n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n@@ -1986,6 +1987,9 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\t\t\tbreak;\n \t\t\t}\n \n+\t\t\t/* parse parent in case it is in a base graph */\n+\t\t\tparse_commit_in_graph_one(r, g, graph_parents->item);\n+\n \t\t\tif (!oideq(&graph_parents->item->object.oid, &odb_parents->item->object.oid))\n \t\t\t\tgraph_report(_(\"commit-graph parent for %s is %s != %s\"),\n \t\t\t\t\t oid_to_hex(&cur_oid),\n@@ -2037,7 +2041,12 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t}\n \tstop_progress(&progress);\n \n-\treturn verify_commit_graph_error;\n+\tlocal_error = verify_commit_graph_error;\n+\n+\tif (!(flags & COMMIT_GRAPH_VERIFY_SHALLOW) && g->base_graph)\n+\t\tlocal_error |= verify_commit_graph(r, g->base_graph, flags);\n+\n+\treturn local_error;\n }\n \n void free_commit_graph(struct commit_graph *g)\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 194acab2b7..84e5e91fc6 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -61,7 +61,7 @@ struct commit_graph {\n };\n \n struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st);\n-\n+struct commit_graph *read_commit_graph_one(struct repository *r, const char *obj_dir);\n struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t\t\t\t\tsize_t graph_size);\n \n@@ -89,7 +89,9 @@ int write_commit_graph(const char *obj_dir,\n \t\t unsigned int flags,\n \t\t const struct split_commit_graph_opts *split_opts);\n \n-int verify_commit_graph(struct repository *r, struct commit_graph *g);\n+#define COMMIT_GRAPH_VERIFY_SHALLOW\t(1 << 0)\n+\n+int verify_commit_graph(struct repository *r, struct commit_graph *g, int flags);\n \n void close_commit_graph(struct repository *);\n void free_commit_graph(struct commit_graph *);\ndiff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\nindex a915ef388c..b2bc07d72c 100755\n--- a/t/t5323-split-commit-graph.sh\n+++ b/t/t5323-split-commit-graph.sh\n@@ -204,4 +204,25 @@ test_expect_success 'test merge stragety constants' '\n \t)\n '\n \n+corrupt_file() {\n+\tfile=$1\n+\tpos=$2\n+\tdata=\"${3:-\\0}\"\n+\tprintf \"$data\" | dd of=\"$file\" bs=1 seek=\"$pos\" conv=notrunc\n+}\n+\n+test_expect_success 'verify shallow' '\n+\tgit clone . verify &&\n+\t(\n+\t\tcd verify &&\n+\t\tgit commit-graph verify &&\n+\t\tbase_file=$graphdir/graph-$(head -n 1 $graphdir/commit-graph-chain).graph &&\n+\t\tcorrupt_file \"$base_file\" 1760 \"\\01\" &&\n+\t\tgit commit-graph verify --shallow &&\n+\t\ttest_must_fail git commit-graph verify 2>test_err &&\n+\t\tgrep -v \"^+\" test_err >err &&\n+\t\ttest_i18ngrep \"incorrect checksum\" err\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"376765","messageId":"6084bbd16496d6328e0a50e604a7d05594e17f5e.1559830527.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 14/14] commit-graph: clean up chains after flattened write","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-06T14:15:43Z","receivedAt":"2019-06-06T14:15:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf we write a commit-graph file without the split option, then\nwe write to $OBJDIR/info/commit-graph and start to ignore\nthe chains in $OBJDIR/info/commit-graphs/.\n\nUnlink the commit-graph-chain file and expire the graph-{hash}.graph\nfiles in $OBJDIR/info/commit-graphs/ during every write.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 12 +++++++++---\n t/t5323-split-commit-graph.sh | 12 ++++++++++++\n 2 files changed, 21 insertions(+), 3 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 07856959c1..affa969e79 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1685,6 +1685,12 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \n \tif (ctx->split_opts && ctx->split_opts->expire_time)\n \t\texpire_time -= ctx->split_opts->expire_time;\n+\tif (!ctx->split) {\n+\t\tchar *chain_file_name = get_chain_filename(ctx->obj_dir);\n+\t\tunlink(chain_file_name);\n+\t\tfree(chain_file_name);\n+\t\tctx->num_commit_graphs_after = 0;\n+\t}\n \n \tstrbuf_addstr(&path, ctx->obj_dir);\n \tstrbuf_addstr(&path, \"/info/commit-graphs\");\n@@ -1838,10 +1844,10 @@ int write_commit_graph(const char *obj_dir,\n \n \tres = write_commit_graph_file(ctx);\n \n-\tif (ctx->split) {\n+\tif (ctx->split)\n \t\tmark_commit_graphs(ctx);\n-\t\texpire_commit_graphs(ctx);\n-\t}\n+\n+\texpire_commit_graphs(ctx);\n \n cleanup:\n \tfree(ctx->graph_name);\ndiff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\nindex b2bc07d72c..bd2e90e512 100755\n--- a/t/t5323-split-commit-graph.sh\n+++ b/t/t5323-split-commit-graph.sh\n@@ -204,6 +204,18 @@ test_expect_success 'test merge stragety constants' '\n \t)\n '\n \n+test_expect_success 'remove commit-graph-chain file after flattening' '\n+\tgit clone . flatten &&\n+\t(\n+\t\tcd flatten &&\n+\t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\t\tgit commit-graph write --reachable &&\n+\t\ttest_path_is_missing $graphdir/commit-graph-chain &&\n+\t\tls $graphdir >graph-files &&\n+\t\ttest_line_count = 0 graph-files\n+\t)\n+'\n+\n corrupt_file() {\n \tfile=$1\n \tpos=$2\n-- \ngitgitgadget\n"},{"id":"376767","messageId":"287d8877-cbeb-6712-68df-b231dac247e4@iee.org","threadId":"51052","inReplyTo":"d0dc154a27c80a8113ff47eb60a6b6f98fbeec91.1559830527.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 02/14] commit-graph: prepare for commit-graph chains","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2019-06-06T15:19:35Z","receivedAt":"2019-06-06T15:19:37Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"spelling nit in the commit message..\nOn 06/06/2019 15:15, Derrick Stolee via GitGitGadget wrote:\n> * graph position: the posiiton within the concatenated order\ns/posiiton/position/\n--\nPhilip\n"},{"id":"376770","messageId":"xmqqmuiur6qy.fsf@gitster-ct.c.googlers.com","threadId":"51052","inReplyTo":"pull.184.v4.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 00/14] Commit-graph: Write incremental files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-06T16:57:57Z","receivedAt":"2019-06-06T16:58:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> This is based on ds/commit-graph-write-refactor.\n>\n> Thanks, -Stolee\n>\n> [1] \n> https://github.com/git/git/commit/43d356180556180b4ef6ac232a14498a5bb2b446\n> commit-graph write: don't die if the existing graph is corrupt\n>\n> Derrick Stolee (14):\n> commit-graph: document commit-graph chains\n> commit-graph: prepare for commit-graph chains\n> commit-graph: rename commit_compare to oid_compare\n> commit-graph: load commit-graph chains\n> commit-graph: add base graphs chunk\n> commit-graph: rearrange chunk count logic\n> commit-graph: write commit-graph chains\n> commit-graph: add --split option to builtin\n> commit-graph: merge commit-graph chains\n> commit-graph: allow cross-alternate chains\n> commit-graph: expire commit-graph files\n> commit-graph: create options for split files\n> commit-graph: verify chains with --shallow mode\n> commit-graph: clean up chains after flattened write\n>\n> Documentation/git-commit-graph.txt | 26 +-\n> .../technical/commit-graph-format.txt | 11 +-\n> Documentation/technical/commit-graph.txt | 195 +++++\n> builtin/commit-graph.c | 53 +-\n> builtin/commit.c | 2 +-\n> builtin/gc.c | 3 +-\n> commit-graph.c | 794 +++++++++++++++++-\n> commit-graph.h | 25 +-\n> t/t5318-commit-graph.sh | 2 +-\n> t/t5323-split-commit-graph.sh | 240 ++++++\n\nThis breaks test-lint, as t5323 is already taken in 'pu' by another\ntopic. I tentatively moved it to 5234 for now.\n\nThanks.\n"},{"id":"376771","messageId":"37599a09-c89f-6ef7-1743-d21316348db2@iee.org","threadId":"51052","inReplyTo":"62b3fca582814c2cee050bab07533409e22e2c3c.1559830527.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 10/14] commit-graph: allow cross-alternate chains","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2019-06-06T17:00:30Z","receivedAt":"2019-06-06T17:00:31Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Is this a spelling nit?\nOn 06/06/2019 15:15, Derrick Stolee via GitGitGadget wrote:\n> 3. When writing a new commit-graph chain based on a commit-graph file\n> in another object directory, do not allow success if the base file\n> has of the name \"commit-graph\" instead of\n> \"commit-graphs/graoh-{hash}.graph\".\ns/graoh-/graph-/    ?\n--\nPhilip\n"},{"id":"376772","messageId":"909f355c-4c44-1556-71c7-168b28abddfc@gmail.com","threadId":"51052","inReplyTo":"6a6a0b72-e503-1fdd-e67e-2a84d825f89b@iee.org","subject":"Re: [PATCH v3 01/14] commit-graph: document commit-graph chains","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-06-06T17:09:11Z","receivedAt":"2019-06-06T17:09:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/6/2019 8:10 AM, Philip Oakley wrote:\n> Hi Derrick ,\n> \n> On 03/06/2019 17:03, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n>>\n>> Add a basic description of commit-graph chains.\n> Not really your problem, but I did notice that we don't actually explain what we mean here by a commit graph (before we start chaining them), and the distinction between the generic concept and the specific implementation.\n> \n> If I understand it correctly, the regular DAG (directed acyclic graph) already inherently contains the commit graph, showing the parent(s) of each commit. Hence, why do we need another? (which then needs explaining the what/why/how)\n> \n> So, in one sense, another commit chain is potentially duplicated redundant data. What hasn't been surfaced (for the reader coming later) is probably that accessing the DAG commit graph can be (a) slow, (b) one way (no child relationships), and (c) accesses large amounts of other data that isn't relevant to the task at hand.\n> \n> So the commit graph (implementation) is [I think] a fast, compact, sorted(?), list of commit oids that provides two way linkage through the commit graph (?) to allow fast queries within the Git codebase.\n> \n> The commit graph is normally considered immutable,\n\n_Commits_ are immutable. The graph grows as commits are added.\n\nThis may be the crux of your confusion, since the commit-graph\nfile can become stale as commits are added by 'git commit' or\n'git fetch'. The point of the incremental file format is to\nupdate the commit-graph data without rewriting the entire thing\nevery time.\n\nDoes this help clarify what's going on?\n\n> however the DAG commit graph can be extended by new commits, trimmed by branch deletion, rebasing, forced push, etc, or even reorganised via 'replace' or grafts commits, which must then be reflected in the commit graph (implementation).\n\nThese things create new commit objects, which would not be in\nthe commit-graph file until it is rewritten.\n\n> It just felt that there is a gap between the high level DAG, explained in the glossary, and the commit-graph That perhaps the technical/commit-graph.txt ought to summarise.\n\nI do think that technical/commit-graph.txt does summarize a lot\nabout the commit-graph _file_ and how that accelerates walks on\nthe high-level DAG. The added content in this patch does assume\na full understanding of the previous contents of that file.\n\nThanks,\n-Stolee\n\n"},{"id":"376776","messageId":"02c9a0e2-4505-14a4-fa68-958a7f0c7935@ramsayjones.plus.com","threadId":"51052","inReplyTo":"ac5586a20f7ed600f8912b1aed07dea5098ea046.1559830527.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 12/14] commit-graph: create options for split files","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2019-06-06T18:41:46Z","receivedAt":"2019-06-06T18:42:02Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"\n\nOn 06/06/2019 15:15, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The split commit-graph feature is now fully implemented, but needs\n> some more run-time configurability. Allow direct callers to 'git\n> commit-graph write --split' to specify the values used in the\n> merge strategy and the expire time.\n> \n> Update the documentation to specify these values.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-commit-graph.txt | 21 +++++++++++++-\n> Documentation/technical/commit-graph.txt | 7 +++--\n> builtin/commit-graph.c | 20 +++++++++++---\n> builtin/commit.c | 2 +-\n> builtin/gc.c | 3 +-\n> commit-graph.c | 35 ++++++++++++++++--------\n> commit-graph.h | 12 ++++++--\n> t/t5323-split-commit-graph.sh | 35 ++++++++++++++++++++++++\n> 8 files changed, 112 insertions(+), 23 deletions(-)\n> \n> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> index 624470e198..365e145e82 100644\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -26,7 +26,7 @@ OPTIONS\n> \tUse given directory for the location of packfiles and commit-graph\n> \tfile. This parameter exists to specify the location of an alternate\n> \tthat only has the objects directory, not a full `.git` directory. The\n> -\tcommit-graph file is expected to be at `/info/commit-graph` and\n> +\tcommit-graph file is expected to be in the `/info` directory and\n> \tthe packfiles are expected to be in `/pack`.\n> \n> \n> @@ -51,6 +51,25 @@ or `--stdin-packs`.)\n> +\n> With the `--append` option, include all commits that are present in the\n> existing commit-graph file.\n> ++\n> +With the `--split` option, write the commit-graph as a chain of multiple\n> +commit-graph files stored in `/info/commit-graphs`. The new commits\n> +not already in the commit-graph are added in a new \"tip\" file. This file\n> +is merged with the existing file if the following merge conditions are\n> +met:\n> ++\n> +* If `--size-multiple=` is not specified, let `X` equal 2. If the new\n> +tip file would have `N` commits and the previous tip has `M` commits and\n> +`X` times `N` is greater than `M`, instead merge the two files into a\n> +single file.\n> ++\n> +* If `--max-commits=` is specified with `M` a positive integer, and the\n> +new tip file would have more than `M` commits, then instead merge the new\n> +tip with the previous tip.\n> ++\n> +Finally, if `--expire-time=` is not specified, let `datetime`\n> +be the current time. After writing the split commit-graph, delete all\n> +unused commit-graph whose modified times are older than `datetime`.\n> \n> 'read'::\n> \n> diff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\n> index aed4350a59..729fbcb32f 100644\n> --- a/Documentation/technical/commit-graph.txt\n> +++ b/Documentation/technical/commit-graph.txt\n> @@ -248,10 +248,11 @@ When writing a set of commits that do not exist in the commit-graph stack of\n> height N, we default to creating a new file at level N + 1. We then decide to\n> merge with the Nth level if one of two conditions hold:\n> \n> - 1. The expected file size for level N + 1 is at least half the file size for\n> - level N.\n> + 1. `--size-multiple=` is specified or X = 2, and the number of commits in\n> + level N is less than X times the number of commits in level N + 1.\n> \n> - 2. Level N + 1 contains more than 64,0000 commits.\n> + 2. `--max-commits=` is specified with non-zero C and the number of commits\n> + in level N + 1 is more than C commits.\n> \n> This decision cascades down the levels: when we merge a level we create a new\n> set of commits that then compares to the next level.\n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index c2c07d3917..18e3b61fb6 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -10,7 +10,7 @@ static char const * const builtin_commit_graph_usage[] = {\n> \tN_(\"git commit-graph [--object-dir ]\"),\n> \tN_(\"git commit-graph read [--object-dir ]\"),\n> \tN_(\"git commit-graph verify [--object-dir ]\"),\n> -\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n> +\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] \"),\n> \tNULL\n> };\n> \n> @@ -25,7 +25,7 @@ static const char * const builtin_commit_graph_read_usage[] = {\n> };\n> \n> static const char * const builtin_commit_graph_write_usage[] = {\n> -\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n> +\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] \"),\n> \tNULL\n> };\n> \n> @@ -135,6 +135,7 @@ static int graph_read(int argc, const char **argv)\n> }\n> \n> extern int read_replace_refs;\n> +struct split_commit_graph_opts split_opts;\n\nThis 'split_opts' variable needs to be marked 'static'.\n\nThanks.\n\nATB,\nRamsay Jones\n\n"},{"id":"376782","messageId":"xmqqd0jqqu8p.fsf@gitster-ct.c.googlers.com","threadId":"51052","inReplyTo":"d0dc154a27c80a8113ff47eb60a6b6f98fbeec91.1559830527.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 02/14] commit-graph: prepare for commit-graph chains","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-06T21:28:06Z","receivedAt":"2019-06-06T21:28:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> +static void load_oid_from_graph(struct commit_graph *g, int pos, struct object_id *oid)\n> +{\n> +\tuint32_t lex_index;\n> +\n> +\tif (!g)\n> +\t\tBUG(\"NULL commit-graph\");\n> +\n> +\twhile (pos < g->num_commits_in_base)\n> +\t\tg = g->base_graph;\n\nIf a rogue caller calls this function with pos < 0, this loop would\neventually exhaust the chain and make g==NULL, I think. Shouldn't a\nsimilar assert exist upfront for \"if (pos < 0)\" or perhaps make pos\nunsigned int instead?\n\n> +\tif (pos >= g->num_commits + g->num_commits_in_base)\n> +\t\tBUG(\"position %d is beyond the scope of this commit-graph (%d local + %d base commits)\",\n> +\t\t pos, g->num_commits, g->num_commits_in_base);\n\nWhere does 'pos' typically come from? Taken from a parent commit\nfield of a commit-graph file or something like that?\n\nAs this is a \"BUG()\" and not a \"die()\", the callers of this function\nare responsible for making sure that, even if they are fed a set of\ncorrupt commit-graph files, they never feed 'pos' that is out of\nbounds to this function. The same is true for the other BUG() in\nfill_commit_in_graph().\n\nI am wondering if they have already sufficient protection, or if we\nare better off having die() instead saying \"corrupted commit graph\nfile\" or something. I dunno.\n\n"},{"id":"376785","messageId":"504867ac-557e-7c9c-3d0f-076c1a7f0e58@iee.org","threadId":"51052","inReplyTo":"909f355c-4c44-1556-71c7-168b28abddfc@gmail.com","subject":"Re: [PATCH v3 01/14] commit-graph: document commit-graph chains","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2019-06-06T21:59:00Z","receivedAt":"2019-06-06T21:59:03Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Hi Stolee,\n\nWe may be talking at cross-purposes.\nOn 06/06/2019 18:09, Derrick Stolee wrote:\n> On 6/6/2019 8:10 AM, Philip Oakley wrote:\n>> Hi Derrick ,\n>>\n>> On 03/06/2019 17:03, Derrick Stolee via GitGitGadget wrote:\n>>> From: Derrick Stolee \n>>>\n>>> Add a basic description of commit-graph chains.\n>> Not really your problem, but I did notice that we don't actually explain what we mean here by a commit graph (before we start chaining them), and the distinction between the generic concept and the specific implementation.\nThe purpose of my comment is here. We have/had not explained why we need \n(another) commit graph, either within the man page, or the technical \ndocs. It's an understanding gap.\n>>\n>> If I understand it correctly, the regular DAG (directed acyclic graph) already inherently contains the commit graph, showing the parent(s) of each commit. Hence, why do we need another? (which then needs explaining the what/why/how)\n>>\n>> So, in one sense, another commit chain is potentially duplicated redundant data. What hasn't been surfaced (for the reader coming later) is probably that accessing the DAG commit graph can be (a) slow, (b) one way (no child relationships), and (c) accesses large amounts of other data that isn't relevant to the task at hand.\n>>\n>> So the commit graph (implementation) is [I think] a fast, compact, sorted(?), list of commit oids that provides two way linkage through the commit graph (?) to allow fast queries within the Git codebase.\n>>\n>> The commit graph is normally considered immutable,\n> _Commits_ are immutable. The graph grows as commits are added.\nI was aware that individual commits are immutable. However the tips, \ngrafts and replacements can change the topology of the graph (especially \nthe grafts and replacements, hence the desire to have something that \nacts as a guide as to what, generally, is trying to be achieved).\n>\n> This may be the crux of your confusion, since the commit-graph\n> file can become stale as commits are added by 'git commit' or\n> 'git fetch'. The point of the incremental file format is to\n> update the commit-graph data without rewriting the entire thing\n> every time.\n>\n> Does this help clarify what's going on?\nOnly slightly, see below.\n>\n>> however the DAG commit graph can be extended by new commits, trimmed by branch deletion, rebasing, forced push, etc, or even reorganised via 'replace' or grafts commits, which must then be reflected in the commit graph (implementation).\n> These things create new commit objects, which would not be in\n> the commit-graph file until it is rewritten.\n>\n>> It just felt that there is a gap between the high level DAG, explained in the glossary, and the commit-graph That perhaps the technical/commit-graph.txt ought to summarise.\n> I do think that technical/commit-graph.txt does summarize a lot\n> about the commit-graph _file_ and how that accelerates walks on\n> the high-level DAG. The added content in this patch does assume\n> a full understanding of the previous contents of that file.\nThe current (prior) documentation is a bit Catch 22 with regard to that \nassumed full understanding, hence my comment, including the \"Not really \nyour problem,\" bit.\n>\n> Thanks,\n> -Stolee\n>\nPhilip\n"},{"id":"376786","messageId":"xmqq4l52qrt0.fsf@gitster-ct.c.googlers.com","threadId":"51052","inReplyTo":"ca670536dfe08277e953b2ef5f017dc32c1581d5.1559830527.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 04/14] commit-graph: load commit-graph chains","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-06T22:20:43Z","receivedAt":"2019-06-06T22:20:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> +\tif (stat(chain_name, &st)) {\n> ...\n> +\tif (st.st_size <= the_hash_algo->hexsz) {\n> ...\n> +\tfp = fopen(chain_name, \"r\");\n> +\tfree(chain_name);\n> +\n> +\tif (!fp)\n> +\t\treturn NULL;\n\nChecking for size before opening is an invitation for an unnecessary\nrace, isn't it? Perhaps fopen() followed by fstat() is a better\nalternative?\n\n> +\toids = xcalloc(st.st_size / (the_hash_algo->hexsz + 1), sizeof(struct object_id));\n> +\n> +\twhile (strbuf_getline_lf(&line, fp) != EOF && valid) {\n> +\t\tchar *graph_name;\n> +\t\tstruct commit_graph *g;\n\nI am imagining an evil tester growing the file after you called\nxcalloc() above ;-) Should we at least protect ourselves not to read\nmore than we planned to read originally? I would imagine that the\nideal code organization would be more like\n\n\tvalid = 1; have_read_all = 0;\n\n\tfopen();\n\tfstat(fp->fileno);\n\tcount = st.st_size / hashsize;\n\toids = xcalloc();\n\n\tfor (i = 0; i < count; i++) {\n \tif (getline() == EOF) {\n\t\t\thave_read_all = 1;\n\t\t\tbreak;\n\t\t}\n\t\tadd one graph based on the line;\n\t\tif (error) {\n\t\t\tvalid = 0;\n\t\t\tbreak;\n\t\t}\n\t}\n\tif (valid && i < count)\n\t\tdie(\"file truncated while we are reading?\");\n\tif (valid && !have_read_all)\n\t\tdie(\"file grew while we are reading?\");\n\nif we really care, but even without going to that extreme, at least\nwe should refrain from reading more than we allocated.\n\n"},{"id":"376818","messageId":"9fb274a2-a618-201f-9a45-bf0245bdf67b@gmail.com","threadId":"51052","inReplyTo":"xmqqmuiur6qy.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 00/14] Commit-graph: Write incremental files","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-06-07T12:37:07Z","receivedAt":"2019-06-07T12:37:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/6/2019 12:57 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:>\n>> t/t5323-split-commit-graph.sh | 240 ++++++\n> \n> This breaks test-lint, as t5323 is already taken in 'pu' by another\n> topic. I tentatively moved it to 5234 for now.\n\nSorry for not noticing this. I've moved the file to 5234 in my local\ncopy, so the next version will not hit this problem.\n\nThanks,\n-Stolee\n"},{"id":"376819","messageId":"4e2263fb-1499-7aae-d8f9-c0f789b83744@gmail.com","threadId":"51052","inReplyTo":"xmqqd0jqqu8p.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 02/14] commit-graph: prepare for commit-graph chains","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-06-07T12:44:42Z","receivedAt":"2019-06-07T12:44:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/6/2019 5:28 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> +static void load_oid_from_graph(struct commit_graph *g, int pos, struct object_id *oid)\n>> +{\n>> +\tuint32_t lex_index;\n>> +\n>> +\tif (!g)\n>> +\t\tBUG(\"NULL commit-graph\");\n>> +\n>> +\twhile (pos < g->num_commits_in_base)\n>> +\t\tg = g->base_graph;\n> \n> If a rogue caller calls this function with pos < 0, this loop would\n> eventually exhaust the chain and make g==NULL, I think. Shouldn't a\n> similar assert exist upfront for \"if (pos < 0)\" or perhaps make pos\n> unsigned int instead?\n\nThis is a good point. I will use 'uint32_t' since the only caller is\ninsert_parent_or_die() which has an unsigned position. I did notice that\ninsert_parent_or_die() uses \"uint64_t pos\" but its only caller passes a\n\"uint32_t edge_value\". The 32-bit value makes more sense because of the\nbuilt-in limits in the commit-graph format for number of commits. I'll\nchange insert_parent_or_die() to use 32-bits as well.\n\nAs for the while loop, it would also be good to rearrange the checks\nas follows:\n\n while (g && pos < g->num_commits_in_base)\n g = g->base_graph;\n\n if (!g)\n BUG(\"NULL commit-graph\");\n\n> \n>> +\tif (pos >= g->num_commits + g->num_commits_in_base)\n>> +\t\tBUG(\"position %d is beyond the scope of this commit-graph (%d local + %d base commits)\",\n>> +\t\t pos, g->num_commits, g->num_commits_in_base);\n> \n> Where does 'pos' typically come from? Taken from a parent commit\n> field of a commit-graph file or something like that?\n\nIt comes from the commit-graph file.\n\n> As this is a \"BUG()\" and not a \"die()\", the callers of this function\n> are responsible for making sure that, even if they are fed a set of\n> corrupt commit-graph files, they never feed 'pos' that is out of\n> bounds to this function. The same is true for the other BUG() in\n> fill_commit_in_graph().> \n> I am wondering if they have already sufficient protection, or if we\n> are better off having die() instead saying \"corrupted commit graph\n> file\" or something. I dunno.\n\nI can replace this with a die() that points to a corrupt commit-graph\nfile. Perhaps \"BUG()\" made more sense while I was developing the feature\nand wanted to tell myself why the error condition happened. That doesn't\nmake sense any more now that the feature is working.\n\nThanks,\n-Stolee\n\n"},{"id":"376820","messageId":"684d1184-71c4-92c2-bdb9-50cf4e547f1e@gmail.com","threadId":"51052","inReplyTo":"xmqq4l52qrt0.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 04/14] commit-graph: load commit-graph chains","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-06-07T12:53:57Z","receivedAt":"2019-06-07T12:54:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/6/2019 6:20 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> +\tif (stat(chain_name, &st)) {\n>> ...\n>> +\tif (st.st_size <= the_hash_algo->hexsz) {\n>> ...\n>> +\tfp = fopen(chain_name, \"r\");\n>> +\tfree(chain_name);\n>> +\n>> +\tif (!fp)\n>> +\t\treturn NULL;\n> \n> Checking for size before opening is an invitation for an unnecessary\n> race, isn't it? Perhaps fopen() followed by fstat() is a better\n> alternative?\n> \n>> +\toids = xcalloc(st.st_size / (the_hash_algo->hexsz + 1), sizeof(struct object_id));\n>> +\n>> +\twhile (strbuf_getline_lf(&line, fp) != EOF && valid) {\n>> +\t\tchar *graph_name;\n>> +\t\tstruct commit_graph *g;\n> \n> I am imagining an evil tester growing the file after you called\n> xcalloc() above ;-) Should we at least protect ourselves not to read\n> more than we planned to read originally? I would imagine that the\n> ideal code organization would be more like\n> \n> \tvalid = 1; have_read_all = 0;\n> \n> \tfopen();\n> \tfstat(fp->fileno);\n> \tcount = st.st_size / hashsize;\n> \toids = xcalloc();\n> \n> \tfor (i = 0; i < count; i++) {\n> \tif (getline() == EOF) {\n> \t\t\thave_read_all = 1;\n> \t\t\tbreak;\n> \t\t}\n> \t\tadd one graph based on the line;\n> \t\tif (error) {\n> \t\t\tvalid = 0;\n> \t\t\tbreak;\n> \t\t}\n> \t}\n> \tif (valid && i < count)\n> \t\tdie(\"file truncated while we are reading?\");\n> \tif (valid && !have_read_all)\n> \t\tdie(\"file grew while we are reading?\");\n> \n> if we really care, but even without going to that extreme, at least\n> we should refrain from reading more than we allocated.\n\nThanks! I clearly was not careful enough with this input, which should\nhave been easy to get right. I think all your points are valid. The\ncode looks much cleaner after rewriting it to care about counts and to\nproperly order the stat() call.\n\n-Stolee\n"},{"id":"376828","messageId":"xmqqmuitp8h6.fsf@gitster-ct.c.googlers.com","threadId":"51052","inReplyTo":"df44cbc1bf6e18c3682652c2de69de7a9caf5fc2.1559830527.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 05/14] commit-graph: add base graphs chunk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-07T18:15:49Z","receivedAt":"2019-06-07T18:15:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> - 1-byte (reserved for later use)\n> - Current clients should ignore this value.\n> + 1-byte number (B) of base commit-graphs\n> + We infer the length (H*B) of the Base Graphs chunk\n> + from this value.\n> \n> CHUNK LOOKUP:\n> \n> @@ -92,6 +93,12 @@ CHUNK DATA:\n> positions for the parents until reaching a value with the most-significant\n> bit on. The other bits correspond to the position of the last parent.\n> \n> + Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]\n> + This list of H-byte hashes describe a set of B commit-graph files that\n> + form a commit-graph chain. The graph position for the ith commit in this\n> + file's OID Lookup chunk is equal to i plus the number of commits in all\n> + base graphs. If B is non-zero, this chunk must exist.\n\nHmph, an obvious alternative design would be to make the base list\nself describing without using the \"reserved for future use\" byte,\nwhich would allow more than 256 bases (not that being able to use\n300 bases is necessarily a useful feature) and also leave the\nreserved byte unused.\n\nIt's not like being able to detect discrepancy (e.g. B!=0 but BASE\nchunk is missing, and/or BASE chunk appears but B==0) adds value by\noffering more protection against file corruption, so I am wondering\nwhy it is a good idea to consume the reserved byte for this.\n\n> +\tif (n && !g->chunk_base_graphs) {\n> +\t\twarning(_(\"commit-graph has no base graphs chunk\"));\n> +\t\treturn 0;\n> +\t}\n> +\nn> \twhile (n) {\n> \t\tn--;\n> +\n> +\t\tif (!oideq(&oids[n], &cur_g->oid) ||\n> +\t\t !hasheq(oids[n].hash, g->chunk_base_graphs + g->hash_len * n)) {\n\nHere, load_commit_graph_chain() that goes over the on-disk chain\nfile that lists graph files called us with 'n', which can run up to\nthe number of graph files listed in that file---and that number can\nbe more than what is recorded in the graph-list chunk, in which case\nwe are over-reading with this hasheq(), right?\n\nIt seems that parse_commit_graph() only cares about the beginning of\neach chunk, and a crafted graph file can record two chunks with a\ngap in between, or two chunks that overlap, and nobody would notice.\nIs that true?\n\nWasted space in the file between two chunks (i.e. a gap) is not\nnecessarily bad and may not be a warning-worthy thing, but two\nchunks that overlap is probably not a good idea and worth noticing.\nThe only sanity check it seems to do is to forbid chunks of the same\nkind from appearing twice.\n"},{"id":"376829","messageId":"xmqqimthp84t.fsf@gitster-ct.c.googlers.com","threadId":"51052","inReplyTo":"e65f9e841d577c13d26c1737ebb8a2cf65713be9.1559830527.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 06/14] commit-graph: rearrange chunk count logic","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-07T18:23:14Z","receivedAt":"2019-06-07T18:23:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> The number of chunks in a commit-graph file can change depending on\n> whether we need the Extra Edges Chunk. We are going to add more optional\n> chunks, and it will be helpful to rearrange this logic around the chunk\n> count before doing so.\n>\n> Specifically, we need to finalize the number of chunks before writing\n> the commit-graph header. Further, we also need to fill out the chunk\n> lookup table dynamically and using \"num_chunks\" as we add optional\n> chunks is useful for adding optional chunks in the future.\n\nYup. The resulting code may be slightly longer at this step than\nbefore, but it certainly is easier to follow the logic. Good.\n\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 35 +++++++++++++++++++++--------------\n> 1 file changed, 21 insertions(+), 14 deletions(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 909c841db5..80df6d6d9d 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -1206,7 +1206,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n> \tuint64_t chunk_offsets[5];\n> \tconst unsigned hashsz = the_hash_algo->rawsz;\n> \tstruct strbuf progress_title = STRBUF_INIT;\n> -\tint num_chunks = ctx->num_extra_edges ? 4 : 3;\n> +\tint num_chunks = 3;\n> \n> \tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n> \tif (safe_create_leading_directories(ctx->graph_name)) {\n> @@ -1219,27 +1219,34 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n> \thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n> \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n> \n> -\thashwrite_be32(f, GRAPH_SIGNATURE);\n> -\n> -\thashwrite_u8(f, GRAPH_VERSION);\n> -\thashwrite_u8(f, oid_version());\n> -\thashwrite_u8(f, num_chunks);\n> -\thashwrite_u8(f, 0); /* unused padding byte */\n> -\n> \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n> \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n> \tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n> -\tif (ctx->num_extra_edges)\n> -\t\tchunk_ids[3] = GRAPH_CHUNKID_EXTRAEDGES;\n> -\telse\n> -\t\tchunk_ids[3] = 0;\n> -\tchunk_ids[4] = 0;\n> +\tif (ctx->num_extra_edges) {\n> +\t\tchunk_ids[num_chunks] = GRAPH_CHUNKID_EXTRAEDGES;\n> +\t\tnum_chunks++;\n> +\t}\n> +\n> +\tchunk_ids[num_chunks] = 0;\n> \n> \tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n> \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n> \tchunk_offsets[2] = chunk_offsets[1] + hashsz * ctx->commits.nr;\n> \tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * ctx->commits.nr;\n> -\tchunk_offsets[4] = chunk_offsets[3] + 4 * ctx->num_extra_edges;\n> +\n> +\tnum_chunks = 3;\n> +\tif (ctx->num_extra_edges) {\n> +\t\tchunk_offsets[num_chunks + 1] = chunk_offsets[num_chunks] +\n> +\t\t\t\t\t\t4 * ctx->num_extra_edges;\n> +\t\tnum_chunks++;\n> +\t}\n> +\n> +\thashwrite_be32(f, GRAPH_SIGNATURE);\n> +\n> +\thashwrite_u8(f, GRAPH_VERSION);\n> +\thashwrite_u8(f, oid_version());\n> +\thashwrite_u8(f, num_chunks);\n> +\thashwrite_u8(f, 0);\n> \n> \tfor (i = 0; i <= num_chunks; i++) {\n> \t\tuint32_t chunk_write[3];\n"},{"id":"376830","messageId":"7a0bfaaa6d81cbfdfe3f46307de6231e98ef9ec9.1559932699.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 01/16] commit-graph: document commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-07T18:38:21Z","receivedAt":"2019-06-07T18:38:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd a basic description of commit-graph chains. More details about the\nfeature will be added as we add functionality. This introduction gives a\nhigh-level overview to the goals of the feature and the basic layout of\ncommit-graph chains.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 59 ++++++++++++++++++++++++\n 1 file changed, 59 insertions(+)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex fb53341d5e..1dca3bd8fe 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -127,6 +127,65 @@ Design Details\n helpful for these clones, anyway. The commit-graph will not be read or\n written when shallow commits are present.\n \n+Commit Graphs Chains\n+--------------------\n+\n+Typically, repos grow with near-constant velocity (commits per day). Over time,\n+the number of commits added by a fetch operation is much smaller than the\n+number of commits in the full history. By creating a \"chain\" of commit-graphs,\n+we enable fast writes of new commit data without rewriting the entire commit\n+history -- at least, most of the time.\n+\n+## File Layout\n+\n+A commit-graph chain uses multiple files, and we use a fixed naming convention\n+to organize these files. Each commit-graph file has a name\n+`$OBJDIR/info/commit-graphs/graph-{hash}.graph` where `{hash}` is the hex-\n+valued hash stored in the footer of that file (which is a hash of the file's\n+contents before that hash). For a chain of commit-graph files, a plain-text\n+file at `$OBJDIR/info/commit-graphs/commit-graph-chain` contains the\n+hashes for the files in order from \"lowest\" to \"highest\".\n+\n+For example, if the `commit-graph-chain` file contains the lines\n+\n+```\n+\t{hash0}\n+\t{hash1}\n+\t{hash2}\n+```\n+\n+then the commit-graph chain looks like the following diagram:\n+\n+ +-----------------------+\n+ | graph-{hash2}.graph |\n+ +-----------------------+\n+\t |\n+ +-----------------------+\n+ | |\n+ | graph-{hash1}.graph |\n+ | |\n+ +-----------------------+\n+\t |\n+ +-----------------------+\n+ | |\n+ | |\n+ | |\n+ | graph-{hash0}.graph |\n+ | |\n+ | |\n+ | |\n+ +-----------------------+\n+\n+Let X0 be the number of commits in `graph-{hash0}.graph`, X1 be the number of\n+commits in `graph-{hash1}.graph`, and X2 be the number of commits in\n+`graph-{hash2}.graph`. If a commit appears in position i in `graph-{hash2}.graph`,\n+then we interpret this as being the commit in position (X0 + X1 + i), and that\n+will be used as its \"graph position\". The commits in `graph-{hash2}.graph` use these\n+positions to refer to their parents, which may be in `graph-{hash1}.graph` or\n+`graph-{hash0}.graph`. We can navigate to an arbitrary commit in position j by checking\n+its containment in the intervals [0, X0), [X0, X0 + X1), [X0 + X1, X0 + X1 +\n+X2).\n+\n Related Links\n -------------\n [0] https://bugs.chromium.org/p/git/issues/detail?id=8\n-- \ngitgitgadget\n\n"},{"id":"376831","messageId":"pull.184.v5.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v5 00/16] Commit-graph: Write incremental files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-07T18:38:20Z","receivedAt":"2019-06-07T18:38:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This version is now ready for review.\n\nThe commit-graph is a valuable performance feature for repos with large\ncommit histories, but suffers from the same problem as git repack: it\nrewrites the entire file every time. This can be slow when there are\nmillions of commits, especially after we stopped reading from the\ncommit-graph file during a write in 43d3561 (commit-graph write: don't die\nif the existing graph is corrupt).\n\nInstead, create a \"chain\" of commit-graphs in the\n.git/objects/info/commit-graphs folder with name graph-{hash}.graph. The\nlist of hashes is given by the commit-graph-chain file, and also in a \"base\ngraph chunk\" in the commit-graph format. As we read a chain, we can verify\nthat the hashes match the trailing hash of each commit-graph we read along\nthe way and each hash below a level is expected by that graph file.\n\nWhen writing, we don't always want to add a new level to the stack. This\nwould eventually result in performance degradation, especially when\nsearching for a commit (before we know its graph position). We decide to\nmerge levels of the stack when the new commits we will write is less than\nhalf of the commits in the level above. This can be tweaked by the\n--size-multiple and --max-commits options.\n\nThe performance is necessarily amortized across multiple writes, so I tested\nby writing commit-graphs from the (non-rc) tags in the Linux repo. My test\nincluded 72 tags, and wrote everything reachable from the tag using \n--stdin-commits. Here are the overall perf numbers:\n\nwrite --stdin-commits: 8m 12s\nwrite --stdin-commits --split: 28s\nwrite --split && verify --shallow: 60s\n\nUpdates in V3:\n\n * git commit-graph verify now works on commit-graph chains. We do a simple\n test to check the behavior of a new --shallow option.\n \n \n * When someone writes a flat commit-graph, we now expire the old chain\n according to the expire time.\n \n \n * The \"max commits\" limit is no longer enabled by default, but instead is\n enabled by a --max-commits= option. Ignored if n=0.\n \n \n\nUpdates in V4:\n\nJohannes pointed out some test failures on the Windows platform. We found\nthat the tests were not running on Windows in the gitgitgadget PR builds,\nwhich is now resolved.\n\n * We need to close commit-graphs recursively down the chain. This prevented\n an unlink() from working because of an open handle.\n \n \n * Creating the alternates file used a path-specification that didn't work\n on Windows.\n \n \n * Renaming a file to the same name failed, but is probably related to the\n unlink() error mentioned above.\n \n \n\nUpdates in V5:\n\n * Responding to multiple items of feedback. Thanks Philip, Junio, and\n Ramsay!\n \n \n * Used the test coverage report to find holes in the test coverage. While\n adding tests, I found a bug in octopus merges. The fix is in the rewrite\n of \"deduplicate_commits()\" as \"sort_and_scan_merged_commits()\" and\n covered by the new tests.\n \n \n\nThis is based on ds/commit-graph-write-refactor.\n\nThanks, -Stolee\n\n[1] \nhttps://github.com/git/git/commit/43d356180556180b4ef6ac232a14498a5bb2b446\ncommit-graph write: don't die if the existing graph is corrupt\n\nDerrick Stolee (16):\n commit-graph: document commit-graph chains\n commit-graph: prepare for commit-graph chains\n commit-graph: rename commit_compare to oid_compare\n commit-graph: load commit-graph chains\n commit-graph: add base graphs chunk\n commit-graph: rearrange chunk count logic\n commit-graph: write commit-graph chains\n commit-graph: add --split option to builtin\n commit-graph: merge commit-graph chains\n commit-graph: allow cross-alternate chains\n commit-graph: expire commit-graph files\n commit-graph: create options for split files\n commit-graph: verify chains with --shallow mode\n commit-graph: clean up chains after flattened write\n commit-graph: test octopus merges with --split\n commit-graph: test --split across alternate without --split\n\n Documentation/git-commit-graph.txt | 26 +-\n .../technical/commit-graph-format.txt | 11 +-\n Documentation/technical/commit-graph.txt | 195 +++++\n builtin/commit-graph.c | 53 +-\n builtin/commit.c | 2 +-\n builtin/gc.c | 3 +-\n commit-graph.c | 799 +++++++++++++++++-\n commit-graph.h | 25 +-\n t/t5318-commit-graph.sh | 2 +-\n t/t5324-split-commit-graph.sh | 319 +++++++\n 10 files changed, 1365 insertions(+), 70 deletions(-)\n create mode 100755 t/t5324-split-commit-graph.sh\n\n\nbase-commit: 8520d7fc7c6edd4d71582c69a873436029b6cb1b\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-184%2Fderrickstolee%2Fgraph%2Fincremental-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-184/derrickstolee/graph/incremental-v5\nPull-Request: https://github.com/gitgitgadget/git/pull/184\n\nRange-diff vs v4:\n\n 1: b184919255 = 1: 7a0bfaaa6d commit-graph: document commit-graph chains\n 2: d0dc154a27 ! 2: ce139d80df commit-graph: prepare for commit-graph chains\n @@ -14,7 +14,7 @@\n * lexicographic index : the position within the lexicographic\n order in a single commit-graph file.\n \n - * graph position: the posiiton within the concatenated order\n + * graph position: the position within the concatenated order\n of multiple commit-graph files\n \n Given the lexicographic index of a commit in a graph, we can\n @@ -22,28 +22,53 @@\n the lower-level graphs. To find the lexicographic index of\n a commit, we subtract the number of commits in lower-level graphs.\n \n + While here, change insert_parent_or_die() to take a uint32_t\n + position, as that is the type used by its only caller and that\n + makes more sense with the limits in the commit-graph format.\n +\n Signed-off-by: Derrick Stolee \n \n diff --git a/commit-graph.c b/commit-graph.c\n --- a/commit-graph.c\n +++ b/commit-graph.c\n +@@\n + \treturn !!first_generation;\n + }\n + \n ++static void close_commit_graph_one(struct commit_graph *g)\n ++{\n ++\tif (!g)\n ++\t\treturn;\n ++\n ++\tclose_commit_graph_one(g->base_graph);\n ++\tfree_commit_graph(g);\n ++}\n ++\n + void close_commit_graph(struct repository *r)\n + {\n +-\tfree_commit_graph(r->objects->commit_graph);\n ++\tclose_commit_graph_one(r->objects->commit_graph);\n + \tr->objects->commit_graph = NULL;\n + }\n + \n @@\n \t\t\t g->chunk_oid_lookup, g->hash_len, pos);\n }\n \n -+static void load_oid_from_graph(struct commit_graph *g, int pos, struct object_id *oid)\n ++static void load_oid_from_graph(struct commit_graph *g,\n ++\t\t\t\tuint32_t pos,\n ++\t\t\t\tstruct object_id *oid)\n +{\n +\tuint32_t lex_index;\n +\n ++\twhile (g && pos < g->num_commits_in_base)\n ++\t\tg = g->base_graph;\n ++\n +\tif (!g)\n +\t\tBUG(\"NULL commit-graph\");\n +\n -+\twhile (pos < g->num_commits_in_base)\n -+\t\tg = g->base_graph;\n -+\n +\tif (pos >= g->num_commits + g->num_commits_in_base)\n -+\t\tBUG(\"position %d is beyond the scope of this commit-graph (%d local + %d base commits)\",\n -+\t\t pos, g->num_commits, g->num_commits_in_base);\n ++\t\tdie(_(\"invalid commit position. commit-graph is likely corrupt\"));\n +\n +\tlex_index = pos - g->num_commits_in_base;\n +\n @@ -52,14 +77,17 @@\n +\n static struct commit_list **insert_parent_or_die(struct repository *r,\n \t\t\t\t\t\t struct commit_graph *g,\n - \t\t\t\t\t\t uint64_t pos,\n -@@\n +-\t\t\t\t\t\t uint64_t pos,\n ++\t\t\t\t\t\t uint32_t pos,\n + \t\t\t\t\t\t struct commit_list **pptr)\n + {\n \tstruct commit *c;\n \tstruct object_id oid;\n \n -\tif (pos >= g->num_commits)\n +-\t\tdie(\"invalid parent position %\"PRIu64, pos);\n +\tif (pos >= g->num_commits + g->num_commits_in_base)\n - \t\tdie(\"invalid parent position %\"PRIu64, pos);\n ++\t\tdie(\"invalid parent position %\"PRIu32, pos);\n \n -\thashcpy(oid.hash, g->chunk_oid_lookup + g->hash_len * pos);\n +\tload_oid_from_graph(g, pos, &oid);\n @@ -95,8 +123,7 @@\n +\t\tg = g->base_graph;\n +\n +\tif (pos >= g->num_commits + g->num_commits_in_base)\n -+\t\tBUG(\"position %d is beyond the scope of this commit-graph (%d local + %d base commits)\",\n -+\t\t pos, g->num_commits, g->num_commits_in_base);\n ++\t\tdie(_(\"invalid commit position. commit-graph is likely corrupt\"));\n +\n +\t/*\n +\t * Store the \"full\" position, but then use the\n 3: f35b04224a = 3: 2470d2b548 commit-graph: rename commit_compare to oid_compare\n 4: ca670536df ! 4: fc3423046b commit-graph: load commit-graph chains\n @@ -82,32 +82,30 @@\n +\tstruct strbuf line = STRBUF_INIT;\n +\tstruct stat st;\n +\tstruct object_id *oids;\n -+\tint i = 0, valid = 1;\n ++\tint i = 0, valid = 1, count;\n +\tchar *chain_name = get_chain_filename(obj_dir);\n +\tFILE *fp;\n -+\n -+\tif (stat(chain_name, &st)) {\n -+\t\tfree(chain_name);\n -+\t\treturn NULL;\n -+\t}\n -+\n -+\tif (st.st_size <= the_hash_algo->hexsz) {\n -+\t\tfree(chain_name);\n -+\t\treturn NULL;\n -+\t}\n ++\tint stat_res;\n +\n +\tfp = fopen(chain_name, \"r\");\n ++\tstat_res = stat(chain_name, &st);\n +\tfree(chain_name);\n +\n -+\tif (!fp)\n ++\tif (!fp ||\n ++\t stat_res ||\n ++\t st.st_size <= the_hash_algo->hexsz)\n +\t\treturn NULL;\n +\n -+\toids = xcalloc(st.st_size / (the_hash_algo->hexsz + 1), sizeof(struct object_id));\n ++\tcount = st.st_size / (the_hash_algo->hexsz + 1);\n ++\toids = xcalloc(count, sizeof(struct object_id));\n +\n -+\twhile (strbuf_getline_lf(&line, fp) != EOF && valid) {\n ++\tfor (i = 0; i < count && valid; i++) {\n +\t\tchar *graph_name;\n +\t\tstruct commit_graph *g;\n +\n ++\t\tif (strbuf_getline_lf(&line, fp) == EOF)\n ++\t\t\tbreak;\n ++\n +\t\tif (get_oid_hex(line.buf, &oids[i])) {\n +\t\t\twarning(_(\"invalid commit-graph chain: line '%s' not a hash\"),\n +\t\t\t\tline.buf);\n 5: df44cbc1bf ! 5: d14c79f9d5 commit-graph: add base graphs chunk\n @@ -88,12 +88,12 @@\n \twhile (n) {\n \t\tn--;\n +\n -+\t\tif (!oideq(&oids[n], &cur_g->oid) ||\n ++\t\tif (!cur_g ||\n ++\t\t !oideq(&oids[n], &cur_g->oid) ||\n +\t\t !hasheq(oids[n].hash, g->chunk_base_graphs + g->hash_len * n)) {\n +\t\t\twarning(_(\"commit-graph chain does not match\"));\n +\t\t\treturn 0;\n +\t\t}\n -+\n +\n \t\tcur_g = cur_g->base_graph;\n \t}\n 6: e65f9e841d = 6: 5238bbbec3 commit-graph: rearrange chunk count logic\n 7: fe0aa343cd = 7: 02b0359571 commit-graph: write commit-graph chains\n 8: c42e683ef6 ! 8: a0330ebd2d commit-graph: add --split option to builtin\n @@ -9,7 +9,7 @@\n are not in the existing commit-graph or commit-graph chain. Later changes\n will allow merging the chain and expiring out-dated files.\n \n - Add a new test script (t5323-split-commit-graph.sh) that demonstrates this\n + Add a new test script (t5324-split-commit-graph.sh) that demonstrates this\n behavior.\n \n Helped-by: Johannes Schindelin \n @@ -92,10 +92,10 @@\n \t\t} else {\n \t\t\tchar *graph_name = get_commit_graph_filename(ctx->obj_dir);\n \n - diff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\n + diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\n new file mode 100755\n --- /dev/null\n - +++ b/t/t5323-split-commit-graph.sh\n + +++ b/t/t5324-split-commit-graph.sh\n @@\n +#!/bin/sh\n +\n 9: d065758454 ! 9: 28eccfa52b commit-graph: merge commit-graph chains\n @@ -242,30 +242,27 @@\n +\treturn oidcmp(&a->object.oid, &b->object.oid);\n +}\n +\n -+static void deduplicate_commits(struct write_commit_graph_context *ctx)\n ++static void sort_and_scan_merged_commits(struct write_commit_graph_context *ctx)\n +{\n -+\tuint32_t i, num_parents, last_distinct = 0, duplicates = 0;\n ++\tuint32_t i, num_parents;\n +\tstruct commit_list *parent;\n +\n +\tif (ctx->report_progress)\n +\t\tctx->progress = start_delayed_progress(\n -+\t\t\t\t\t_(\"De-duplicating merged commits\"),\n ++\t\t\t\t\t_(\"Scanning merged commits\"),\n +\t\t\t\t\tctx->commits.nr);\n +\n +\tQSORT(ctx->commits.list, ctx->commits.nr, commit_compare);\n +\n +\tctx->num_extra_edges = 0;\n -+\tfor (i = 1; i < ctx->commits.nr; i++) {\n ++\tfor (i = 0; i < ctx->commits.nr; i++) {\n +\t\tdisplay_progress(ctx->progress, i);\n +\n -+\t\tif (oideq(&ctx->commits.list[last_distinct]->object.oid,\n ++\t\tif (i && oideq(&ctx->commits.list[i - 1]->object.oid,\n +\t\t\t &ctx->commits.list[i]->object.oid)) {\n -+\t\t\tduplicates++;\n ++\t\t\tdie(_(\"unexpected duplicate commit id %s\"),\n ++\t\t\t oid_to_hex(&ctx->commits.list[i]->object.oid));\n +\t\t} else {\n -+\t\t\tif (duplicates)\n -+\t\t\t\tctx->commits.list[last_distinct + 1] = ctx->commits.list[i];\n -+\t\t\tlast_distinct++;\n -+\n +\t\t\tnum_parents = 0;\n +\t\t\tfor (parent = ctx->commits.list[i]->parents; parent; parent = parent->next)\n +\t\t\t\tnum_parents++;\n @@ -275,7 +272,6 @@\n +\t\t}\n +\t}\n +\n -+\tctx->commits.nr -= duplicates;\n +\tstop_progress(&ctx->progress);\n +}\n +\n @@ -308,7 +304,7 @@\n +\tif (ctx->new_base_graph)\n +\t\tctx->base_graph_name = xstrdup(ctx->new_base_graph->filename);\n +\n -+\tdeduplicate_commits(ctx);\n ++\tsort_and_scan_merged_commits(ctx);\n +}\n +\n int write_commit_graph(const char *obj_dir,\n @@ -340,9 +336,9 @@\n \n \tcompute_generation_numbers(ctx);\n \n - diff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\n - --- a/t/t5323-split-commit-graph.sh\n - +++ b/t/t5323-split-commit-graph.sh\n + diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\n + --- a/t/t5324-split-commit-graph.sh\n + +++ b/t/t5324-split-commit-graph.sh\n @@\n \n graph_git_behavior 'three-layer commit-graph: commit 11 vs 6' commits/11 commits/6\n 10: 62b3fca582 ! 10: 2093bab5b1 commit-graph: allow cross-alternate chains\n @@ -18,7 +18,7 @@\n 3. When writing a new commit-graph chain based on a commit-graph file\n in another object directory, do not allow success if the base file\n has of the name \"commit-graph\" instead of\n - \"commit-graphs/graoh-{hash}.graph\".\n + \"commit-graphs/graph-{hash}.graph\".\n \n Signed-off-by: Derrick Stolee \n \n @@ -91,15 +91,17 @@\n }\n \n @@\n - \toids = xcalloc(st.st_size / (the_hash_algo->hexsz + 1), sizeof(struct object_id));\n + \tcount = st.st_size / (the_hash_algo->hexsz + 1);\n + \toids = xcalloc(count, sizeof(struct object_id));\n \n - \twhile (strbuf_getline_lf(&line, fp) != EOF && valid) {\n +-\tfor (i = 0; i < count && valid; i++) {\n -\t\tchar *graph_name;\n -\t\tstruct commit_graph *g;\n ++\tfor (i = 0; i < count; i++) {\n +\t\tstruct object_directory *odb;\n \n - \t\tif (get_oid_hex(line.buf, &oids[i])) {\n - \t\t\twarning(_(\"invalid commit-graph chain: line '%s' not a hash\"),\n + \t\tif (strbuf_getline_lf(&line, fp) == EOF)\n + \t\t\tbreak;\n @@\n \t\t\tbreak;\n \t\t}\n @@ -107,6 +109,7 @@\n -\t\tgraph_name = get_split_graph_filename(obj_dir, line.buf);\n -\t\tg = load_commit_graph_one(graph_name);\n -\t\tfree(graph_name);\n ++\t\tvalid = 0;\n +\t\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n +\t\t\tchar *graph_name = get_split_graph_filename(odb->path, line.buf);\n +\t\t\tstruct commit_graph *g = load_commit_graph_one(graph_name);\n @@ -120,13 +123,18 @@\n +\t\t\tif (g) {\n +\t\t\t\tg->obj_dir = odb->path;\n +\n -+\t\t\t\tif (add_graph_to_chain(g, graph_chain, oids, i))\n ++\t\t\t\tif (add_graph_to_chain(g, graph_chain, oids, i)) {\n +\t\t\t\t\tgraph_chain = g;\n -+\t\t\t\telse\n -+\t\t\t\t\tvalid = 0;\n ++\t\t\t\t\tvalid = 1;\n ++\t\t\t\t}\n +\n +\t\t\t\tbreak;\n +\t\t\t}\n ++\t\t}\n ++\n ++\t\tif (!valid) {\n ++\t\t\twarning(_(\"unable to find all commit-graph files\"));\n ++\t\t\tbreak;\n +\t\t}\n \t}\n \n @@ -182,9 +190,9 @@\n \tuint32_t num_commits_in_base;\n \tstruct commit_graph *base_graph;\n \n - diff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\n - --- a/t/t5323-split-commit-graph.sh\n - +++ b/t/t5323-split-commit-graph.sh\n + diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\n + --- a/t/t5324-split-commit-graph.sh\n + +++ b/t/t5324-split-commit-graph.sh\n @@\n \tgraph_read_expect 12\n '\n @@ -195,7 +203,7 @@\n +\t(\n +\t\tcd fork &&\n +\t\trm .git/objects/info/commit-graph &&\n -+\t\techo \"$(pwd)\"/../.git/objects >.git/objects/info/alternates &&\n ++\t\techo \"$(pwd)/../.git/objects\" >.git/objects/info/alternates &&\n +\t\ttest_commit new-commit &&\n +\t\tgit commit-graph write --reachable --split &&\n +\t\ttest_path_is_file $graphdir/commit-graph-chain &&\n @@ -217,7 +225,7 @@\n +\t\tcd fork &&\n +\t\tgit config core.commitGraph true &&\n +\t\trm -rf $graphdir &&\n -+\t\techo \"$(pwd)\"/../.git/objects >.git/objects/info/alternates &&\n ++\t\techo \"$(pwd)/../.git/objects\" >.git/objects/info/alternates &&\n +\t\ttest_commit 13 &&\n +\t\tgit branch commits/13 &&\n +\t\tgit commit-graph write --reachable --split &&\n 11: b5aeeed909 ! 11: 554880e3d7 commit-graph: expire commit-graph files\n @@ -46,27 +46,7 @@\n --- a/commit-graph.c\n +++ b/commit-graph.c\n @@\n - \treturn !!first_generation;\n - }\n - \n -+static void close_commit_graph_one(struct commit_graph *g)\n -+{\n -+\tif (!g)\n -+\t\treturn;\n -+\n -+\tclose_commit_graph_one(g->base_graph);\n -+\tfree_commit_graph(g);\n -+}\n -+\n - void close_commit_graph(struct repository *r)\n - {\n --\tfree_commit_graph(r->objects->commit_graph);\n -+\tclose_commit_graph_one(r->objects->commit_graph);\n - \tr->objects->commit_graph = NULL;\n - }\n - \n -@@\n - \tdeduplicate_commits(ctx);\n + \tsort_and_scan_merged_commits(ctx);\n }\n \n +static void mark_commit_graphs(struct write_commit_graph_context *ctx)\n @@ -129,6 +109,7 @@\n +\n +\t\tif (!found)\n +\t\t\tunlink(path.buf);\n ++\n +\t}\n +}\n +\n @@ -148,9 +129,9 @@\n \tfree(ctx->graph_name);\n \tfree(ctx->commits.list);\n \n - diff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\n - --- a/t/t5323-split-commit-graph.sh\n - +++ b/t/t5323-split-commit-graph.sh\n + diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\n + --- a/t/t5324-split-commit-graph.sh\n + +++ b/t/t5324-split-commit-graph.sh\n @@\n \ttest_path_is_file $graphdir/commit-graph-chain &&\n \ttest_line_count = 2 $graphdir/commit-graph-chain &&\n 12: ac5586a20f ! 12: 66be8b03a8 commit-graph: create options for split files\n @@ -94,7 +94,7 @@\n }\n \n extern int read_replace_refs;\n -+struct split_commit_graph_opts split_opts;\n ++static struct split_commit_graph_opts split_opts;\n \n static int graph_write(int argc, const char **argv)\n {\n @@ -294,9 +294,9 @@\n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \n \n - diff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\n - --- a/t/t5323-split-commit-graph.sh\n - +++ b/t/t5323-split-commit-graph.sh\n + diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\n + --- a/t/t5324-split-commit-graph.sh\n + +++ b/t/t5324-split-commit-graph.sh\n @@\n \n graph_git_behavior 'alternate: commit 13 vs 6' commits/13 commits/6\n @@ -333,6 +333,18 @@\n +\t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n +\t\tls $graphdir/graph-*.graph >graph-files &&\n +\t\ttest_line_count = 3 graph-files\n ++\t) &&\n ++\tgit clone --no-hardlinks . max-commits &&\n ++\t(\n ++\t\tcd max-commits &&\n ++\t\tgit config core.commitGraph true &&\n ++\t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n ++\t\ttest_commit 16 &&\n ++\t\ttest_commit 17 &&\n ++\t\tgit commit-graph write --reachable --split --max-commits=1 &&\n ++\t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n ++\t\tls $graphdir/graph-*.graph >graph-files &&\n ++\t\ttest_line_count = 1 graph-files\n +\t)\n +'\n +\n 13: 548ec69d01 ! 13: 9fec4f9a36 commit-graph: verify chains with --shallow mode\n @@ -197,9 +197,9 @@\n void close_commit_graph(struct repository *);\n void free_commit_graph(struct commit_graph *);\n \n - diff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\n - --- a/t/t5323-split-commit-graph.sh\n - +++ b/t/t5323-split-commit-graph.sh\n + diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\n + --- a/t/t5324-split-commit-graph.sh\n + +++ b/t/t5324-split-commit-graph.sh\n @@\n \t)\n '\n @@ -211,18 +211,59 @@\n +\tprintf \"$data\" | dd of=\"$file\" bs=1 seek=\"$pos\" conv=notrunc\n +}\n +\n -+test_expect_success 'verify shallow' '\n -+\tgit clone . verify &&\n ++test_expect_success 'verify hashes along chain, even in shallow' '\n ++\tgit clone --no-hardlinks . verify &&\n +\t(\n +\t\tcd verify &&\n +\t\tgit commit-graph verify &&\n +\t\tbase_file=$graphdir/graph-$(head -n 1 $graphdir/commit-graph-chain).graph &&\n +\t\tcorrupt_file \"$base_file\" 1760 \"\\01\" &&\n ++\t\ttest_must_fail git commit-graph verify --shallow 2>test_err &&\n ++\t\tgrep -v \"^+\" test_err >err &&\n ++\t\ttest_i18ngrep \"incorrect checksum\" err\n ++\t)\n ++'\n ++\n ++test_expect_success 'verify --shallow does not check base contents' '\n ++\tgit clone --no-hardlinks . verify-shallow &&\n ++\t(\n ++\t\tcd verify-shallow &&\n ++\t\tgit commit-graph verify &&\n ++\t\tbase_file=$graphdir/graph-$(head -n 1 $graphdir/commit-graph-chain).graph &&\n ++\t\tcorrupt_file \"$base_file\" 1000 \"\\01\" &&\n +\t\tgit commit-graph verify --shallow &&\n +\t\ttest_must_fail git commit-graph verify 2>test_err &&\n +\t\tgrep -v \"^+\" test_err >err &&\n +\t\ttest_i18ngrep \"incorrect checksum\" err\n +\t)\n +'\n ++\n ++test_expect_success 'warn on base graph chunk incorrect' '\n ++\tgit clone --no-hardlinks . base-chunk &&\n ++\t(\n ++\t\tcd base-chunk &&\n ++\t\tgit commit-graph verify &&\n ++\t\tbase_file=$graphdir/graph-$(tail -n 1 $graphdir/commit-graph-chain).graph &&\n ++\t\tcorrupt_file \"$base_file\" 1376 \"\\01\" &&\n ++\t\tgit commit-graph verify --shallow 2>test_err &&\n ++\t\tgrep -v \"^+\" test_err >err &&\n ++\t\ttest_i18ngrep \"commit-graph chain does not match\" err\n ++\t)\n ++'\n ++\n ++test_expect_success 'verify after commit-graph-chain corruption' '\n ++\tgit clone --no-hardlinks . verify-chain &&\n ++\t(\n ++\t\tcd verify-chain &&\n ++\t\tcorrupt_file \"$graphdir/commit-graph-chain\" 60 \"G\" &&\n ++\t\tgit commit-graph verify 2>test_err &&\n ++\t\tgrep -v \"^+\" test_err >err &&\n ++\t\ttest_i18ngrep \"invalid commit-graph chain\" err &&\n ++\t\tcorrupt_file \"$graphdir/commit-graph-chain\" 60 \"A\" &&\n ++\t\tgit commit-graph verify 2>test_err &&\n ++\t\tgrep -v \"^+\" test_err >err &&\n ++\t\ttest_i18ngrep \"unable to find all commit-graph files\" err\n ++\t)\n ++'\n +\n test_done\n 14: 6084bbd164 ! 14: 795ea36ff4 commit-graph: clean up chains after flattened write\n @@ -42,9 +42,9 @@\n cleanup:\n \tfree(ctx->graph_name);\n \n - diff --git a/t/t5323-split-commit-graph.sh b/t/t5323-split-commit-graph.sh\n - --- a/t/t5323-split-commit-graph.sh\n - +++ b/t/t5323-split-commit-graph.sh\n + diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\n + --- a/t/t5324-split-commit-graph.sh\n + +++ b/t/t5324-split-commit-graph.sh\n @@\n \t)\n '\n -: ---------- > 15: 101792b92d commit-graph: test octopus merges with --split\n -: ---------- > 16: 84a3ff7c61 commit-graph: test --split across alternate without --split\n\n-- \ngitgitgadget\n"},{"id":"376832","messageId":"ce139d80df9c6f847d9a88be0d0ff35a2670c600.1559932699.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 02/16] commit-graph: prepare for commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-07T18:38:22Z","receivedAt":"2019-06-07T18:38:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTo prepare for a chain of commit-graph files, augment the\ncommit_graph struct to point to a base commit_graph. As we load\ncommits from the graph, we may actually want to read from a base\nfile according to the graph position.\n\nThe \"graph position\" of a commit is given by concatenating the\nlexicographic commit orders from each of the commit-graph files in\nthe chain. This means that we must distinguish two values:\n\n * lexicographic index : the position within the lexicographic\n order in a single commit-graph file.\n\n * graph position: the position within the concatenated order\n of multiple commit-graph files\n\nGiven the lexicographic index of a commit in a graph, we can\ncompute the graph position by adding the number of commits in\nthe lower-level graphs. To find the lexicographic index of\na commit, we subtract the number of commits in lower-level graphs.\n\nWhile here, change insert_parent_or_die() to take a uint32_t\nposition, as that is the type used by its only caller and that\nmakes more sense with the limits in the commit-graph format.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 89 +++++++++++++++++++++++++++++++++++++++++++-------\n commit-graph.h | 3 ++\n 2 files changed, 81 insertions(+), 11 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 7723156964..8c3598037b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -359,9 +359,18 @@ int generation_numbers_enabled(struct repository *r)\n \treturn !!first_generation;\n }\n \n+static void close_commit_graph_one(struct commit_graph *g)\n+{\n+\tif (!g)\n+\t\treturn;\n+\n+\tclose_commit_graph_one(g->base_graph);\n+\tfree_commit_graph(g);\n+}\n+\n void close_commit_graph(struct repository *r)\n {\n-\tfree_commit_graph(r->objects->commit_graph);\n+\tclose_commit_graph_one(r->objects->commit_graph);\n \tr->objects->commit_graph = NULL;\n }\n \n@@ -371,18 +380,38 @@ static int bsearch_graph(struct commit_graph *g, struct object_id *oid, uint32_t\n \t\t\t g->chunk_oid_lookup, g->hash_len, pos);\n }\n \n+static void load_oid_from_graph(struct commit_graph *g,\n+\t\t\t\tuint32_t pos,\n+\t\t\t\tstruct object_id *oid)\n+{\n+\tuint32_t lex_index;\n+\n+\twhile (g && pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tif (!g)\n+\t\tBUG(\"NULL commit-graph\");\n+\n+\tif (pos >= g->num_commits + g->num_commits_in_base)\n+\t\tdie(_(\"invalid commit position. commit-graph is likely corrupt\"));\n+\n+\tlex_index = pos - g->num_commits_in_base;\n+\n+\thashcpy(oid->hash, g->chunk_oid_lookup + g->hash_len * lex_index);\n+}\n+\n static struct commit_list **insert_parent_or_die(struct repository *r,\n \t\t\t\t\t\t struct commit_graph *g,\n-\t\t\t\t\t\t uint64_t pos,\n+\t\t\t\t\t\t uint32_t pos,\n \t\t\t\t\t\t struct commit_list **pptr)\n {\n \tstruct commit *c;\n \tstruct object_id oid;\n \n-\tif (pos >= g->num_commits)\n-\t\tdie(\"invalid parent position %\"PRIu64, pos);\n+\tif (pos >= g->num_commits + g->num_commits_in_base)\n+\t\tdie(\"invalid parent position %\"PRIu32, pos);\n \n-\thashcpy(oid.hash, g->chunk_oid_lookup + g->hash_len * pos);\n+\tload_oid_from_graph(g, pos, &oid);\n \tc = lookup_commit(r, &oid);\n \tif (!c)\n \t\tdie(_(\"could not find commit %s\"), oid_to_hex(&oid));\n@@ -392,7 +421,14 @@ static struct commit_list **insert_parent_or_die(struct repository *r,\n \n static void fill_commit_graph_info(struct commit *item, struct commit_graph *g, uint32_t pos)\n {\n-\tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n+\tconst unsigned char *commit_data;\n+\tuint32_t lex_index;\n+\n+\twhile (pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tlex_index = pos - g->num_commits_in_base;\n+\tcommit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * lex_index;\n \titem->graph_pos = pos;\n \titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n }\n@@ -405,10 +441,25 @@ static int fill_commit_in_graph(struct repository *r,\n \tuint32_t *parent_data_ptr;\n \tuint64_t date_low, date_high;\n \tstruct commit_list **pptr;\n-\tconst unsigned char *commit_data = g->chunk_commit_data + (g->hash_len + 16) * pos;\n+\tconst unsigned char *commit_data;\n+\tuint32_t lex_index;\n \n-\titem->object.parsed = 1;\n+\twhile (pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tif (pos >= g->num_commits + g->num_commits_in_base)\n+\t\tdie(_(\"invalid commit position. commit-graph is likely corrupt\"));\n+\n+\t/*\n+\t * Store the \"full\" position, but then use the\n+\t * \"local\" position for the rest of the calculation.\n+\t */\n \titem->graph_pos = pos;\n+\tlex_index = pos - g->num_commits_in_base;\n+\n+\tcommit_data = g->chunk_commit_data + (g->hash_len + 16) * lex_index;\n+\n+\titem->object.parsed = 1;\n \n \titem->maybe_tree = NULL;\n \n@@ -452,7 +503,18 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \t\t*pos = item->graph_pos;\n \t\treturn 1;\n \t} else {\n-\t\treturn bsearch_graph(g, &(item->object.oid), pos);\n+\t\tstruct commit_graph *cur_g = g;\n+\t\tuint32_t lex_index;\n+\n+\t\twhile (cur_g && !bsearch_graph(cur_g, &(item->object.oid), &lex_index))\n+\t\t\tcur_g = cur_g->base_graph;\n+\n+\t\tif (cur_g) {\n+\t\t\t*pos = lex_index + cur_g->num_commits_in_base;\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\treturn 0;\n \t}\n }\n \n@@ -492,8 +554,13 @@ static struct tree *load_tree_for_commit(struct repository *r,\n \t\t\t\t\t struct commit *c)\n {\n \tstruct object_id oid;\n-\tconst unsigned char *commit_data = g->chunk_commit_data +\n-\t\t\t\t\t GRAPH_DATA_WIDTH * (c->graph_pos);\n+\tconst unsigned char *commit_data;\n+\n+\twhile (c->graph_pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tcommit_data = g->chunk_commit_data +\n+\t\t\tGRAPH_DATA_WIDTH * (c->graph_pos - g->num_commits_in_base);\n \n \thashcpy(oid.hash, commit_data);\n \tc->maybe_tree = lookup_tree(r, &oid);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 70f4caf0c7..f9fe32ebe3 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -48,6 +48,9 @@ struct commit_graph {\n \tuint32_t num_commits;\n \tstruct object_id oid;\n \n+\tuint32_t num_commits_in_base;\n+\tstruct commit_graph *base_graph;\n+\n \tconst uint32_t *chunk_oid_fanout;\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_commit_data;\n-- \ngitgitgadget\n\n"},{"id":"376833","messageId":"02b0359571516d457dfe31ccf9e2cb01b8d42050.1559932699.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 07/16] commit-graph: write commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-07T18:38:25Z","receivedAt":"2019-06-07T18:38:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nExtend write_commit_graph() to write a commit-graph chain when given the\nCOMMIT_GRAPH_SPLIT flag.\n\nThis implementation is purposefully simplistic in how it creates a new\nchain. The commits not already in the chain are added to a new tip\ncommit-graph file.\n\nMuch of the logic around writing a graph-{hash}.graph file and updating\nthe commit-graph-chain file is the same as the commit-graph file case.\nHowever, there are several places where we need to do some extra logic\nin the split case.\n\nTrack the list of graph filenames before and after the planned write.\nThis will be more important when we start merging graph files, but it\nalso allows us to upgrade our commit-graph file to the appropriate\ngraph-{hash}.graph file when we upgrade to a chain of commit-graphs.\n\nNote that we use the eighth byte of the commit-graph header to store the\nnumber of base graph files. This determines the length of the base\ngraphs chunk.\n\nA subtle change of behavior with the new logic is that we do not write a\ncommit-graph if we our commit list is empty. This extends to the typical\ncase, which is reflected in t5318-commit-graph.sh.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 286 ++++++++++++++++++++++++++++++++++++++--\n commit-graph.h | 2 +\n t/t5318-commit-graph.sh | 2 +-\n 3 files changed, 278 insertions(+), 12 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 5f25dff193..eb6d79567a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -300,12 +300,18 @@ static struct commit_graph *load_commit_graph_one(const char *graph_file)\n \n \tstruct stat st;\n \tint fd;\n+\tstruct commit_graph *g;\n \tint open_ok = open_commit_graph(graph_file, &fd, &st);\n \n \tif (!open_ok)\n \t\treturn NULL;\n \n-\treturn load_commit_graph_one_fd_st(fd, &st);\n+\tg = load_commit_graph_one_fd_st(fd, &st);\n+\n+\tif (g)\n+\t\tg->filename = xstrdup(graph_file);\n+\n+\treturn g;\n }\n \n static struct commit_graph *load_commit_graph_v1(struct repository *r, const char *obj_dir)\n@@ -730,8 +736,19 @@ struct write_commit_graph_context {\n \tstruct progress *progress;\n \tint progress_done;\n \tuint64_t progress_cnt;\n+\n+\tchar *base_graph_name;\n+\tint num_commit_graphs_before;\n+\tint num_commit_graphs_after;\n+\tchar **commit_graph_filenames_before;\n+\tchar **commit_graph_filenames_after;\n+\tchar **commit_graph_hash_after;\n+\tuint32_t new_num_commits_in_base;\n+\tstruct commit_graph *new_base_graph;\n+\n \tunsigned append:1,\n-\t\t report_progress:1;\n+\t\t report_progress:1,\n+\t\t split:1;\n };\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n@@ -801,6 +818,16 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t commit_to_sha1);\n \n+\t\t\tif (edge_value >= 0)\n+\t\t\t\tedge_value += ctx->new_num_commits_in_base;\n+\t\t\telse {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (find_commit_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n+\t\t\t\t\tedge_value = pos;\n+\t\t\t}\n+\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n \t\t\t\t oid_to_hex(&parent->item->object.oid),\n@@ -821,6 +848,17 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\t\t\t ctx->commits.list,\n \t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t commit_to_sha1);\n+\n+\t\t\tif (edge_value >= 0)\n+\t\t\t\tedge_value += ctx->new_num_commits_in_base;\n+\t\t\telse {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (find_commit_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n+\t\t\t\t\tedge_value = pos;\n+\t\t\t}\n+\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n \t\t\t\t oid_to_hex(&parent->item->object.oid),\n@@ -878,6 +916,16 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n \t\t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t\t commit_to_sha1);\n \n+\t\t\tif (edge_value >= 0)\n+\t\t\t\tedge_value += ctx->new_num_commits_in_base;\n+\t\t\telse {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (find_commit_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n+\t\t\t\t\tedge_value = pos;\n+\t\t\t}\n+\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n \t\t\t\t oid_to_hex(&parent->item->object.oid),\n@@ -969,7 +1017,13 @@ static void close_reachable(struct write_commit_graph_context *ctx)\n \t\tdisplay_progress(ctx->progress, i + 1);\n \t\tcommit = lookup_commit(ctx->r, &ctx->oids.list[i]);\n \n-\t\tif (commit && !parse_commit_no_graph(commit))\n+\t\tif (!commit)\n+\t\t\tcontinue;\n+\t\tif (ctx->split) {\n+\t\t\tif (!parse_commit(commit) &&\n+\t\t\t commit->graph_pos == COMMIT_NOT_FROM_GRAPH)\n+\t\t\t\tadd_missing_parents(ctx, commit);\n+\t\t} else if (!parse_commit_no_graph(commit))\n \t\t\tadd_missing_parents(ctx, commit);\n \t}\n \tstop_progress(&ctx->progress);\n@@ -1165,8 +1219,16 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n \n \tfor (i = 1; i < ctx->oids.nr; i++) {\n \t\tdisplay_progress(ctx->progress, i + 1);\n-\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n+\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i])) {\n+\t\t\tif (ctx->split) {\n+\t\t\t\tstruct commit *c = lookup_commit(ctx->r, &ctx->oids.list[i]);\n+\n+\t\t\t\tif (!c || c->graph_pos != COMMIT_NOT_FROM_GRAPH)\n+\t\t\t\t\tcontinue;\n+\t\t\t}\n+\n \t\t\tcount_distinct++;\n+\t\t}\n \t}\n \tstop_progress(&ctx->progress);\n \n@@ -1189,7 +1251,13 @@ static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n \t\tif (i > 0 && oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n \t\t\tcontinue;\n \n+\t\tALLOC_GROW(ctx->commits.list, ctx->commits.nr + 1, ctx->commits.alloc);\n \t\tctx->commits.list[ctx->commits.nr] = lookup_commit(ctx->r, &ctx->oids.list[i]);\n+\n+\t\tif (ctx->split &&\n+\t\t ctx->commits.list[ctx->commits.nr]->graph_pos != COMMIT_NOT_FROM_GRAPH)\n+\t\t\tcontinue;\n+\n \t\tparse_commit_no_graph(ctx->commits.list[ctx->commits.nr]);\n \n \t\tfor (parent = ctx->commits.list[ctx->commits.nr]->parents;\n@@ -1204,18 +1272,86 @@ static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n+static int write_graph_chunk_base_1(struct hashfile *f,\n+\t\t\t\t struct commit_graph *g)\n+{\n+\tint num = 0;\n+\n+\tif (!g)\n+\t\treturn 0;\n+\n+\tnum = write_graph_chunk_base_1(f, g->base_graph);\n+\thashwrite(f, g->oid.hash, the_hash_algo->rawsz);\n+\treturn num + 1;\n+}\n+\n+static int write_graph_chunk_base(struct hashfile *f,\n+\t\t\t\t struct write_commit_graph_context *ctx)\n+{\n+\tint num = write_graph_chunk_base_1(f, ctx->new_base_graph);\n+\n+\tif (num != ctx->num_commit_graphs_after - 1) {\n+\t\terror(_(\"failed to write correct number of base graph ids\"));\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void init_commit_graph_chain(struct write_commit_graph_context *ctx)\n+{\n+\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\tuint32_t i;\n+\n+\tctx->new_base_graph = g;\n+\tctx->base_graph_name = xstrdup(g->filename);\n+\tctx->new_num_commits_in_base = g->num_commits + g->num_commits_in_base;\n+\n+\tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n+\n+\tALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);\n+\tALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);\n+\n+\tfor (i = 0; i < ctx->num_commit_graphs_before - 1; i++)\n+\t\tctx->commit_graph_filenames_after[i] = xstrdup(ctx->commit_graph_filenames_before[i]);\n+\n+\tif (ctx->num_commit_graphs_before)\n+\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_before - 1] =\n+\t\t\tget_split_graph_filename(ctx->obj_dir, oid_to_hex(&g->oid));\n+\n+\ti = ctx->num_commit_graphs_before - 1;\n+\n+\twhile (g) {\n+\t\tctx->commit_graph_hash_after[i] = xstrdup(oid_to_hex(&g->oid));\n+\t\ti--;\n+\t\tg = g->base_graph;\n+\t}\n+}\n+\n static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n+\tint fd;\n \tstruct hashfile *f;\n \tstruct lock_file lk = LOCK_INIT;\n-\tuint32_t chunk_ids[5];\n-\tuint64_t chunk_offsets[5];\n+\tuint32_t chunk_ids[6];\n+\tuint64_t chunk_offsets[6];\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tint num_chunks = 3;\n+\tstruct object_id file_hash;\n+\n+\tif (ctx->split) {\n+\t\tstruct strbuf tmp_file = STRBUF_INIT;\n+\n+\t\tstrbuf_addf(&tmp_file,\n+\t\t\t \"%s/info/commit-graphs/tmp_graph_XXXXXX\",\n+\t\t\t ctx->obj_dir);\n+\t\tctx->graph_name = strbuf_detach(&tmp_file, NULL);\n+\t} else {\n+\t\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n+\t}\n \n-\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n \tif (safe_create_leading_directories(ctx->graph_name)) {\n \t\tUNLEAK(ctx->graph_name);\n \t\terror(_(\"unable to create leading directories of %s\"),\n@@ -1223,8 +1359,23 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\treturn errno;\n \t}\n \n-\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n-\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n+\tif (ctx->split) {\n+\t\tchar *lock_name = get_chain_filename(ctx->obj_dir);\n+\n+\t\thold_lock_file_for_update(&lk, lock_name, LOCK_DIE_ON_ERROR);\n+\n+\t\tfd = git_mkstemp_mode(ctx->graph_name, 0444);\n+\t\tif (fd < 0) {\n+\t\t\terror(_(\"unable to create '%s'\"), ctx->graph_name);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tf = hashfd(fd, ctx->graph_name);\n+\t} else {\n+\t\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n+\t\tfd = lk.tempfile->fd;\n+\t\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n+\t}\n \n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n@@ -1233,6 +1384,10 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\tchunk_ids[num_chunks] = GRAPH_CHUNKID_EXTRAEDGES;\n \t\tnum_chunks++;\n \t}\n+\tif (ctx->num_commit_graphs_after > 1) {\n+\t\tchunk_ids[num_chunks] = GRAPH_CHUNKID_BASE;\n+\t\tnum_chunks++;\n+\t}\n \n \tchunk_ids[num_chunks] = 0;\n \n@@ -1247,13 +1402,18 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\t\t\t\t4 * ctx->num_extra_edges;\n \t\tnum_chunks++;\n \t}\n+\tif (ctx->num_commit_graphs_after > 1) {\n+\t\tchunk_offsets[num_chunks + 1] = chunk_offsets[num_chunks] +\n+\t\t\t\t\t\thashsz * (ctx->num_commit_graphs_after - 1);\n+\t\tnum_chunks++;\n+\t}\n \n \thashwrite_be32(f, GRAPH_SIGNATURE);\n \n \thashwrite_u8(f, GRAPH_VERSION);\n \thashwrite_u8(f, oid_version());\n \thashwrite_u8(f, num_chunks);\n-\thashwrite_u8(f, 0);\n+\thashwrite_u8(f, ctx->num_commit_graphs_after - 1);\n \n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tuint32_t chunk_write[3];\n@@ -1279,11 +1439,67 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \twrite_graph_chunk_data(f, hashsz, ctx);\n \tif (ctx->num_extra_edges)\n \t\twrite_graph_chunk_extra_edges(f, ctx);\n+\tif (ctx->num_commit_graphs_after > 1 &&\n+\t write_graph_chunk_base(f, ctx)) {\n+\t\treturn -1;\n+\t}\n \tstop_progress(&ctx->progress);\n \tstrbuf_release(&progress_title);\n \n+\tif (ctx->split && ctx->base_graph_name && ctx->num_commit_graphs_after > 1) {\n+\t\tchar *new_base_hash = xstrdup(oid_to_hex(&ctx->new_base_graph->oid));\n+\t\tchar *new_base_name = get_split_graph_filename(ctx->obj_dir, new_base_hash);\n+\n+\t\tfree(ctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2]);\n+\t\tfree(ctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 2]);\n+\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2] = new_base_name;\n+\t\tctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 2] = new_base_hash;\n+\t}\n+\n \tclose_commit_graph(ctx->r);\n-\tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n+\tfinalize_hashfile(f, file_hash.hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n+\n+\tif (ctx->split) {\n+\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n+\t\tchar *final_graph_name;\n+\t\tint result;\n+\n+\t\tclose(fd);\n+\n+\t\tif (!chainf) {\n+\t\t\terror(_(\"unable to open commit-graph chain file\"));\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (ctx->base_graph_name) {\n+\t\t\tresult = rename(ctx->base_graph_name,\n+\t\t\t\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2]);\n+\n+\t\t\tif (result) {\n+\t\t\t\terror(_(\"failed to rename base commit-graph file\"));\n+\t\t\t\treturn -1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tchar *graph_name = get_commit_graph_filename(ctx->obj_dir);\n+\t\t\tunlink(graph_name);\n+\t\t}\n+\n+\t\tctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 1] = xstrdup(oid_to_hex(&file_hash));\n+\t\tfinal_graph_name = get_split_graph_filename(ctx->obj_dir,\n+\t\t\t\t\tctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 1]);\n+\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 1] = final_graph_name;\n+\n+\t\tresult = rename(ctx->graph_name, final_graph_name);\n+\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++)\n+\t\t\tfprintf(lk.tempfile->fp, \"%s\\n\", ctx->commit_graph_hash_after[i]);\n+\n+\t\tif (result) {\n+\t\t\terror(_(\"failed to rename temporary commit-graph file\"));\n+\t\t\treturn -1;\n+\t\t}\n+\t}\n+\n \tcommit_lock_file(&lk);\n \n \treturn 0;\n@@ -1306,6 +1522,30 @@ int write_commit_graph(const char *obj_dir,\n \tctx->obj_dir = obj_dir;\n \tctx->append = flags & COMMIT_GRAPH_APPEND ? 1 : 0;\n \tctx->report_progress = flags & COMMIT_GRAPH_PROGRESS ? 1 : 0;\n+\tctx->split = flags & COMMIT_GRAPH_SPLIT ? 1 : 0;\n+\n+\tif (ctx->split) {\n+\t\tstruct commit_graph *g;\n+\t\tprepare_commit_graph(ctx->r);\n+\n+\t\tg = ctx->r->objects->commit_graph;\n+\n+\t\twhile (g) {\n+\t\t\tctx->num_commit_graphs_before++;\n+\t\t\tg = g->base_graph;\n+\t\t}\n+\n+\t\tif (ctx->num_commit_graphs_before) {\n+\t\t\tALLOC_ARRAY(ctx->commit_graph_filenames_before, ctx->num_commit_graphs_before);\n+\t\t\ti = ctx->num_commit_graphs_before;\n+\t\t\tg = ctx->r->objects->commit_graph;\n+\n+\t\t\twhile (g) {\n+\t\t\t\tctx->commit_graph_filenames_before[--i] = xstrdup(g->filename);\n+\t\t\t\tg = g->base_graph;\n+\t\t\t}\n+\t\t}\n+\t}\n \n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n@@ -1360,6 +1600,14 @@ int write_commit_graph(const char *obj_dir,\n \t\tgoto cleanup;\n \t}\n \n+\tif (!ctx->commits.nr)\n+\t\tgoto cleanup;\n+\n+\tif (ctx->split)\n+\t\tinit_commit_graph_chain(ctx);\n+\telse\n+\t\tctx->num_commit_graphs_after = 1;\n+\n \tcompute_generation_numbers(ctx);\n \n \tres = write_commit_graph_file(ctx);\n@@ -1368,6 +1616,21 @@ int write_commit_graph(const char *obj_dir,\n \tfree(ctx->graph_name);\n \tfree(ctx->commits.list);\n \tfree(ctx->oids.list);\n+\n+\tif (ctx->commit_graph_filenames_after) {\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++) {\n+\t\t\tfree(ctx->commit_graph_filenames_after[i]);\n+\t\t\tfree(ctx->commit_graph_hash_after[i]);\n+\t\t}\n+\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_before; i++)\n+\t\t\tfree(ctx->commit_graph_filenames_before[i]);\n+\n+\t\tfree(ctx->commit_graph_filenames_after);\n+\t\tfree(ctx->commit_graph_filenames_before);\n+\t\tfree(ctx->commit_graph_hash_after);\n+\t}\n+\n \tfree(ctx);\n \n \treturn res;\n@@ -1555,5 +1818,6 @@ void free_commit_graph(struct commit_graph *g)\n \t\tg->data = NULL;\n \t\tclose(g->graph_fd);\n \t}\n+\tfree(g->filename);\n \tfree(g);\n }\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 80f4917ddb..5c48c4f66a 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -47,6 +47,7 @@ struct commit_graph {\n \tunsigned char num_chunks;\n \tuint32_t num_commits;\n \tstruct object_id oid;\n+\tchar *filename;\n \n \tuint32_t num_commits_in_base;\n \tstruct commit_graph *base_graph;\n@@ -71,6 +72,7 @@ int generation_numbers_enabled(struct repository *r);\n \n #define COMMIT_GRAPH_APPEND (1 << 0)\n #define COMMIT_GRAPH_PROGRESS (1 << 1)\n+#define COMMIT_GRAPH_SPLIT (1 << 2)\n \n int write_commit_graph_reachable(const char *obj_dir, unsigned int flags);\n int write_commit_graph(const char *obj_dir,\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 3b6fd0d728..063f906b3e 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -20,7 +20,7 @@ test_expect_success 'verify graph with no graph file' '\n test_expect_success 'write graph with no packs' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write --object-dir . &&\n-\ttest_path_is_file info/commit-graph\n+\ttest_path_is_missing info/commit-graph\n '\n \n test_expect_success 'close with correct error on bad input' '\n-- \ngitgitgadget\n\n"},{"id":"376835","messageId":"a0330ebd2d9889b2fe1352cb232ac2652d32ac5d.1559932699.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 08/16] commit-graph: add --split option to builtin","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-07T18:38:26Z","receivedAt":"2019-06-07T18:38:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd a new \"--split\" option to the 'git commit-graph write' subcommand. This\noption allows the optional behavior of writing a commit-graph chain.\n\nThe current behavior will add a tip commit-graph containing any commits that\nare not in the existing commit-graph or commit-graph chain. Later changes\nwill allow merging the chain and expiring out-dated files.\n\nAdd a new test script (t5324-split-commit-graph.sh) that demonstrates this\nbehavior.\n\nHelped-by: Johannes Schindelin \nSigned-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 10 ++-\n commit-graph.c | 14 ++--\n t/t5324-split-commit-graph.sh | 122 ++++++++++++++++++++++++++++++++++\n 3 files changed, 138 insertions(+), 8 deletions(-)\n create mode 100755 t/t5324-split-commit-graph.sh\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 828b1a713f..c2c07d3917 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -10,7 +10,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph verify [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -25,7 +25,7 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -35,9 +35,9 @@ static struct opts_commit_graph {\n \tint stdin_packs;\n \tint stdin_commits;\n \tint append;\n+\tint split;\n } opts;\n \n-\n static int graph_verify(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n@@ -156,6 +156,8 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"start walk at commits listed by stdin\")),\n \t\tOPT_BOOL(0, \"append\", &opts.append,\n \t\t\tN_(\"include all commits already in the commit-graph file\")),\n+\t\tOPT_BOOL(0, \"split\", &opts.split,\n+\t\t\tN_(\"allow writing an incremental commit-graph file\")),\n \t\tOPT_END(),\n \t};\n \n@@ -169,6 +171,8 @@ static int graph_write(int argc, const char **argv)\n \t\topts.obj_dir = get_object_directory();\n \tif (opts.append)\n \t\tflags |= COMMIT_GRAPH_APPEND;\n+\tif (opts.split)\n+\t\tflags |= COMMIT_GRAPH_SPLIT;\n \n \tread_replace_refs = 0;\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex eb6d79567a..f94538c3df 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1472,12 +1472,16 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t}\n \n \t\tif (ctx->base_graph_name) {\n-\t\t\tresult = rename(ctx->base_graph_name,\n-\t\t\t\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2]);\n+\t\t\tconst char *dest = ctx->commit_graph_filenames_after[\n+\t\t\t\t\t\tctx->num_commit_graphs_after - 2];\n \n-\t\t\tif (result) {\n-\t\t\t\terror(_(\"failed to rename base commit-graph file\"));\n-\t\t\t\treturn -1;\n+\t\t\tif (strcmp(ctx->base_graph_name, dest)) {\n+\t\t\t\tresult = rename(ctx->base_graph_name, dest);\n+\n+\t\t\t\tif (result) {\n+\t\t\t\t\terror(_(\"failed to rename base commit-graph file\"));\n+\t\t\t\t\treturn -1;\n+\t\t\t\t}\n \t\t\t}\n \t\t} else {\n \t\t\tchar *graph_name = get_commit_graph_filename(ctx->obj_dir);\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nnew file mode 100755\nindex 0000000000..ccd24bd22b\n--- /dev/null\n+++ b/t/t5324-split-commit-graph.sh\n@@ -0,0 +1,122 @@\n+#!/bin/sh\n+\n+test_description='split commit graph'\n+. ./test-lib.sh\n+\n+GIT_TEST_COMMIT_GRAPH=0\n+\n+test_expect_success 'setup repo' '\n+\tgit init &&\n+\tgit config core.commitGraph true &&\n+\tinfodir=\".git/objects/info\" &&\n+\tgraphdir=\"$infodir/commit-graphs\" &&\n+\ttest_oid_init\n+'\n+\n+graph_read_expect() {\n+\tNUM_BASE=0\n+\tif test ! -z $2\n+\tthen\n+\t\tNUM_BASE=$2\n+\tfi\n+\tcat >expect <<- EOF\n+\theader: 43475048 1 1 3 $NUM_BASE\n+\tnum_commits: $1\n+\tchunks: oid_fanout oid_lookup commit_metadata\n+\tEOF\n+\tgit commit-graph read >output &&\n+\ttest_cmp expect output\n+}\n+\n+test_expect_success 'create commits and write commit-graph' '\n+\tfor i in $(test_seq 3)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i || return 1\n+\tdone &&\n+\tgit commit-graph write --reachable &&\n+\ttest_path_is_file $infodir/commit-graph &&\n+\tgraph_read_expect 3\n+'\n+\n+graph_git_two_modes() {\n+\tgit -c core.commitGraph=true $1 >output\n+\tgit -c core.commitGraph=false $1 >expect\n+\ttest_cmp expect output\n+}\n+\n+graph_git_behavior() {\n+\tMSG=$1\n+\tBRANCH=$2\n+\tCOMPARE=$3\n+\ttest_expect_success \"check normal git operations: $MSG\" '\n+\t\tgraph_git_two_modes \"log --oneline $BRANCH\" &&\n+\t\tgraph_git_two_modes \"log --topo-order $BRANCH\" &&\n+\t\tgraph_git_two_modes \"log --graph $COMPARE..$BRANCH\" &&\n+\t\tgraph_git_two_modes \"branch -vv\" &&\n+\t\tgraph_git_two_modes \"merge-base -a $BRANCH $COMPARE\"\n+\t'\n+}\n+\n+graph_git_behavior 'graph exists' commits/3 commits/1\n+\n+verify_chain_files_exist() {\n+\tfor hash in $(cat $1/commit-graph-chain)\n+\tdo\n+\t\ttest_path_is_file $1/graph-$hash.graph || return 1\n+\tdone\n+}\n+\n+test_expect_success 'add more commits, and write a new base graph' '\n+\tgit reset --hard commits/1 &&\n+\tfor i in $(test_seq 4 5)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i || return 1\n+\tdone &&\n+\tgit reset --hard commits/2 &&\n+\tfor i in $(test_seq 6 10)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i || return 1\n+\tdone &&\n+\tgit reset --hard commits/2 &&\n+\tgit merge commits/4 &&\n+\tgit branch merge/1 &&\n+\tgit reset --hard commits/4 &&\n+\tgit merge commits/6 &&\n+\tgit branch merge/2 &&\n+\tgit commit-graph write --reachable &&\n+\tgraph_read_expect 12\n+'\n+\n+test_expect_success 'add three more commits, write a tip graph' '\n+\tgit reset --hard commits/3 &&\n+\tgit merge merge/1 &&\n+\tgit merge commits/5 &&\n+\tgit merge merge/2 &&\n+\tgit branch merge/3 &&\n+\tgit commit-graph write --reachable --split &&\n+\ttest_path_is_missing $infodir/commit-graph &&\n+\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\tls $graphdir/graph-*.graph >graph-files &&\n+\ttest_line_count = 2 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n+graph_git_behavior 'split commit-graph: merge 3 vs 2' merge/3 merge/2\n+\n+test_expect_success 'add one commit, write a tip graph' '\n+\ttest_commit 11 &&\n+\tgit branch commits/11 &&\n+\tgit commit-graph write --reachable --split &&\n+\ttest_path_is_missing $infodir/commit-graph &&\n+\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\tls $graphdir/graph-*.graph >graph-files &&\n+\ttest_line_count = 3 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n+graph_git_behavior 'three-layer commit-graph: commit 11 vs 6' commits/11 commits/6\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"376834","messageId":"d14c79f9d508067137baa0b8a8a038b681f5c450.1559932699.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 05/16] commit-graph: add base graphs chunk","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-07T18:38:24Z","receivedAt":"2019-06-07T18:38:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTo quickly verify a commit-graph chain is valid on load, we will\nread from the new \"Base Graphs Chunk\" of each file in the chain.\nThis will prevent accidentally loading incorrect data from manually\nediting the commit-graph-chain file or renaming graph-{hash}.graph\nfiles.\n\nThe commit_graph struct already had an object_id struct \"oid\", but\nit was never initialized or used. Add a line to read the hash from\nthe end of the commit-graph file and into the oid member.\n\nSigned-off-by: Derrick Stolee \n---\n .../technical/commit-graph-format.txt | 11 ++++++++--\n commit-graph.c | 22 +++++++++++++++++++\n commit-graph.h | 1 +\n 3 files changed, 32 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex 16452a0504..a4f17441ae 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -44,8 +44,9 @@ HEADER:\n \n 1-byte number (C) of \"chunks\"\n \n- 1-byte (reserved for later use)\n- Current clients should ignore this value.\n+ 1-byte number (B) of base commit-graphs\n+ We infer the length (H*B) of the Base Graphs chunk\n+ from this value.\n \n CHUNK LOOKUP:\n \n@@ -92,6 +93,12 @@ CHUNK DATA:\n positions for the parents until reaching a value with the most-significant\n bit on. The other bits correspond to the position of the last parent.\n \n+ Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]\n+ This list of H-byte hashes describe a set of B commit-graph files that\n+ form a commit-graph chain. The graph position for the ith commit in this\n+ file's OID Lookup chunk is equal to i plus the number of commits in all\n+ base graphs. If B is non-zero, this chunk must exist.\n+\n TRAILER:\n \n \tH-byte HASH-checksum of all of the above.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 263c73282e..0ff1a4f379 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -22,6 +22,7 @@\n #define GRAPH_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define GRAPH_CHUNKID_DATA 0x43444154 /* \"CDAT\" */\n #define GRAPH_CHUNKID_EXTRAEDGES 0x45444745 /* \"EDGE\" */\n+#define GRAPH_CHUNKID_BASE 0x42415345 /* \"BASE\" */\n \n #define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n \n@@ -262,6 +263,12 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t\t\telse\n \t\t\t\tgraph->chunk_extra_edges = data + chunk_offset;\n \t\t\tbreak;\n+\n+\t\tcase GRAPH_CHUNKID_BASE:\n+\t\t\tif (graph->chunk_base_graphs)\n+\t\t\t\tchunk_repeated = 1;\n+\t\t\telse\n+\t\t\t\tgraph->chunk_base_graphs = data + chunk_offset;\n \t\t}\n \n \t\tif (chunk_repeated) {\n@@ -280,6 +287,8 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t\tlast_chunk_offset = chunk_offset;\n \t}\n \n+\thashcpy(graph->oid.hash, graph->data + graph->data_len - graph->hash_len);\n+\n \tif (verify_commit_graph_lite(graph))\n \t\treturn NULL;\n \n@@ -315,8 +324,21 @@ static int add_graph_to_chain(struct commit_graph *g,\n {\n \tstruct commit_graph *cur_g = chain;\n \n+\tif (n && !g->chunk_base_graphs) {\n+\t\twarning(_(\"commit-graph has no base graphs chunk\"));\n+\t\treturn 0;\n+\t}\n+\n \twhile (n) {\n \t\tn--;\n+\n+\t\tif (!cur_g ||\n+\t\t !oideq(&oids[n], &cur_g->oid) ||\n+\t\t !hasheq(oids[n].hash, g->chunk_base_graphs + g->hash_len * n)) {\n+\t\t\twarning(_(\"commit-graph chain does not match\"));\n+\t\t\treturn 0;\n+\t\t}\n+\n \t\tcur_g = cur_g->base_graph;\n \t}\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex f9fe32ebe3..80f4917ddb 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -55,6 +55,7 @@ struct commit_graph {\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_commit_data;\n \tconst unsigned char *chunk_extra_edges;\n+\tconst unsigned char *chunk_base_graphs;\n };\n \n struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st);\n-- \ngitgitgadget\n\n"},{"id":"376836","messageId":"fc3423046b9b2fc43bf8323dd41281f0a29b9d0a.1559932699.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 04/16] commit-graph: load commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-07T18:38:23Z","receivedAt":"2019-06-07T18:38:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nPrepare the logic for reading a chain of commit-graphs.\n\nFirst, look for a file at $OBJDIR/info/commit-graph. If it exists,\nthen use that file and stop.\n\nNext, look for the chain file at $OBJDIR/info/commit-graphs/commit-graph-chain.\nIf this file exists, then load the hash values as line-separated values in that\nfile and load $OBJDIR/info/commit-graphs/graph-{hash[i]}.graph for each hash[i]\nin that file. The file is given in order, so the first hash corresponds to the\n\"base\" file and the final hash corresponds to the \"tip\" file.\n\nThis implementation assumes that all of the graph-{hash}.graph files are in\nthe same object directory as the commit-graph-chain file. This will be updated\nin a future change. This change is purposefully simple so we can isolate the\ndifferent concerns.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 112 ++++++++++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 106 insertions(+), 6 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 70d5889892..263c73282e 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -45,6 +45,19 @@ char *get_commit_graph_filename(const char *obj_dir)\n \treturn xstrfmt(\"%s/info/commit-graph\", obj_dir);\n }\n \n+static char *get_split_graph_filename(const char *obj_dir,\n+\t\t\t\t const char *oid_hex)\n+{\n+\treturn xstrfmt(\"%s/info/commit-graphs/graph-%s.graph\",\n+\t\t obj_dir,\n+\t\t oid_hex);\n+}\n+\n+static char *get_chain_filename(const char *obj_dir)\n+{\n+\treturn xstrfmt(\"%s/info/commit-graphs/commit-graph-chain\", obj_dir);\n+}\n+\n static uint8_t oid_version(void)\n {\n \treturn 1;\n@@ -286,18 +299,105 @@ static struct commit_graph *load_commit_graph_one(const char *graph_file)\n \treturn load_commit_graph_one_fd_st(fd, &st);\n }\n \n+static struct commit_graph *load_commit_graph_v1(struct repository *r, const char *obj_dir)\n+{\n+\tchar *graph_name = get_commit_graph_filename(obj_dir);\n+\tstruct commit_graph *g = load_commit_graph_one(graph_name);\n+\tfree(graph_name);\n+\n+\treturn g;\n+}\n+\n+static int add_graph_to_chain(struct commit_graph *g,\n+\t\t\t struct commit_graph *chain,\n+\t\t\t struct object_id *oids,\n+\t\t\t int n)\n+{\n+\tstruct commit_graph *cur_g = chain;\n+\n+\twhile (n) {\n+\t\tn--;\n+\t\tcur_g = cur_g->base_graph;\n+\t}\n+\n+\tg->base_graph = chain;\n+\n+\tif (chain)\n+\t\tg->num_commits_in_base = chain->num_commits + chain->num_commits_in_base;\n+\n+\treturn 1;\n+}\n+\n+static struct commit_graph *load_commit_graph_chain(struct repository *r, const char *obj_dir)\n+{\n+\tstruct commit_graph *graph_chain = NULL;\n+\tstruct strbuf line = STRBUF_INIT;\n+\tstruct stat st;\n+\tstruct object_id *oids;\n+\tint i = 0, valid = 1, count;\n+\tchar *chain_name = get_chain_filename(obj_dir);\n+\tFILE *fp;\n+\tint stat_res;\n+\n+\tfp = fopen(chain_name, \"r\");\n+\tstat_res = stat(chain_name, &st);\n+\tfree(chain_name);\n+\n+\tif (!fp ||\n+\t stat_res ||\n+\t st.st_size <= the_hash_algo->hexsz)\n+\t\treturn NULL;\n+\n+\tcount = st.st_size / (the_hash_algo->hexsz + 1);\n+\toids = xcalloc(count, sizeof(struct object_id));\n+\n+\tfor (i = 0; i < count && valid; i++) {\n+\t\tchar *graph_name;\n+\t\tstruct commit_graph *g;\n+\n+\t\tif (strbuf_getline_lf(&line, fp) == EOF)\n+\t\t\tbreak;\n+\n+\t\tif (get_oid_hex(line.buf, &oids[i])) {\n+\t\t\twarning(_(\"invalid commit-graph chain: line '%s' not a hash\"),\n+\t\t\t\tline.buf);\n+\t\t\tvalid = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tgraph_name = get_split_graph_filename(obj_dir, line.buf);\n+\t\tg = load_commit_graph_one(graph_name);\n+\t\tfree(graph_name);\n+\n+\t\tif (g && add_graph_to_chain(g, graph_chain, oids, i))\n+\t\t\tgraph_chain = g;\n+\t\telse\n+\t\t\tvalid = 0;\n+\t}\n+\n+\tfree(oids);\n+\tfclose(fp);\n+\n+\treturn graph_chain;\n+}\n+\n+static struct commit_graph *read_commit_graph_one(struct repository *r, const char *obj_dir)\n+{\n+\tstruct commit_graph *g = load_commit_graph_v1(r, obj_dir);\n+\n+\tif (!g)\n+\t\tg = load_commit_graph_chain(r, obj_dir);\n+\n+\treturn g;\n+}\n+\n static void prepare_commit_graph_one(struct repository *r, const char *obj_dir)\n {\n-\tchar *graph_name;\n \n \tif (r->objects->commit_graph)\n \t\treturn;\n \n-\tgraph_name = get_commit_graph_filename(obj_dir);\n-\tr->objects->commit_graph =\n-\t\tload_commit_graph_one(graph_name);\n-\n-\tFREE_AND_NULL(graph_name);\n+\tr->objects->commit_graph = read_commit_graph_one(r, obj_dir);\n }\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"376837","messageId":"2470d2b5485439f49dc9ec5418cc3d10bcad0e0a.1559932699.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 03/16] commit-graph: rename commit_compare to oid_compare","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-07T18:38:23Z","receivedAt":"2019-06-07T18:38:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe helper function commit_compare() actually compares object_id\nstructs, not commits. A future change to commit-graph.c will need\nto sort commit structs, so rename this function in advance.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8c3598037b..70d5889892 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -770,7 +770,7 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n \t}\n }\n \n-static int commit_compare(const void *_a, const void *_b)\n+static int oid_compare(const void *_a, const void *_b)\n {\n \tconst struct object_id *a = (const struct object_id *)_a;\n \tconst struct object_id *b = (const struct object_id *)_b;\n@@ -1039,7 +1039,7 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n \t\t\t_(\"Counting distinct commits in commit graph\"),\n \t\t\tctx->oids.nr);\n \tdisplay_progress(ctx->progress, 0); /* TODO: Measure QSORT() progress */\n-\tQSORT(ctx->oids.list, ctx->oids.nr, commit_compare);\n+\tQSORT(ctx->oids.list, ctx->oids.nr, oid_compare);\n \n \tfor (i = 1; i < ctx->oids.nr; i++) {\n \t\tdisplay_progress(ctx->progress, i + 1);\n-- \ngitgitgadget\n\n"},{"id":"376838","messageId":"2093bab5b1e99e97e87a9efe130d7dc2938ce127.1559932699.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 10/16] commit-graph: allow cross-alternate chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-07T18:38:28Z","receivedAt":"2019-06-07T18:38:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an environment like a fork network, it is helpful to have a\ncommit-graph chain that spans both the base repo and the fork repo. The\nfork is usually a small set of data on top of the large repo, but\nsometimes the fork is much larger. For example, git-for-windows/git has\nalmost double the number of commits as git/git because it rebases its\ncommits on every major version update.\n\nTo allow cross-alternate commit-graph chains, we need a few pieces:\n\n1. When looking for a graph-{hash}.graph file, check all alternates.\n\n2. When merging commit-graph chains, do not merge across alternates.\n\n3. When writing a new commit-graph chain based on a commit-graph file\n in another object directory, do not allow success if the base file\n has of the name \"commit-graph\" instead of\n \"commit-graphs/graph-{hash}.graph\".\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 40 ++++++++++++++++++\n commit-graph.c | 54 +++++++++++++++++++-----\n commit-graph.h | 1 +\n t/t5324-split-commit-graph.sh | 37 ++++++++++++++++\n 4 files changed, 121 insertions(+), 11 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex d9c6253b0a..473032e476 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -266,6 +266,42 @@ The merge strategy values (2 for the size multiple, 64,000 for the maximum\n number of commits) could be extracted into config settings for full\n flexibility.\n \n+## Chains across multiple object directories\n+\n+In a repo with alternates, we look for the `commit-graph-chain` file starting\n+in the local object directory and then in each alternate. The first file that\n+exists defines our chain. As we look for the `graph-{hash}` files for\n+each `{hash}` in the chain file, we follow the same pattern for the host\n+directories.\n+\n+This allows commit-graphs to be split across multiple forks in a fork network.\n+The typical case is a large \"base\" repo with many smaller forks.\n+\n+As the base repo advances, it will likely update and merge its commit-graph\n+chain more frequently than the forks. If a fork updates their commit-graph after\n+the base repo, then it should \"reparent\" the commit-graph chain onto the new\n+chain in the base repo. When reading each `graph-{hash}` file, we track\n+the object directory containing it. During a write of a new commit-graph file,\n+we check for any changes in the source object directory and read the\n+`commit-graph-chain` file for that source and create a new file based on those\n+files. During this \"reparent\" operation, we necessarily need to collapse all\n+levels in the fork, as all of the files are invalid against the new base file.\n+\n+It is crucial to be careful when cleaning up \"unreferenced\" `graph-{hash}.graph`\n+files in this scenario. It falls to the user to define the proper settings for\n+their custom environment:\n+\n+ 1. When merging levels in the base repo, the unreferenced files may still be\n+ referenced by chains from fork repos.\n+\n+ 2. The expiry time should be set to a length of time such that every fork has\n+ time to recompute their commit-graph chain to \"reparent\" onto the new base\n+ file(s).\n+\n+ 3. If the commit-graph chain is updated in the base, the fork will not have\n+ access to the new chain until its chain is updated to reference those files.\n+ (This may change in the future [5].)\n+\n Related Links\n -------------\n [0] https://bugs.chromium.org/p/git/issues/detail?id=8\n@@ -292,3 +328,7 @@ Related Links\n \n [4] https://public-inbox.org/git/20180108154822.54829-1-git@jeffhostetler.com/T/#u\n A patch to remove the ahead-behind calculation from 'status'.\n+\n+[5] https://public-inbox.org/git/f27db281-abad-5043-6d71-cbb083b1c877@gmail.com/\n+ A discussion of a \"two-dimensional graph position\" that can allow reading\n+ multiple commit-graph chains at the same time.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 288b5e0280..22b167d9eb 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -320,6 +320,9 @@ static struct commit_graph *load_commit_graph_v1(struct repository *r, const cha\n \tstruct commit_graph *g = load_commit_graph_one(graph_name);\n \tfree(graph_name);\n \n+\tif (g)\n+\t\tg->obj_dir = obj_dir;\n+\n \treturn g;\n }\n \n@@ -379,9 +382,8 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r, const\n \tcount = st.st_size / (the_hash_algo->hexsz + 1);\n \toids = xcalloc(count, sizeof(struct object_id));\n \n-\tfor (i = 0; i < count && valid; i++) {\n-\t\tchar *graph_name;\n-\t\tstruct commit_graph *g;\n+\tfor (i = 0; i < count; i++) {\n+\t\tstruct object_directory *odb;\n \n \t\tif (strbuf_getline_lf(&line, fp) == EOF)\n \t\t\tbreak;\n@@ -393,14 +395,29 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r, const\n \t\t\tbreak;\n \t\t}\n \n-\t\tgraph_name = get_split_graph_filename(obj_dir, line.buf);\n-\t\tg = load_commit_graph_one(graph_name);\n-\t\tfree(graph_name);\n+\t\tvalid = 0;\n+\t\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n+\t\t\tchar *graph_name = get_split_graph_filename(odb->path, line.buf);\n+\t\t\tstruct commit_graph *g = load_commit_graph_one(graph_name);\n \n-\t\tif (g && add_graph_to_chain(g, graph_chain, oids, i))\n-\t\t\tgraph_chain = g;\n-\t\telse\n-\t\t\tvalid = 0;\n+\t\t\tfree(graph_name);\n+\n+\t\t\tif (g) {\n+\t\t\t\tg->obj_dir = odb->path;\n+\n+\t\t\t\tif (add_graph_to_chain(g, graph_chain, oids, i)) {\n+\t\t\t\t\tgraph_chain = g;\n+\t\t\t\t\tvalid = 1;\n+\t\t\t\t}\n+\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (!valid) {\n+\t\t\twarning(_(\"unable to find all commit-graph files\"));\n+\t\t\tbreak;\n+\t\t}\n \t}\n \n \tfree(oids);\n@@ -1418,7 +1435,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \n \tif (ctx->split && ctx->base_graph_name && ctx->num_commit_graphs_after > 1) {\n \t\tchar *new_base_hash = xstrdup(oid_to_hex(&ctx->new_base_graph->oid));\n-\t\tchar *new_base_name = get_split_graph_filename(ctx->obj_dir, new_base_hash);\n+\t\tchar *new_base_name = get_split_graph_filename(ctx->new_base_graph->obj_dir, new_base_hash);\n \n \t\tfree(ctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2]);\n \t\tfree(ctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 2]);\n@@ -1493,6 +1510,9 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \n \twhile (g && (g->num_commits <= split_strategy_size_mult * num_commits ||\n \t\t num_commits > split_strategy_max_commits)) {\n+\t\tif (strcmp(g->obj_dir, ctx->obj_dir))\n+\t\t\tbreak;\n+\n \t\tnum_commits += g->num_commits;\n \t\tg = g->base_graph;\n \n@@ -1501,6 +1521,18 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \n \tctx->new_base_graph = g;\n \n+\tif (ctx->num_commit_graphs_after == 2) {\n+\t\tchar *old_graph_name = get_commit_graph_filename(g->obj_dir);\n+\n+\t\tif (!strcmp(g->filename, old_graph_name) &&\n+\t\t strcmp(g->obj_dir, ctx->obj_dir)) {\n+\t\t\tctx->num_commit_graphs_after = 1;\n+\t\t\tctx->new_base_graph = NULL;\n+\t\t}\n+\n+\t\tfree(old_graph_name);\n+\t}\n+\n \tALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);\n \tALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex 5c48c4f66a..10466bc064 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -48,6 +48,7 @@ struct commit_graph {\n \tuint32_t num_commits;\n \tstruct object_id oid;\n \tchar *filename;\n+\tconst char *obj_dir;\n \n \tuint32_t num_commits_in_base;\n \tstruct commit_graph *base_graph;\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 5cb5663a30..46f0832f68 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -90,6 +90,21 @@ test_expect_success 'add more commits, and write a new base graph' '\n \tgraph_read_expect 12\n '\n \n+test_expect_success 'fork and fail to base a chain on a commit-graph file' '\n+\ttest_when_finished rm -rf fork &&\n+\tgit clone . fork &&\n+\t(\n+\t\tcd fork &&\n+\t\trm .git/objects/info/commit-graph &&\n+\t\techo \"$(pwd)/../.git/objects\" >.git/objects/info/alternates &&\n+\t\ttest_commit new-commit &&\n+\t\tgit commit-graph write --reachable --split &&\n+\t\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n+\t\tverify_chain_files_exist $graphdir\n+\t)\n+'\n+\n test_expect_success 'add three more commits, write a tip graph' '\n \tgit reset --hard commits/3 &&\n \tgit merge merge/1 &&\n@@ -132,4 +147,26 @@ test_expect_success 'add one commit, write a merged graph' '\n \n graph_git_behavior 'merged commit-graph: commit 12 vs 6' commits/12 commits/6\n \n+test_expect_success 'create fork and chain across alternate' '\n+\tgit clone . fork &&\n+\t(\n+\t\tcd fork &&\n+\t\tgit config core.commitGraph true &&\n+\t\trm -rf $graphdir &&\n+\t\techo \"$(pwd)/../.git/objects\" >.git/objects/info/alternates &&\n+\t\ttest_commit 13 &&\n+\t\tgit branch commits/13 &&\n+\t\tgit commit-graph write --reachable --split &&\n+\t\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\t\ttest_line_count = 3 $graphdir/commit-graph-chain &&\n+\t\tls $graphdir/graph-*.graph >graph-files &&\n+\t\ttest_line_count = 1 graph-files &&\n+\t\tgit -c core.commitGraph=true rev-list HEAD >expect &&\n+\t\tgit -c core.commitGraph=false rev-list HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+graph_git_behavior 'alternate: commit 13 vs 6' commits/13 commits/6\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"376839","messageId":"28eccfa52b7fcce41de98cfa8ccd6d02da5c75f9.1559932699.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 09/16] commit-graph: merge commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-07T18:38:27Z","receivedAt":"2019-06-07T18:38:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen searching for a commit in a commit-graph chain of G graphs with N\ncommits, the search takes O(G log N) time. If we always add a new tip\ngraph with every write, the linear G term will start to dominate and\nslow the lookup process.\n\nTo keep lookups fast, but also keep most incremental writes fast, create\na strategy for merging levels of the commit-graph chain. The strategy is\ndetailed in the commit-graph design document, but is summarized by these\ntwo conditions:\n\n 1. If the number of commits we are adding is more than half the number\n of commits in the graph below, then merge with that graph.\n\n 2. If we are writing more than 64,000 commits into a single graph,\n then merge with all lower graphs.\n\nThe numeric values in the conditions above are currently constant, but\ncan become config options in a future update.\n\nAs we merge levels of the commit-graph chain, check that the commits\nstill exist in the repository. A garbage-collection operation may have\nremoved those commits from the object store and we do not want to\npersist them in the commit-graph chain. This is a non-issue if the\n'git gc' process wrote a new, single-level commit-graph file.\n\nAfter we merge levels, the old graph-{hash}.graph files are no longer\nreferenced by the commit-graph-chain file. We will expire these files in\na future change.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 80 ++++++++++\n commit-graph.c | 180 ++++++++++++++++++-----\n t/t5324-split-commit-graph.sh | 13 ++\n 3 files changed, 240 insertions(+), 33 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex 1dca3bd8fe..d9c6253b0a 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -186,6 +186,86 @@ positions to refer to their parents, which may be in `graph-{hash1}.graph` or\n its containment in the intervals [0, X0), [X0, X0 + X1), [X0 + X1, X0 + X1 +\n X2).\n \n+Each commit-graph file (except the base, `graph-{hash0}.graph`) contains data\n+specifying the hashes of all files in the lower layers. In the above example,\n+`graph-{hash1}.graph` contains `{hash0}` while `graph-{hash2}.graph` contains\n+`{hash0}` and `{hash1}`.\n+\n+## Merging commit-graph files\n+\n+If we only added a new commit-graph file on every write, we would run into a\n+linear search problem through many commit-graph files. Instead, we use a merge\n+strategy to decide when the stack should collapse some number of levels.\n+\n+The diagram below shows such a collapse. As a set of new commits are added, it\n+is determined by the merge strategy that the files should collapse to\n+`graph-{hash1}`. Thus, the new commits, the commits in `graph-{hash2}` and\n+the commits in `graph-{hash1}` should be combined into a new `graph-{hash3}`\n+file.\n+\n+\t\t\t +---------------------+\n+\t\t\t | |\n+\t\t\t | (new commits) |\n+\t\t\t | |\n+\t\t\t +---------------------+\n+\t\t\t | |\n+ +-----------------------+ +---------------------+\n+ | graph-{hash2} |->| |\n+ +-----------------------+ +---------------------+\n+\t | | |\n+ +-----------------------+ +---------------------+\n+ | | | |\n+ | graph-{hash1} |->| |\n+ | | | |\n+ +-----------------------+ +---------------------+\n+\t | tmp_graphXXX\n+ +-----------------------+\n+ | |\n+ | |\n+ | |\n+ | graph-{hash0} |\n+ | |\n+ | |\n+ | |\n+ +-----------------------+\n+\n+During this process, the commits to write are combined, sorted and we write the\n+contents to a temporary file, all while holding a `commit-graph-chain.lock`\n+lock-file. When the file is flushed, we rename it to `graph-{hash3}`\n+according to the computed `{hash3}`. Finally, we write the new chain data to\n+`commit-graph-chain.lock`:\n+\n+```\n+\t{hash3}\n+\t{hash0}\n+```\n+\n+We then close the lock-file.\n+\n+## Merge Strategy\n+\n+When writing a set of commits that do not exist in the commit-graph stack of\n+height N, we default to creating a new file at level N + 1. We then decide to\n+merge with the Nth level if one of two conditions hold:\n+\n+ 1. The expected file size for level N + 1 is at least half the file size for\n+ level N.\n+\n+ 2. Level N + 1 contains more than 64,0000 commits.\n+\n+This decision cascades down the levels: when we merge a level we create a new\n+set of commits that then compares to the next level.\n+\n+The first condition bounds the number of levels to be logarithmic in the total\n+number of commits. The second condition bounds the total number of commits in\n+a `graph-{hashN}` file and not in the `commit-graph` file, preventing\n+significant performance issues when the stack merges and another process only\n+partially reads the previous stack.\n+\n+The merge strategy values (2 for the size multiple, 64,000 for the maximum\n+number of commits) could be extracted into config settings for full\n+flexibility.\n+\n Related Links\n -------------\n [0] https://bugs.chromium.org/p/git/issues/detail?id=8\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f94538c3df..288b5e0280 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1298,36 +1298,6 @@ static int write_graph_chunk_base(struct hashfile *f,\n \treturn 0;\n }\n \n-static void init_commit_graph_chain(struct write_commit_graph_context *ctx)\n-{\n-\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n-\tuint32_t i;\n-\n-\tctx->new_base_graph = g;\n-\tctx->base_graph_name = xstrdup(g->filename);\n-\tctx->new_num_commits_in_base = g->num_commits + g->num_commits_in_base;\n-\n-\tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n-\n-\tALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);\n-\tALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);\n-\n-\tfor (i = 0; i < ctx->num_commit_graphs_before - 1; i++)\n-\t\tctx->commit_graph_filenames_after[i] = xstrdup(ctx->commit_graph_filenames_before[i]);\n-\n-\tif (ctx->num_commit_graphs_before)\n-\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_before - 1] =\n-\t\t\tget_split_graph_filename(ctx->obj_dir, oid_to_hex(&g->oid));\n-\n-\ti = ctx->num_commit_graphs_before - 1;\n-\n-\twhile (g) {\n-\t\tctx->commit_graph_hash_after[i] = xstrdup(oid_to_hex(&g->oid));\n-\t\ti--;\n-\t\tg = g->base_graph;\n-\t}\n-}\n-\n static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n@@ -1509,6 +1479,145 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \treturn 0;\n }\n \n+static int split_strategy_max_commits = 64000;\n+static float split_strategy_size_mult = 2.0f;\n+\n+static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n+{\n+\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\tuint32_t num_commits = ctx->commits.nr;\n+\tuint32_t i;\n+\n+\tg = ctx->r->objects->commit_graph;\n+\tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n+\n+\twhile (g && (g->num_commits <= split_strategy_size_mult * num_commits ||\n+\t\t num_commits > split_strategy_max_commits)) {\n+\t\tnum_commits += g->num_commits;\n+\t\tg = g->base_graph;\n+\n+\t\tctx->num_commit_graphs_after--;\n+\t}\n+\n+\tctx->new_base_graph = g;\n+\n+\tALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);\n+\tALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);\n+\n+\tfor (i = 0; i < ctx->num_commit_graphs_after &&\n+\t\t i < ctx->num_commit_graphs_before; i++)\n+\t\tctx->commit_graph_filenames_after[i] = xstrdup(ctx->commit_graph_filenames_before[i]);\n+\n+\ti = ctx->num_commit_graphs_before - 1;\n+\tg = ctx->r->objects->commit_graph;\n+\n+\twhile (g) {\n+\t\tif (i < ctx->num_commit_graphs_after)\n+\t\t\tctx->commit_graph_hash_after[i] = xstrdup(oid_to_hex(&g->oid));\n+\n+\t\ti--;\n+\t\tg = g->base_graph;\n+\t}\n+}\n+\n+static void merge_commit_graph(struct write_commit_graph_context *ctx,\n+\t\t\t struct commit_graph *g)\n+{\n+\tuint32_t i;\n+\tuint32_t offset = g->num_commits_in_base;\n+\n+\tALLOC_GROW(ctx->commits.list, ctx->commits.nr + g->num_commits, ctx->commits.alloc);\n+\n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct object_id oid;\n+\t\tstruct commit *result;\n+\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\n+\t\tload_oid_from_graph(g, i + offset, &oid);\n+\n+\t\t/* only add commits if they still exist in the repo */\n+\t\tresult = lookup_commit_reference_gently(ctx->r, &oid, 1);\n+\n+\t\tif (result) {\n+\t\t\tctx->commits.list[ctx->commits.nr] = result;\n+\t\t\tctx->commits.nr++;\n+\t\t}\n+\t}\n+}\n+\n+static int commit_compare(const void *_a, const void *_b)\n+{\n+\tconst struct commit *a = *(const struct commit **)_a;\n+\tconst struct commit *b = *(const struct commit **)_b;\n+\treturn oidcmp(&a->object.oid, &b->object.oid);\n+}\n+\n+static void sort_and_scan_merged_commits(struct write_commit_graph_context *ctx)\n+{\n+\tuint32_t i, num_parents;\n+\tstruct commit_list *parent;\n+\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"Scanning merged commits\"),\n+\t\t\t\t\tctx->commits.nr);\n+\n+\tQSORT(ctx->commits.list, ctx->commits.nr, commit_compare);\n+\n+\tctx->num_extra_edges = 0;\n+\tfor (i = 0; i < ctx->commits.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i);\n+\n+\t\tif (i && oideq(&ctx->commits.list[i - 1]->object.oid,\n+\t\t\t &ctx->commits.list[i]->object.oid)) {\n+\t\t\tdie(_(\"unexpected duplicate commit id %s\"),\n+\t\t\t oid_to_hex(&ctx->commits.list[i]->object.oid));\n+\t\t} else {\n+\t\t\tnum_parents = 0;\n+\t\t\tfor (parent = ctx->commits.list[i]->parents; parent; parent = parent->next)\n+\t\t\t\tnum_parents++;\n+\n+\t\t\tif (num_parents > 2)\n+\t\t\t\tctx->num_extra_edges += num_parents - 2;\n+\t\t}\n+\t}\n+\n+\tstop_progress(&ctx->progress);\n+}\n+\n+static void merge_commit_graphs(struct write_commit_graph_context *ctx)\n+{\n+\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\tuint32_t current_graph_number = ctx->num_commit_graphs_before;\n+\tstruct strbuf progress_title = STRBUF_INIT;\n+\n+\twhile (g && current_graph_number >= ctx->num_commit_graphs_after) {\n+\t\tcurrent_graph_number--;\n+\n+\t\tif (ctx->report_progress) {\n+\t\t\tstrbuf_addstr(&progress_title, _(\"Merging commit-graph\"));\n+\t\t\tctx->progress = start_delayed_progress(progress_title.buf, 0);\n+\t\t}\n+\n+\t\tmerge_commit_graph(ctx, g);\n+\t\tstop_progress(&ctx->progress);\n+\t\tstrbuf_release(&progress_title);\n+\n+\t\tg = g->base_graph;\n+\t}\n+\n+\tif (g) {\n+\t\tctx->new_base_graph = g;\n+\t\tctx->new_num_commits_in_base = g->num_commits + g->num_commits_in_base;\n+\t}\n+\n+\tif (ctx->new_base_graph)\n+\t\tctx->base_graph_name = xstrdup(ctx->new_base_graph->filename);\n+\n+\tsort_and_scan_merged_commits(ctx);\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -1554,6 +1663,9 @@ int write_commit_graph(const char *obj_dir,\n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n \n+\tif (ctx->split && ctx->oids.alloc > split_strategy_max_commits)\n+\t\tctx->oids.alloc = split_strategy_max_commits;\n+\n \tif (ctx->append) {\n \t\tprepare_commit_graph_one(ctx->r, ctx->obj_dir);\n \t\tif (ctx->r->objects->commit_graph)\n@@ -1607,9 +1719,11 @@ int write_commit_graph(const char *obj_dir,\n \tif (!ctx->commits.nr)\n \t\tgoto cleanup;\n \n-\tif (ctx->split)\n-\t\tinit_commit_graph_chain(ctx);\n-\telse\n+\tif (ctx->split) {\n+\t\tsplit_graph_merge_strategy(ctx);\n+\n+\t\tmerge_commit_graphs(ctx);\n+\t} else\n \t\tctx->num_commit_graphs_after = 1;\n \n \tcompute_generation_numbers(ctx);\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex ccd24bd22b..5cb5663a30 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -119,4 +119,17 @@ test_expect_success 'add one commit, write a tip graph' '\n \n graph_git_behavior 'three-layer commit-graph: commit 11 vs 6' commits/11 commits/6\n \n+test_expect_success 'add one commit, write a merged graph' '\n+\ttest_commit 12 &&\n+\tgit branch commits/12 &&\n+\tgit commit-graph write --reachable --split &&\n+\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\tls $graphdir/graph-*.graph >graph-files &&\n+\ttest_line_count = 4 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n+graph_git_behavior 'merged commit-graph: commit 12 vs 6' commits/12 commits/6\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"376840","messageId":"9fec4f9a36fafdb2b3d7148ec00ccbbce74b6fff.1559932699.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 13/16] commit-graph: verify chains with --shallow mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-07T18:38:30Z","receivedAt":"2019-06-07T18:38:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf we wrote a commit-graph chain, we only modified the tip file in\nthe chain. It is valuable to verify what we wrote, but not waste\ntime checking files we did not write.\n\nAdd a '--shallow' option to the 'git commit-graph verify' subcommand\nand check that it does not read the base graph in a two-file chain.\n\nMaking the verify subcommand read from a chain of commit-graphs takes\nsome rearranging of the builtin code.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 5 ++-\n builtin/commit-graph.c | 27 +++++++++----\n commit-graph.c | 15 ++++++--\n commit-graph.h | 6 ++-\n t/t5324-split-commit-graph.sh | 62 ++++++++++++++++++++++++++++++\n 5 files changed, 101 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 365e145e82..eb5e7865f0 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -10,7 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git commit-graph read' [--object-dir ]\n-'git commit-graph verify' [--object-dir ]\n+'git commit-graph verify' [--object-dir ] [--shallow]\n 'git commit-graph write' [--object-dir ]\n \n \n@@ -80,6 +80,9 @@ Used for debugging purposes.\n \n Read the commit-graph file and verify its contents against the object\n database. Used to check for corrupted data.\n++\n+With the `--shallow` option, only check the tip commit-graph file in\n+a chain of split commit-graphs.\n \n \n EXAMPLES\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex cc4af67820..2e8e4abf57 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -5,17 +5,18 @@\n #include \"parse-options.h\"\n #include \"repository.h\"\n #include \"commit-graph.h\"\n+#include \"object-store.h\"\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n-\tN_(\"git commit-graph verify [--object-dir ]\"),\n+\tN_(\"git commit-graph verify [--object-dir ] [--shallow]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] \"),\n \tNULL\n };\n \n static const char * const builtin_commit_graph_verify_usage[] = {\n-\tN_(\"git commit-graph verify [--object-dir ]\"),\n+\tN_(\"git commit-graph verify [--object-dir ] [--shallow]\"),\n \tNULL\n };\n \n@@ -36,6 +37,7 @@ static struct opts_commit_graph {\n \tint stdin_commits;\n \tint append;\n \tint split;\n+\tint shallow;\n } opts;\n \n static int graph_verify(int argc, const char **argv)\n@@ -45,11 +47,14 @@ static int graph_verify(int argc, const char **argv)\n \tint open_ok;\n \tint fd;\n \tstruct stat st;\n+\tint flags = 0;\n \n \tstatic struct option builtin_commit_graph_verify_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n \t\t\t N_(\"dir\"),\n \t\t\t N_(\"The object directory to store the graph\")),\n+\t\tOPT_BOOL(0, \"shallow\", &opts.shallow,\n+\t\t\t N_(\"if the commit-graph is split, only verify the tip file\")),\n \t\tOPT_END(),\n \t};\n \n@@ -59,21 +64,27 @@ static int graph_verify(int argc, const char **argv)\n \n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n+\tif (opts.shallow)\n+\t\tflags |= COMMIT_GRAPH_VERIFY_SHALLOW;\n \n \tgraph_name = get_commit_graph_filename(opts.obj_dir);\n \topen_ok = open_commit_graph(graph_name, &fd, &st);\n-\tif (!open_ok && errno == ENOENT)\n-\t\treturn 0;\n-\tif (!open_ok)\n+\tif (!open_ok && errno != ENOENT)\n \t\tdie_errno(_(\"Could not open commit-graph '%s'\"), graph_name);\n-\tgraph = load_commit_graph_one_fd_st(fd, &st);\n+\n \tFREE_AND_NULL(graph_name);\n \n+\tif (open_ok)\n+\t\tgraph = load_commit_graph_one_fd_st(fd, &st);\n+\t else\n+\t\tgraph = read_commit_graph_one(the_repository, opts.obj_dir);\n+\n+\t/* Return failure if open_ok predicted success */\n \tif (!graph)\n-\t\treturn 1;\n+\t\treturn !!open_ok;\n \n \tUNLEAK(graph);\n-\treturn verify_commit_graph(the_repository, graph);\n+\treturn verify_commit_graph(the_repository, graph, flags);\n }\n \n static int graph_read(int argc, const char **argv)\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 888eaa7d03..431ad2cf1d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -426,7 +426,7 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r, const\n \treturn graph_chain;\n }\n \n-static struct commit_graph *read_commit_graph_one(struct repository *r, const char *obj_dir)\n+struct commit_graph *read_commit_graph_one(struct repository *r, const char *obj_dir)\n {\n \tstruct commit_graph *g = load_commit_graph_v1(r, obj_dir);\n \n@@ -1885,7 +1885,7 @@ static void graph_report(const char *fmt, ...)\n #define GENERATION_ZERO_EXISTS 1\n #define GENERATION_NUMBER_EXISTS 2\n \n-int verify_commit_graph(struct repository *r, struct commit_graph *g)\n+int verify_commit_graph(struct repository *r, struct commit_graph *g, int flags)\n {\n \tuint32_t i, cur_fanout_pos = 0;\n \tstruct object_id prev_oid, cur_oid, checksum;\n@@ -1893,6 +1893,7 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \tstruct hashfile *f;\n \tint devnull;\n \tstruct progress *progress = NULL;\n+\tint local_error = 0;\n \n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n@@ -1987,6 +1988,9 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\t\t\tbreak;\n \t\t\t}\n \n+\t\t\t/* parse parent in case it is in a base graph */\n+\t\t\tparse_commit_in_graph_one(r, g, graph_parents->item);\n+\n \t\t\tif (!oideq(&graph_parents->item->object.oid, &odb_parents->item->object.oid))\n \t\t\t\tgraph_report(_(\"commit-graph parent for %s is %s != %s\"),\n \t\t\t\t\t oid_to_hex(&cur_oid),\n@@ -2038,7 +2042,12 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t}\n \tstop_progress(&progress);\n \n-\treturn verify_commit_graph_error;\n+\tlocal_error = verify_commit_graph_error;\n+\n+\tif (!(flags & COMMIT_GRAPH_VERIFY_SHALLOW) && g->base_graph)\n+\t\tlocal_error |= verify_commit_graph(r, g->base_graph, flags);\n+\n+\treturn local_error;\n }\n \n void free_commit_graph(struct commit_graph *g)\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 194acab2b7..84e5e91fc6 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -61,7 +61,7 @@ struct commit_graph {\n };\n \n struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st);\n-\n+struct commit_graph *read_commit_graph_one(struct repository *r, const char *obj_dir);\n struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t\t\t\t\tsize_t graph_size);\n \n@@ -89,7 +89,9 @@ int write_commit_graph(const char *obj_dir,\n \t\t unsigned int flags,\n \t\t const struct split_commit_graph_opts *split_opts);\n \n-int verify_commit_graph(struct repository *r, struct commit_graph *g);\n+#define COMMIT_GRAPH_VERIFY_SHALLOW\t(1 << 0)\n+\n+int verify_commit_graph(struct repository *r, struct commit_graph *g, int flags);\n \n void close_commit_graph(struct repository *);\n void free_commit_graph(struct commit_graph *);\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 1b699a543c..3df90ae58f 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -216,4 +216,66 @@ test_expect_success 'test merge stragety constants' '\n \t)\n '\n \n+corrupt_file() {\n+\tfile=$1\n+\tpos=$2\n+\tdata=\"${3:-\\0}\"\n+\tprintf \"$data\" | dd of=\"$file\" bs=1 seek=\"$pos\" conv=notrunc\n+}\n+\n+test_expect_success 'verify hashes along chain, even in shallow' '\n+\tgit clone --no-hardlinks . verify &&\n+\t(\n+\t\tcd verify &&\n+\t\tgit commit-graph verify &&\n+\t\tbase_file=$graphdir/graph-$(head -n 1 $graphdir/commit-graph-chain).graph &&\n+\t\tcorrupt_file \"$base_file\" 1760 \"\\01\" &&\n+\t\ttest_must_fail git commit-graph verify --shallow 2>test_err &&\n+\t\tgrep -v \"^+\" test_err >err &&\n+\t\ttest_i18ngrep \"incorrect checksum\" err\n+\t)\n+'\n+\n+test_expect_success 'verify --shallow does not check base contents' '\n+\tgit clone --no-hardlinks . verify-shallow &&\n+\t(\n+\t\tcd verify-shallow &&\n+\t\tgit commit-graph verify &&\n+\t\tbase_file=$graphdir/graph-$(head -n 1 $graphdir/commit-graph-chain).graph &&\n+\t\tcorrupt_file \"$base_file\" 1000 \"\\01\" &&\n+\t\tgit commit-graph verify --shallow &&\n+\t\ttest_must_fail git commit-graph verify 2>test_err &&\n+\t\tgrep -v \"^+\" test_err >err &&\n+\t\ttest_i18ngrep \"incorrect checksum\" err\n+\t)\n+'\n+\n+test_expect_success 'warn on base graph chunk incorrect' '\n+\tgit clone --no-hardlinks . base-chunk &&\n+\t(\n+\t\tcd base-chunk &&\n+\t\tgit commit-graph verify &&\n+\t\tbase_file=$graphdir/graph-$(tail -n 1 $graphdir/commit-graph-chain).graph &&\n+\t\tcorrupt_file \"$base_file\" 1376 \"\\01\" &&\n+\t\tgit commit-graph verify --shallow 2>test_err &&\n+\t\tgrep -v \"^+\" test_err >err &&\n+\t\ttest_i18ngrep \"commit-graph chain does not match\" err\n+\t)\n+'\n+\n+test_expect_success 'verify after commit-graph-chain corruption' '\n+\tgit clone --no-hardlinks . verify-chain &&\n+\t(\n+\t\tcd verify-chain &&\n+\t\tcorrupt_file \"$graphdir/commit-graph-chain\" 60 \"G\" &&\n+\t\tgit commit-graph verify 2>test_err &&\n+\t\tgrep -v \"^+\" test_err >err &&\n+\t\ttest_i18ngrep \"invalid commit-graph chain\" err &&\n+\t\tcorrupt_file \"$graphdir/commit-graph-chain\" 60 \"A\" &&\n+\t\tgit commit-graph verify 2>test_err &&\n+\t\tgrep -v \"^+\" test_err >err &&\n+\t\ttest_i18ngrep \"unable to find all commit-graph files\" err\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"376841","messageId":"101792b92db169a88d74d5f9292a7de30799e90c.1559932699.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 15/16] commit-graph: test octopus merges with --split","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-07T18:38:32Z","receivedAt":"2019-06-07T18:38:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nOctopus merges require an extra chunk of data in the commit-graph\nfile format. Create a test that ensures the new --split option\ncontinues to work with an octopus merge. Specifically, ensure\nthat the octopus merge has parents across layers to truly check\nthat our graph position logic holds up correctly.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5324-split-commit-graph.sh | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex e8df35c30b..704def70bb 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -290,4 +290,15 @@ test_expect_success 'verify after commit-graph-chain corruption' '\n \t)\n '\n \n+test_expect_success 'add octopus merge' '\n+\tgit reset --hard commits/10 &&\n+\tgit merge commits/3 commits/4 &&\n+\tgit branch merge/octopus &&\n+\tgit commit-graph write --reachable --split &&\n+\tgit commit-graph verify &&\n+\ttest_line_count = 3 $graphdir/commit-graph-chain\n+'\n+\n+graph_git_behavior 'graph exists' merge/octopus commits/12\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"376842","messageId":"84a3ff7c61a27bab2d7f01267f09b314fb292d57.1559932699.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 16/16] commit-graph: test --split across alternate without --split","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-07T18:38:33Z","receivedAt":"2019-06-07T18:38:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWe allow sharing commit-graph files across alternates. When we are\nwriting a split commit-graph, we allow adding tip graph files that\nare not in the alternate, but include commits from our local repo.\n\nHowever, if our alternate is not using the split commit-graph format,\nits file is at .git/objects/info/commit-graph and we are trying to\nwrite files in .git/objects/info/commit-graphs/graph-{hash}.graph.\n\nWe already have logic to ensure we do not merge across alternate\nboundaries, but we also cannot have a commit-graph chain to our\nalternate if uses the old filename structure.\n\nCreate a test that verifies we create a new split commit-graph\nwith only one level and we do not modify the existing commit-graph\nin the alternate.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5324-split-commit-graph.sh | 15 +++++++++++++++\n 1 file changed, 15 insertions(+)\n\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 704def70bb..130f2baf44 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -301,4 +301,19 @@ test_expect_success 'add octopus merge' '\n \n graph_git_behavior 'graph exists' merge/octopus commits/12\n \n+test_expect_success 'split across alternate where alternate is not split' '\n+\tgit commit-graph write --reachable &&\n+\ttest_path_is_file .git/objects/info/commit-graph &&\n+\tcp .git/objects/info/commit-graph . &&\n+\tgit clone --no-hardlinks . alt-split &&\n+\t(\n+\t\tcd alt-split &&\n+\t\techo \"$(pwd)\"/../.git/objects >.git/objects/info/alternates &&\n+\t\ttest_commit 18 &&\n+\t\tgit commit-graph write --reachable --split &&\n+\t\ttest_line_count = 1 $graphdir/commit-graph-chain\n+\t) &&\n+\ttest_cmp commit-graph .git/objects/info/commit-graph\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"376843","messageId":"795ea36ff41a5838b1dda781b48163f8e20993c1.1559932699.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 14/16] commit-graph: clean up chains after flattened write","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-07T18:38:31Z","receivedAt":"2019-06-07T18:38:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf we write a commit-graph file without the split option, then\nwe write to $OBJDIR/info/commit-graph and start to ignore\nthe chains in $OBJDIR/info/commit-graphs/.\n\nUnlink the commit-graph-chain file and expire the graph-{hash}.graph\nfiles in $OBJDIR/info/commit-graphs/ during every write.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 12 +++++++++---\n t/t5324-split-commit-graph.sh | 12 ++++++++++++\n 2 files changed, 21 insertions(+), 3 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 431ad2cf1d..8842f93910 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1685,6 +1685,12 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \n \tif (ctx->split_opts && ctx->split_opts->expire_time)\n \t\texpire_time -= ctx->split_opts->expire_time;\n+\tif (!ctx->split) {\n+\t\tchar *chain_file_name = get_chain_filename(ctx->obj_dir);\n+\t\tunlink(chain_file_name);\n+\t\tfree(chain_file_name);\n+\t\tctx->num_commit_graphs_after = 0;\n+\t}\n \n \tstrbuf_addstr(&path, ctx->obj_dir);\n \tstrbuf_addstr(&path, \"/info/commit-graphs\");\n@@ -1839,10 +1845,10 @@ int write_commit_graph(const char *obj_dir,\n \n \tres = write_commit_graph_file(ctx);\n \n-\tif (ctx->split) {\n+\tif (ctx->split)\n \t\tmark_commit_graphs(ctx);\n-\t\texpire_commit_graphs(ctx);\n-\t}\n+\n+\texpire_commit_graphs(ctx);\n \n cleanup:\n \tfree(ctx->graph_name);\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 3df90ae58f..e8df35c30b 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -216,6 +216,18 @@ test_expect_success 'test merge stragety constants' '\n \t)\n '\n \n+test_expect_success 'remove commit-graph-chain file after flattening' '\n+\tgit clone . flatten &&\n+\t(\n+\t\tcd flatten &&\n+\t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\t\tgit commit-graph write --reachable &&\n+\t\ttest_path_is_missing $graphdir/commit-graph-chain &&\n+\t\tls $graphdir >graph-files &&\n+\t\ttest_line_count = 0 graph-files\n+\t)\n+'\n+\n corrupt_file() {\n \tfile=$1\n \tpos=$2\n-- \ngitgitgadget\n\n"},{"id":"376845","messageId":"66be8b03a80373b99fdc53b83f774ff6bd60380d.1559932699.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 12/16] commit-graph: create options for split files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-07T18:38:30Z","receivedAt":"2019-06-07T18:38:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe split commit-graph feature is now fully implemented, but needs\nsome more run-time configurability. Allow direct callers to 'git\ncommit-graph write --split' to specify the values used in the\nmerge strategy and the expire time.\n\nUpdate the documentation to specify these values.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 21 ++++++++++-\n Documentation/technical/commit-graph.txt | 7 ++--\n builtin/commit-graph.c | 20 ++++++++--\n builtin/commit.c | 2 +-\n builtin/gc.c | 3 +-\n commit-graph.c | 35 ++++++++++++------\n commit-graph.h | 12 +++++-\n t/t5324-split-commit-graph.sh | 47 ++++++++++++++++++++++++\n 8 files changed, 124 insertions(+), 23 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 624470e198..365e145e82 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -26,7 +26,7 @@ OPTIONS\n \tUse given directory for the location of packfiles and commit-graph\n \tfile. This parameter exists to specify the location of an alternate\n \tthat only has the objects directory, not a full `.git` directory. The\n-\tcommit-graph file is expected to be at `/info/commit-graph` and\n+\tcommit-graph file is expected to be in the `/info` directory and\n \tthe packfiles are expected to be in `/pack`.\n \n \n@@ -51,6 +51,25 @@ or `--stdin-packs`.)\n +\n With the `--append` option, include all commits that are present in the\n existing commit-graph file.\n++\n+With the `--split` option, write the commit-graph as a chain of multiple\n+commit-graph files stored in `/info/commit-graphs`. The new commits\n+not already in the commit-graph are added in a new \"tip\" file. This file\n+is merged with the existing file if the following merge conditions are\n+met:\n++\n+* If `--size-multiple=` is not specified, let `X` equal 2. If the new\n+tip file would have `N` commits and the previous tip has `M` commits and\n+`X` times `N` is greater than `M`, instead merge the two files into a\n+single file.\n++\n+* If `--max-commits=` is specified with `M` a positive integer, and the\n+new tip file would have more than `M` commits, then instead merge the new\n+tip with the previous tip.\n++\n+Finally, if `--expire-time=` is not specified, let `datetime`\n+be the current time. After writing the split commit-graph, delete all\n+unused commit-graph whose modified times are older than `datetime`.\n \n 'read'::\n \ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex aed4350a59..729fbcb32f 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -248,10 +248,11 @@ When writing a set of commits that do not exist in the commit-graph stack of\n height N, we default to creating a new file at level N + 1. We then decide to\n merge with the Nth level if one of two conditions hold:\n \n- 1. The expected file size for level N + 1 is at least half the file size for\n- level N.\n+ 1. `--size-multiple=` is specified or X = 2, and the number of commits in\n+ level N is less than X times the number of commits in level N + 1.\n \n- 2. Level N + 1 contains more than 64,0000 commits.\n+ 2. `--max-commits=` is specified with non-zero C and the number of commits\n+ in level N + 1 is more than C commits.\n \n This decision cascades down the levels: when we merge a level we create a new\n set of commits that then compares to the next level.\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex c2c07d3917..cc4af67820 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -10,7 +10,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph verify [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] \"),\n \tNULL\n };\n \n@@ -25,7 +25,7 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] \"),\n \tNULL\n };\n \n@@ -135,6 +135,7 @@ static int graph_read(int argc, const char **argv)\n }\n \n extern int read_replace_refs;\n+static struct split_commit_graph_opts split_opts;\n \n static int graph_write(int argc, const char **argv)\n {\n@@ -158,9 +159,19 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"include all commits already in the commit-graph file\")),\n \t\tOPT_BOOL(0, \"split\", &opts.split,\n \t\t\tN_(\"allow writing an incremental commit-graph file\")),\n+\t\tOPT_INTEGER(0, \"max-commits\", &split_opts.max_commits,\n+\t\t\tN_(\"maximum number of commits in a non-base split commit-graph\")),\n+\t\tOPT_INTEGER(0, \"size-multiple\", &split_opts.size_multiple,\n+\t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n+\t\tOPT_EXPIRY_DATE(0, \"expire-time\", &split_opts.expire_time,\n+\t\t\tN_(\"maximum number of commits in a non-base split commit-graph\")),\n \t\tOPT_END(),\n \t};\n \n+\tsplit_opts.size_multiple = 2;\n+\tsplit_opts.max_commits = 0;\n+\tsplit_opts.expire_time = 0;\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\n@@ -177,7 +188,7 @@ static int graph_write(int argc, const char **argv)\n \tread_replace_refs = 0;\n \n \tif (opts.reachable)\n-\t\treturn write_commit_graph_reachable(opts.obj_dir, flags);\n+\t\treturn write_commit_graph_reachable(opts.obj_dir, flags, &split_opts);\n \n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n@@ -197,7 +208,8 @@ static int graph_write(int argc, const char **argv)\n \tresult = write_commit_graph(opts.obj_dir,\n \t\t\t\t pack_indexes,\n \t\t\t\t commit_hex,\n-\t\t\t\t flags);\n+\t\t\t\t flags,\n+\t\t\t\t &split_opts);\n \n \tUNLEAK(lines);\n \treturn result;\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex b001ef565d..9216e9c043 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1670,7 +1670,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\t \"not exceeded, and then \\\"git reset HEAD\\\" to recover.\"));\n \n \tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n-\t write_commit_graph_reachable(get_object_directory(), 0))\n+\t write_commit_graph_reachable(get_object_directory(), 0, NULL))\n \t\treturn 1;\n \n \trepo_rerere(the_repository, 0);\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex df2573f124..2ab590ffd4 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -666,7 +666,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \n \tif (gc_write_commit_graph &&\n \t write_commit_graph_reachable(get_object_directory(),\n-\t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_PROGRESS : 0))\n+\t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_PROGRESS : 0,\n+\t\t\t\t\t NULL))\n \t\treturn 1;\n \n \tif (auto_gc && too_many_loose_objects())\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 779a628c95..888eaa7d03 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -766,6 +766,8 @@ struct write_commit_graph_context {\n \tunsigned append:1,\n \t\t report_progress:1,\n \t\t split:1;\n+\n+\tconst struct split_commit_graph_opts *split_opts;\n };\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n@@ -1114,14 +1116,15 @@ static int add_ref_to_list(const char *refname,\n \treturn 0;\n }\n \n-int write_commit_graph_reachable(const char *obj_dir, unsigned int flags)\n+int write_commit_graph_reachable(const char *obj_dir, unsigned int flags,\n+\t\t\t\t const struct split_commit_graph_opts *split_opts)\n {\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n \tint result;\n \n \tfor_each_ref(add_ref_to_list, &list);\n \tresult = write_commit_graph(obj_dir, NULL, &list,\n-\t\t\t\t flags);\n+\t\t\t\t flags, split_opts);\n \n \tstring_list_clear(&list, 0);\n \treturn result;\n@@ -1496,20 +1499,25 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \treturn 0;\n }\n \n-static int split_strategy_max_commits = 64000;\n-static float split_strategy_size_mult = 2.0f;\n-\n static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n {\n \tstruct commit_graph *g = ctx->r->objects->commit_graph;\n \tuint32_t num_commits = ctx->commits.nr;\n \tuint32_t i;\n \n+\tint max_commits = 0;\n+\tint size_mult = 2;\n+\n+\tif (ctx->split_opts) {\n+\t\tmax_commits = ctx->split_opts->max_commits;\n+\t\tsize_mult = ctx->split_opts->size_multiple;\n+\t}\n+\n \tg = ctx->r->objects->commit_graph;\n \tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n \n-\twhile (g && (g->num_commits <= split_strategy_size_mult * num_commits ||\n-\t\t num_commits > split_strategy_max_commits)) {\n+\twhile (g && (g->num_commits <= size_mult * num_commits ||\n+\t\t (max_commits && num_commits > max_commits))) {\n \t\tif (strcmp(g->obj_dir, ctx->obj_dir))\n \t\t\tbreak;\n \n@@ -1673,7 +1681,10 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \tDIR *dir;\n \tstruct dirent *de;\n \tsize_t dirnamelen;\n-\ttime_t expire_time = time(NULL);\n+\ttimestamp_t expire_time = time(NULL);\n+\n+\tif (ctx->split_opts && ctx->split_opts->expire_time)\n+\t\texpire_time -= ctx->split_opts->expire_time;\n \n \tstrbuf_addstr(&path, ctx->obj_dir);\n \tstrbuf_addstr(&path, \"/info/commit-graphs\");\n@@ -1717,7 +1728,8 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t unsigned int flags)\n+\t\t unsigned int flags,\n+\t\t const struct split_commit_graph_opts *split_opts)\n {\n \tstruct write_commit_graph_context *ctx;\n \tuint32_t i, count_distinct = 0;\n@@ -1732,6 +1744,7 @@ int write_commit_graph(const char *obj_dir,\n \tctx->append = flags & COMMIT_GRAPH_APPEND ? 1 : 0;\n \tctx->report_progress = flags & COMMIT_GRAPH_PROGRESS ? 1 : 0;\n \tctx->split = flags & COMMIT_GRAPH_SPLIT ? 1 : 0;\n+\tctx->split_opts = split_opts;\n \n \tif (ctx->split) {\n \t\tstruct commit_graph *g;\n@@ -1759,8 +1772,8 @@ int write_commit_graph(const char *obj_dir,\n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n \n-\tif (ctx->split && ctx->oids.alloc > split_strategy_max_commits)\n-\t\tctx->oids.alloc = split_strategy_max_commits;\n+\tif (ctx->split && split_opts && ctx->oids.alloc > split_opts->max_commits)\n+\t\tctx->oids.alloc = split_opts->max_commits;\n \n \tif (ctx->append) {\n \t\tprepare_commit_graph_one(ctx->r, ctx->obj_dir);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 10466bc064..194acab2b7 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -75,11 +75,19 @@ int generation_numbers_enabled(struct repository *r);\n #define COMMIT_GRAPH_PROGRESS (1 << 1)\n #define COMMIT_GRAPH_SPLIT (1 << 2)\n \n-int write_commit_graph_reachable(const char *obj_dir, unsigned int flags);\n+struct split_commit_graph_opts {\n+\tint size_multiple;\n+\tint max_commits;\n+\ttimestamp_t expire_time;\n+};\n+\n+int write_commit_graph_reachable(const char *obj_dir, unsigned int flags,\n+\t\t\t\t const struct split_commit_graph_opts *split_opts);\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t unsigned int flags);\n+\t\t unsigned int flags,\n+\t\t const struct split_commit_graph_opts *split_opts);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 76068ee407..1b699a543c 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -169,4 +169,51 @@ test_expect_success 'create fork and chain across alternate' '\n \n graph_git_behavior 'alternate: commit 13 vs 6' commits/13 commits/6\n \n+test_expect_success 'test merge stragety constants' '\n+\tgit clone . merge-2 &&\n+\t(\n+\t\tcd merge-2 &&\n+\t\tgit config core.commitGraph true &&\n+\t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\t\ttest_commit 14 &&\n+\t\tgit commit-graph write --reachable --split --size-multiple=2 &&\n+\t\ttest_line_count = 3 $graphdir/commit-graph-chain\n+\n+\t) &&\n+\tgit clone . merge-10 &&\n+\t(\n+\t\tcd merge-10 &&\n+\t\tgit config core.commitGraph true &&\n+\t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\t\ttest_commit 14 &&\n+\t\tgit commit-graph write --reachable --split --size-multiple=10 &&\n+\t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n+\t\tls $graphdir/graph-*.graph >graph-files &&\n+\t\ttest_line_count = 1 graph-files\n+\t) &&\n+\tgit clone . merge-10-expire &&\n+\t(\n+\t\tcd merge-10-expire &&\n+\t\tgit config core.commitGraph true &&\n+\t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\t\ttest_commit 15 &&\n+\t\tgit commit-graph write --reachable --split --size-multiple=10 --expire-time=1980-01-01 &&\n+\t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n+\t\tls $graphdir/graph-*.graph >graph-files &&\n+\t\ttest_line_count = 3 graph-files\n+\t) &&\n+\tgit clone --no-hardlinks . max-commits &&\n+\t(\n+\t\tcd max-commits &&\n+\t\tgit config core.commitGraph true &&\n+\t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\t\ttest_commit 16 &&\n+\t\ttest_commit 17 &&\n+\t\tgit commit-graph write --reachable --split --max-commits=1 &&\n+\t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n+\t\tls $graphdir/graph-*.graph >graph-files &&\n+\t\ttest_line_count = 1 graph-files\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"376844","messageId":"5238bbbec35d18918a95c18bd289abfe93e89933.1559932699.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 06/16] commit-graph: rearrange chunk count logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-07T18:38:25Z","receivedAt":"2019-06-07T18:38:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe number of chunks in a commit-graph file can change depending on\nwhether we need the Extra Edges Chunk. We are going to add more optional\nchunks, and it will be helpful to rearrange this logic around the chunk\ncount before doing so.\n\nSpecifically, we need to finalize the number of chunks before writing\nthe commit-graph header. Further, we also need to fill out the chunk\nlookup table dynamically and using \"num_chunks\" as we add optional\nchunks is useful for adding optional chunks in the future.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 35 +++++++++++++++++++++--------------\n 1 file changed, 21 insertions(+), 14 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0ff1a4f379..5f25dff193 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1213,7 +1213,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tuint64_t chunk_offsets[5];\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n-\tint num_chunks = ctx->num_extra_edges ? 4 : 3;\n+\tint num_chunks = 3;\n \n \tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n \tif (safe_create_leading_directories(ctx->graph_name)) {\n@@ -1226,27 +1226,34 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \n-\thashwrite_be32(f, GRAPH_SIGNATURE);\n-\n-\thashwrite_u8(f, GRAPH_VERSION);\n-\thashwrite_u8(f, oid_version());\n-\thashwrite_u8(f, num_chunks);\n-\thashwrite_u8(f, 0); /* unused padding byte */\n-\n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n \tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n-\tif (ctx->num_extra_edges)\n-\t\tchunk_ids[3] = GRAPH_CHUNKID_EXTRAEDGES;\n-\telse\n-\t\tchunk_ids[3] = 0;\n-\tchunk_ids[4] = 0;\n+\tif (ctx->num_extra_edges) {\n+\t\tchunk_ids[num_chunks] = GRAPH_CHUNKID_EXTRAEDGES;\n+\t\tnum_chunks++;\n+\t}\n+\n+\tchunk_ids[num_chunks] = 0;\n \n \tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n \tchunk_offsets[2] = chunk_offsets[1] + hashsz * ctx->commits.nr;\n \tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * ctx->commits.nr;\n-\tchunk_offsets[4] = chunk_offsets[3] + 4 * ctx->num_extra_edges;\n+\n+\tnum_chunks = 3;\n+\tif (ctx->num_extra_edges) {\n+\t\tchunk_offsets[num_chunks + 1] = chunk_offsets[num_chunks] +\n+\t\t\t\t\t\t4 * ctx->num_extra_edges;\n+\t\tnum_chunks++;\n+\t}\n+\n+\thashwrite_be32(f, GRAPH_SIGNATURE);\n+\n+\thashwrite_u8(f, GRAPH_VERSION);\n+\thashwrite_u8(f, oid_version());\n+\thashwrite_u8(f, num_chunks);\n+\thashwrite_u8(f, 0);\n \n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tuint32_t chunk_write[3];\n-- \ngitgitgadget\n\n"},{"id":"376846","messageId":"554880e3d7ea157d79a606f0e45124b77fd24ab4.1559932699.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 11/16] commit-graph: expire commit-graph files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-07T18:38:29Z","receivedAt":"2019-06-07T18:38:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we merge commit-graph files in a commit-graph chain, we should clean\nup the files that are no longer used.\n\nThis change introduces an 'expiry_window' value to the context, which is\nalways zero (for now). We then check the modified time of each\ngraph-{hash}.graph file in the $OBJDIR/info/commit-graphs folder and\nunlink the files that are older than the expiry_window.\n\nSince this is always zero, this immediately clears all unused graph\nfiles. We will update the value to match a config setting in a future\nchange.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 15 ++++++\n commit-graph.c | 69 ++++++++++++++++++++++++\n t/t5324-split-commit-graph.sh | 2 +-\n 3 files changed, 85 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex 473032e476..aed4350a59 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -266,6 +266,21 @@ The merge strategy values (2 for the size multiple, 64,000 for the maximum\n number of commits) could be extracted into config settings for full\n flexibility.\n \n+## Deleting graph-{hash} files\n+\n+After a new tip file is written, some `graph-{hash}` files may no longer\n+be part of a chain. It is important to remove these files from disk, eventually.\n+The main reason to delay removal is that another process could read the\n+`commit-graph-chain` file before it is rewritten, but then look for the\n+`graph-{hash}` files after they are deleted.\n+\n+To allow holding old split commit-graphs for a while after they are unreferenced,\n+we update the modified times of the files when they become unreferenced. Then,\n+we scan the `$OBJDIR/info/commit-graphs/` directory for `graph-{hash}`\n+files whose modified times are older than a given expiry window. This window\n+defaults to zero, but can be changed using command-line arguments or a config\n+setting.\n+\n ## Chains across multiple object directories\n \n In a repo with alternates, we look for the `commit-graph-chain` file starting\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 22b167d9eb..779a628c95 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1650,6 +1650,70 @@ static void merge_commit_graphs(struct write_commit_graph_context *ctx)\n \tsort_and_scan_merged_commits(ctx);\n }\n \n+static void mark_commit_graphs(struct write_commit_graph_context *ctx)\n+{\n+\tuint32_t i;\n+\ttime_t now = time(NULL);\n+\n+\tfor (i = ctx->num_commit_graphs_after - 1; i < ctx->num_commit_graphs_before; i++) {\n+\t\tstruct stat st;\n+\t\tstruct utimbuf updated_time;\n+\n+\t\tstat(ctx->commit_graph_filenames_before[i], &st);\n+\n+\t\tupdated_time.actime = st.st_atime;\n+\t\tupdated_time.modtime = now;\n+\t\tutime(ctx->commit_graph_filenames_before[i], &updated_time);\n+\t}\n+}\n+\n+static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tDIR *dir;\n+\tstruct dirent *de;\n+\tsize_t dirnamelen;\n+\ttime_t expire_time = time(NULL);\n+\n+\tstrbuf_addstr(&path, ctx->obj_dir);\n+\tstrbuf_addstr(&path, \"/info/commit-graphs\");\n+\tdir = opendir(path.buf);\n+\n+\tif (!dir) {\n+\t\tstrbuf_release(&path);\n+\t\treturn;\n+\t}\n+\n+\tstrbuf_addch(&path, '/');\n+\tdirnamelen = path.len;\n+\twhile ((de = readdir(dir)) != NULL) {\n+\t\tstruct stat st;\n+\t\tuint32_t i, found = 0;\n+\n+\t\tstrbuf_setlen(&path, dirnamelen);\n+\t\tstrbuf_addstr(&path, de->d_name);\n+\n+\t\tstat(path.buf, &st);\n+\n+\t\tif (st.st_mtime > expire_time)\n+\t\t\tcontinue;\n+\t\tif (path.len < 6 || strcmp(path.buf + path.len - 6, \".graph\"))\n+\t\t\tcontinue;\n+\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++) {\n+\t\t\tif (!strcmp(ctx->commit_graph_filenames_after[i],\n+\t\t\t\t path.buf)) {\n+\t\t\t\tfound = 1;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (!found)\n+\t\t\tunlink(path.buf);\n+\n+\t}\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -1762,6 +1826,11 @@ int write_commit_graph(const char *obj_dir,\n \n \tres = write_commit_graph_file(ctx);\n \n+\tif (ctx->split) {\n+\t\tmark_commit_graphs(ctx);\n+\t\texpire_commit_graphs(ctx);\n+\t}\n+\n cleanup:\n \tfree(ctx->graph_name);\n \tfree(ctx->commits.list);\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 46f0832f68..76068ee407 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -141,7 +141,7 @@ test_expect_success 'add one commit, write a merged graph' '\n \ttest_path_is_file $graphdir/commit-graph-chain &&\n \ttest_line_count = 2 $graphdir/commit-graph-chain &&\n \tls $graphdir/graph-*.graph >graph-files &&\n-\ttest_line_count = 4 graph-files &&\n+\ttest_line_count = 2 graph-files &&\n \tverify_chain_files_exist $graphdir\n '\n \n-- \ngitgitgadget\n\n"},{"id":"376851","messageId":"xmqqwohxnjmu.fsf@gitster-ct.c.googlers.com","threadId":"51052","inReplyTo":"c42e683ef644e7d34614eb74156e5a2b31392758.1559830527.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 08/14] commit-graph: add --split option to builtin","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-07T21:57:45Z","receivedAt":"2019-06-07T21:57:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> -\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n> +\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n\nNot a comment on the essential part of this feature, but are append\nand split meant to be mutually exclusive?\n\nOne thing that is somewhat curious is that this commit itself does\nnot do much that would affect the correctness of how GRAPH_SPLIT\nworks, as the actual machinery was introduced in the previous step\nand this step merely makes it accessible from the outside. So I had\nto look at the previous step to see if the internal machinery had\nsome safety valve to catch the combination and flag it as an error\nor something like that, but if I am not mistaken, there is nothing\nthat prevents both from being specified.\n\n> @@ -156,6 +156,8 @@ static int graph_write(int argc, const char **argv)\n> \t\t\tN_(\"start walk at commits listed by stdin\")),\n> \t\tOPT_BOOL(0, \"append\", &opts.append,\n> \t\t\tN_(\"include all commits already in the commit-graph file\")),\n> +\t\tOPT_BOOL(0, \"split\", &opts.split,\n> +\t\t\tN_(\"allow writing an incremental commit-graph file\")),\n> \t\tOPT_END(),\n> \t};\n> \n> @@ -169,6 +171,8 @@ static int graph_write(int argc, const char **argv)\n> \t\topts.obj_dir = get_object_directory();\n> \tif (opts.append)\n> \t\tflags |= COMMIT_GRAPH_APPEND;\n> +\tif (opts.split)\n> +\t\tflags |= COMMIT_GRAPH_SPLIT;\n\n"},{"id":"376973","messageId":"xmqqy329kt9j.fsf@gitster-ct.c.googlers.com","threadId":"51052","inReplyTo":"fc3423046b9b2fc43bf8323dd41281f0a29b9d0a.1559932699.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 04/16] commit-graph: load commit-graph chains","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-10T21:47:04Z","receivedAt":"2019-06-10T21:47:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> +\tfp = fopen(chain_name, \"r\");\n> +\tstat_res = stat(chain_name, &st);\n> +\tfree(chain_name);\n> +\n> +\tif (!fp ||\n> +\t stat_res ||\n> +\t st.st_size <= the_hash_algo->hexsz)\n> +\t\treturn NULL;\n> +\n> +\tcount = st.st_size / (the_hash_algo->hexsz + 1);\n> +\toids = xcalloc(count, sizeof(struct object_id));\n> +\n> +\tfor (i = 0; i < count && valid; i++) {\n> +\t\tchar *graph_name;\n> +\t\tstruct commit_graph *g;\n> +\n> +\t\tif (strbuf_getline_lf(&line, fp) == EOF)\n> +\t\t\tbreak;\n> +\n> +\t\tif (get_oid_hex(line.buf, &oids[i])) {\n> +\t\t\twarning(_(\"invalid commit-graph chain: line '%s' not a hash\"),\n> +\t\t\t\tline.buf);\n> +\t\t\tvalid = 0;\n> +\t\t\tbreak;\n> +\t\t}\n> +\n> +\t\tgraph_name = get_split_graph_filename(obj_dir, line.buf);\n> +\t\tg = load_commit_graph_one(graph_name);\n> +\t\tfree(graph_name);\n> +\n> +\t\tif (g && add_graph_to_chain(g, graph_chain, oids, i))\n> +\t\t\tgraph_chain = g;\n> +\t\telse\n> +\t\t\tvalid = 0;\n> +\t}\n\nAt this point, if 'i' is smaller than 'count', that would indicate\nthat the file was corrupt (we hit one of the 'break' in the loop).\n\nHow would we handle such an error? It appears that the strategy\ntaken in this loop is to \"read as many as possible without an error\nand then give up upon the first error---keep whatever we have read\nso far\", so from that point of view, the only thing that is missing\nmay be a warning() after the loop.\n\n> +\n> +\tfree(oids);\n> +\tfclose(fp);\n> +\n> +\treturn graph_chain;\n> +}\n> +\n> +static struct commit_graph *read_commit_graph_one(struct repository *r, const char *obj_dir)\n> +{\n> +\tstruct commit_graph *g = load_commit_graph_v1(r, obj_dir);\n> +\n> +\tif (!g)\n> +\t\tg = load_commit_graph_chain(r, obj_dir);\n> +\n> +\treturn g;\n> +}\n> +\n> static void prepare_commit_graph_one(struct repository *r, const char *obj_dir)\n> {\n> -\tchar *graph_name;\n> \n> \tif (r->objects->commit_graph)\n> \t\treturn;\n> \n> -\tgraph_name = get_commit_graph_filename(obj_dir);\n> -\tr->objects->commit_graph =\n> -\t\tload_commit_graph_one(graph_name);\n> -\n> -\tFREE_AND_NULL(graph_name);\n> +\tr->objects->commit_graph = read_commit_graph_one(r, obj_dir);\n> }\n> \n> /*\n"},{"id":"376988","messageId":"74c1009c-748a-6e68-7c87-c03b01ebf885@gmail.com","threadId":"51052","inReplyTo":"xmqqy329kt9j.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v5 04/16] commit-graph: load commit-graph chains","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-06-10T23:41:44Z","receivedAt":"2019-06-10T23:41:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/10/2019 5:47 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n\n>> +\t\tif (get_oid_hex(line.buf, &oids[i])) {\n>> +\t\t\twarning(_(\"invalid commit-graph chain: line '%s' not a hash\"),\n>> +\t\t\t\tline.buf);\n>> +\t\t\tvalid = 0;\n>> +\t\t\tbreak;\n>> +\t\t}\n>> +\n>> +\t\tgraph_name = get_split_graph_filename(obj_dir, line.buf);\n>> +\t\tg = load_commit_graph_one(graph_name);\n>> +\t\tfree(graph_name);\n>> +\n>> +\t\tif (g && add_graph_to_chain(g, graph_chain, oids, i))\n>> +\t\t\tgraph_chain = g;\n>> +\t\telse\n>> +\t\t\tvalid = 0;\n>> +\t}\n> \n> At this point, if 'i' is smaller than 'count', that would indicate\n> that the file was corrupt (we hit one of the 'break' in the loop).\n\nThis is correct.\n \n> How would we handle such an error? It appears that the strategy\n> taken in this loop is to \"read as many as possible without an error\n> and then give up upon the first error---keep whatever we have read\n> so far\", so from that point of view, the only thing that is missing\n> may be a warning() after the loop.\n\nBased on previous experience with the commit-graph feature, I am\ndeliberately trying to ensure the commit-graph feature does not lead\nto a failure case whenever possible. If we can continue with the data\nat hand (and maybe lose some performance benefits if we cannot read\na graph file) then we should keep going.\n\nI believe that with this \"only warning\" case, the next write would\nfix the issue. The chain builds only from the graphs that successfully\nload. That way, this should be a \"self-healing\" state.\n\nThanks,\n-Stolee\n"},{"id":"377005","messageId":"2dd1552f-72a3-c676-f46c-1393717ffd82@gmail.com","threadId":"51052","inReplyTo":"xmqqwohxnjmu.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 08/14] commit-graph: add --split option to builtin","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-06-11T12:51:50Z","receivedAt":"2019-06-11T12:51:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/7/2019 5:57 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> -\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n>> +\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n> \n> Not a comment on the essential part of this feature, but are append\n> and split meant to be mutually exclusive?\n\nI think the more correct thing to say is that `--split` implies `--append`:\nAs we write a new tip graph, we do not discard any commits from the lower\nlayers, even if we merge them together.\n\nHowever, the interaction between the two features is not currently\nguarded well. If both `--split` and `--append` are specified, we\nwill allocate a larger initial array to store commit ids and then\nscan the existing commit-graph for commits.\n\nLikely the following diff would be the correct place to guard against\na combined set of options, since these are passed to the method from\nother builtins. This block is inside an if (ctx->split) condition.\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8842f93910..9cf5296e4e 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1756,6 +1756,13 @@ int write_commit_graph(const char *obj_dir,\n struct commit_graph *g;\n prepare_commit_graph(ctx->r);\n\n+ /*\n+ * Writing a split graph only adds commits, implying\n+ * a similar behavior to 'append' without scanning the\n+ * existing commit-graph files.\n+ */\n+ ctx->append = 0;\n+\n g = ctx->r->objects->commit_graph;\n\n while (g) {\n\n\n> One thing that is somewhat curious is that this commit itself does\n> not do much that would affect the correctness of how GRAPH_SPLIT\n> works, as the actual machinery was introduced in the previous step\n> and this step merely makes it accessible from the outside. So I had\n> to look at the previous step to see if the internal machinery had\n> some safety valve to catch the combination and flag it as an error\n> or something like that, but if I am not mistaken, there is nothing\n> that prevents both from being specified.\n\nThere is nothing that stops them both from being specified, so this\nusage describes a strictness that is not enforced. I can change this\nto be \"[--append] [--split]\" if that is preferred.\n\nThanks,\n-Stolee\n\n"},{"id":"377021","messageId":"xmqqk1drlxde.fsf@gitster-ct.c.googlers.com","threadId":"51052","inReplyTo":"2dd1552f-72a3-c676-f46c-1393717ffd82@gmail.com","subject":"Re: [PATCH v4 08/14] commit-graph: add --split option to builtin","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-11T19:45:17Z","receivedAt":"2019-06-11T19:45:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> I think the more correct thing to say is that `--split` implies `--append`:\n> As we write a new tip graph, we do not discard any commits from the lower\n> layers, even if we merge them together.\n> ...\n> There is nothing that stops them both from being specified, so this\n> usage describes a strictness that is not enforced. I can change this\n> to be \"[--append] [--split]\" if that is preferred.\n\nGiven that specifying both at the same time does not make much\nsense, I do not think it makes sense more than \"[--append|--split]\".\n\nBetween erroring out when both are given and following the usual\n\"the last one wins\" pattern, I am slightly in favor of the latter.\n\nThanks.\n"},{"id":"377354","messageId":"20190617150207.16849-1-dstolee@microsoft.com","threadId":"51052","inReplyTo":"pull.184.v5.git.gitgitgadget@gmail.com","subject":"[PATCH] commit-graph: normalize commit-graph filenames","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-06-17T15:02:07Z","receivedAt":"2019-06-17T15:02:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When writing commit-graph files, we append path data to an\nobject directory, which may be specified by the user via the\n'--object-dir' option. If the user supplies a trailing slash,\nor some other alternative path format, the resulting path may\nbe usable for writing to the correct location. However, when\nexpiring graph files from the /info/commit-graphs\ndirectory during a write, we need to compare paths with exact\nstring matches.\n\nNormalize the commit-graph filenames to avoid ambiguity. This\ncreates extra allocations, but this is a constant multiple of\nthe number of commit-graph files, which should be a number in\nthe single digits.\n\nTo complete the effectiveness, we need to use the filename\nmethods when iterating over the info/commit-graphs directory\ninstead of constructing the paths manually.\n\nSigned-off-by: Derrick Stolee \n---\n\nJunio,\n\nI noticed the need for this patch while integrating the\nsplit commit-graph series into VFS for Git, since our\nC# code was storing the alternates directory with a trailing\nslash, causing some confusion in the expire logic.\n\nThis could be added to ds/commit-graph-incremental (and I\ncould add it to a future revision, if necessary) or could\nbe a new branch on top of that series.\n\nThanks,\n-Stolee\n\n commit-graph.c | 29 +++++++++++++++++------------\n 1 file changed, 17 insertions(+), 12 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8842f93910..e0f3e8a954 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -43,15 +43,23 @@\n \n char *get_commit_graph_filename(const char *obj_dir)\n {\n-\treturn xstrfmt(\"%s/info/commit-graph\", obj_dir);\n+\tchar *filename = xstrfmt(\"%s/info/commit-graph\", obj_dir);\n+\tchar *normalized = xmalloc(strlen(filename) + 1);\n+\tnormalize_path_copy(normalized, filename);\n+\tfree(filename);\n+\treturn normalized;\n }\n \n static char *get_split_graph_filename(const char *obj_dir,\n \t\t\t\t const char *oid_hex)\n {\n-\treturn xstrfmt(\"%s/info/commit-graphs/graph-%s.graph\",\n-\t\t obj_dir,\n-\t\t oid_hex);\n+\tchar *filename = xstrfmt(\"%s/info/commit-graphs/graph-%s.graph\",\n+\t\t\t\t obj_dir,\n+\t\t\t\t oid_hex);\n+\tchar *normalized = xmalloc(strlen(filename) + 1);\n+\tnormalize_path_copy(normalized, filename);\n+\tfree(filename);\n+\treturn normalized;\n }\n \n static char *get_chain_filename(const char *obj_dir)\n@@ -1680,7 +1688,6 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \tstruct strbuf path = STRBUF_INIT;\n \tDIR *dir;\n \tstruct dirent *de;\n-\tsize_t dirnamelen;\n \ttimestamp_t expire_time = time(NULL);\n \n \tif (ctx->split_opts && ctx->split_opts->expire_time)\n@@ -1701,16 +1708,13 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \t\treturn;\n \t}\n \n-\tstrbuf_addch(&path, '/');\n-\tdirnamelen = path.len;\n \twhile ((de = readdir(dir)) != NULL) {\n \t\tstruct stat st;\n \t\tuint32_t i, found = 0;\n \n-\t\tstrbuf_setlen(&path, dirnamelen);\n-\t\tstrbuf_addstr(&path, de->d_name);\n+\t\tchar *filename = get_split_graph_filename(path.buf, de->d_name);\n \n-\t\tstat(path.buf, &st);\n+\t\tstat(filename, &st);\n \n \t\tif (st.st_mtime > expire_time)\n \t\t\tcontinue;\n@@ -1719,15 +1723,16 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \n \t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++) {\n \t\t\tif (!strcmp(ctx->commit_graph_filenames_after[i],\n-\t\t\t\t path.buf)) {\n+\t\t\t\t filename)) {\n \t\t\t\tfound = 1;\n \t\t\t\tbreak;\n \t\t\t}\n \t\t}\n \n \t\tif (!found)\n-\t\t\tunlink(path.buf);\n+\t\t\tunlink(filename);\n \n+\t\tfree(filename);\n \t}\n }\n \n-- \n2.22.0.430.g4f3aec613b\n\n"},{"id":"377356","messageId":"8251cbca-cbac-ef7f-57fa-a3671fbd6704@gmail.com","threadId":"51052","inReplyTo":"20190617150207.16849-1-dstolee@microsoft.com","subject":"Re: [PATCH] commit-graph: normalize commit-graph filenames","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-06-17T15:07:34Z","receivedAt":"2019-06-17T15:07:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/17/2019 11:02 AM, Derrick Stolee wrote:\n> -\t\tstrbuf_setlen(&path, dirnamelen);\n> -\t\tstrbuf_addstr(&path, de->d_name);\n> +\t\tchar *filename = get_split_graph_filename(path.buf, de->d_name);\n\nPlease ignore this patch for now, as this line is incredibly stupid and\nbreaks the tests that I somehow forgot to run before submitting.\n\nThanks,\n-Stolee\n\n\n"},{"id":"377361","messageId":"20190617180707.11685-1-dstolee@microsoft.com","threadId":"51052","inReplyTo":"20190617150207.16849-1-dstolee@microsoft.com","subject":"[PATCH v2] commit-graph: normalize commit-graph filenames","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-06-17T18:07:07Z","receivedAt":"2019-06-17T18:07:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"When writing commit-graph files, we append path data to an\nobject directory, which may be specified by the user via the\n'--object-dir' option. If the user supplies a trailing slash,\nor some other alternative path format, the resulting path may\nbe usable for writing to the correct location. However, when\nexpiring graph files from the /info/commit-graphs\ndirectory during a write, we need to compare paths with exact\nstring matches.\n\nNormalize the commit-graph filenames to avoid ambiguity. This\ncreates extra allocations, but this is a constant multiple of\nthe number of commit-graph files, which should be a number in\nthe single digits.\n\nFurther normalize the object directory in the context. Due to\na comparison between g->obj_dir and ctx->obj_dir in\nsplit_graph_merge_strategy(), a trailing slash would prevent\nany merging of layers within the same object directory. The\ncheck is there to ensure we do not merge across alternates.\nUpdate the tests to include a case with this trailing slash\nproblem.\n\nSigned-off-by: Derrick Stolee \n---\n\nSorry for the previously-bad patch. This one I believe to\nbe a full correction, and also includes a test to ensure\nwe do not make this mistake again.\n\nThanks,\n-Stolee\n\n commit-graph.c | 36 ++++++++++++++++++++++++++++--------\n 1 file changed, 28 insertions(+), 8 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8842f93910..2836a6fc3d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -43,15 +43,23 @@\n \n char *get_commit_graph_filename(const char *obj_dir)\n {\n-\treturn xstrfmt(\"%s/info/commit-graph\", obj_dir);\n+\tchar *filename = xstrfmt(\"%s/info/commit-graph\", obj_dir);\n+\tchar *normalized = xmalloc(strlen(filename) + 1);\n+\tnormalize_path_copy(normalized, filename);\n+\tfree(filename);\n+\treturn normalized;\n }\n \n static char *get_split_graph_filename(const char *obj_dir,\n \t\t\t\t const char *oid_hex)\n {\n-\treturn xstrfmt(\"%s/info/commit-graphs/graph-%s.graph\",\n-\t\t obj_dir,\n-\t\t oid_hex);\n+\tchar *filename = xstrfmt(\"%s/info/commit-graphs/graph-%s.graph\",\n+\t\t\t\t obj_dir,\n+\t\t\t\t oid_hex);\n+\tchar *normalized = xmalloc(strlen(filename) + 1);\n+\tnormalize_path_copy(normalized, filename);\n+\tfree(filename);\n+\treturn normalized;\n }\n \n static char *get_chain_filename(const char *obj_dir)\n@@ -744,7 +752,7 @@ struct packed_oid_list {\n \n struct write_commit_graph_context {\n \tstruct repository *r;\n-\tconst char *obj_dir;\n+\tchar *obj_dir;\n \tchar *graph_name;\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n@@ -1693,7 +1701,11 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \t}\n \n \tstrbuf_addstr(&path, ctx->obj_dir);\n-\tstrbuf_addstr(&path, \"/info/commit-graphs\");\n+\n+\tif (path.buf[path.len - 1] != '/')\n+\t\tstrbuf_addch(&path, '/');\n+\n+\tstrbuf_addstr(&path, \"info/commit-graphs\");\n \tdir = opendir(path.buf);\n \n \tif (!dir) {\n@@ -1727,7 +1739,6 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \n \t\tif (!found)\n \t\t\tunlink(path.buf);\n-\n \t}\n }\n \n@@ -1739,6 +1750,7 @@ int write_commit_graph(const char *obj_dir,\n {\n \tstruct write_commit_graph_context *ctx;\n \tuint32_t i, count_distinct = 0;\n+\tsize_t len;\n \tint res = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n@@ -1746,7 +1758,14 @@ int write_commit_graph(const char *obj_dir,\n \n \tctx = xcalloc(1, sizeof(struct write_commit_graph_context));\n \tctx->r = the_repository;\n-\tctx->obj_dir = obj_dir;\n+\n+\t/* normalize object dir with no trailing slash */\n+\tctx->obj_dir = xmallocz(strlen(obj_dir) + 1);\n+\tnormalize_path_copy(ctx->obj_dir, obj_dir);\n+\tlen = strlen(ctx->obj_dir);\n+\tif (len && ctx->obj_dir[len - 1] == '/')\n+\t\tctx->obj_dir[len - 1] = 0;\n+\n \tctx->append = flags & COMMIT_GRAPH_APPEND ? 1 : 0;\n \tctx->report_progress = flags & COMMIT_GRAPH_PROGRESS ? 1 : 0;\n \tctx->split = flags & COMMIT_GRAPH_SPLIT ? 1 : 0;\n@@ -1854,6 +1873,7 @@ int write_commit_graph(const char *obj_dir,\n \tfree(ctx->graph_name);\n \tfree(ctx->commits.list);\n \tfree(ctx->oids.list);\n+\tfree(ctx->obj_dir);\n \n \tif (ctx->commit_graph_filenames_after) {\n \t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++) {\n-- \n2.22.0.430.g4f3aec613b\n\n"},{"id":"377430","messageId":"7c3c0df3cdbff9b6537adc2d46363c756aa7bd54.1560881661.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 01/18] commit-graph: document commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-18T18:14:23Z","receivedAt":"2019-06-18T18:14:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd a basic description of commit-graph chains. More details about the\nfeature will be added as we add functionality. This introduction gives a\nhigh-level overview to the goals of the feature and the basic layout of\ncommit-graph chains.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 59 ++++++++++++++++++++++++\n 1 file changed, 59 insertions(+)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex fb53341d5e..1dca3bd8fe 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -127,6 +127,65 @@ Design Details\n helpful for these clones, anyway. The commit-graph will not be read or\n written when shallow commits are present.\n \n+Commit Graphs Chains\n+--------------------\n+\n+Typically, repos grow with near-constant velocity (commits per day). Over time,\n+the number of commits added by a fetch operation is much smaller than the\n+number of commits in the full history. By creating a \"chain\" of commit-graphs,\n+we enable fast writes of new commit data without rewriting the entire commit\n+history -- at least, most of the time.\n+\n+## File Layout\n+\n+A commit-graph chain uses multiple files, and we use a fixed naming convention\n+to organize these files. Each commit-graph file has a name\n+`$OBJDIR/info/commit-graphs/graph-{hash}.graph` where `{hash}` is the hex-\n+valued hash stored in the footer of that file (which is a hash of the file's\n+contents before that hash). For a chain of commit-graph files, a plain-text\n+file at `$OBJDIR/info/commit-graphs/commit-graph-chain` contains the\n+hashes for the files in order from \"lowest\" to \"highest\".\n+\n+For example, if the `commit-graph-chain` file contains the lines\n+\n+```\n+\t{hash0}\n+\t{hash1}\n+\t{hash2}\n+```\n+\n+then the commit-graph chain looks like the following diagram:\n+\n+ +-----------------------+\n+ | graph-{hash2}.graph |\n+ +-----------------------+\n+\t |\n+ +-----------------------+\n+ | |\n+ | graph-{hash1}.graph |\n+ | |\n+ +-----------------------+\n+\t |\n+ +-----------------------+\n+ | |\n+ | |\n+ | |\n+ | graph-{hash0}.graph |\n+ | |\n+ | |\n+ | |\n+ +-----------------------+\n+\n+Let X0 be the number of commits in `graph-{hash0}.graph`, X1 be the number of\n+commits in `graph-{hash1}.graph`, and X2 be the number of commits in\n+`graph-{hash2}.graph`. If a commit appears in position i in `graph-{hash2}.graph`,\n+then we interpret this as being the commit in position (X0 + X1 + i), and that\n+will be used as its \"graph position\". The commits in `graph-{hash2}.graph` use these\n+positions to refer to their parents, which may be in `graph-{hash1}.graph` or\n+`graph-{hash0}.graph`. We can navigate to an arbitrary commit in position j by checking\n+its containment in the intervals [0, X0), [X0, X0 + X1), [X0 + X1, X0 + X1 +\n+X2).\n+\n Related Links\n -------------\n [0] https://bugs.chromium.org/p/git/issues/detail?id=8\n-- \ngitgitgadget\n\n"},{"id":"377431","messageId":"pull.184.v6.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v6 00/18] Commit-graph: Write incremental files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-18T18:14:22Z","receivedAt":"2019-06-18T18:14:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This version is now ready for review.\n\nThe commit-graph is a valuable performance feature for repos with large\ncommit histories, but suffers from the same problem as git repack: it\nrewrites the entire file every time. This can be slow when there are\nmillions of commits, especially after we stopped reading from the\ncommit-graph file during a write in 43d3561 (commit-graph write: don't die\nif the existing graph is corrupt).\n\nInstead, create a \"chain\" of commit-graphs in the\n.git/objects/info/commit-graphs folder with name graph-{hash}.graph. The\nlist of hashes is given by the commit-graph-chain file, and also in a \"base\ngraph chunk\" in the commit-graph format. As we read a chain, we can verify\nthat the hashes match the trailing hash of each commit-graph we read along\nthe way and each hash below a level is expected by that graph file.\n\nWhen writing, we don't always want to add a new level to the stack. This\nwould eventually result in performance degradation, especially when\nsearching for a commit (before we know its graph position). We decide to\nmerge levels of the stack when the new commits we will write is less than\nhalf of the commits in the level above. This can be tweaked by the\n--size-multiple and --max-commits options.\n\nThe performance is necessarily amortized across multiple writes, so I tested\nby writing commit-graphs from the (non-rc) tags in the Linux repo. My test\nincluded 72 tags, and wrote everything reachable from the tag using \n--stdin-commits. Here are the overall perf numbers:\n\nwrite --stdin-commits: 8m 12s\nwrite --stdin-commits --split: 28s\nwrite --split && verify --shallow: 60s\n\nUpdates in V3:\n\n * git commit-graph verify now works on commit-graph chains. We do a simple\n test to check the behavior of a new --shallow option.\n \n \n * When someone writes a flat commit-graph, we now expire the old chain\n according to the expire time.\n \n \n * The \"max commits\" limit is no longer enabled by default, but instead is\n enabled by a --max-commits= option. Ignored if n=0.\n \n \n\nUpdates in V4:\n\nJohannes pointed out some test failures on the Windows platform. We found\nthat the tests were not running on Windows in the gitgitgadget PR builds,\nwhich is now resolved.\n\n * We need to close commit-graphs recursively down the chain. This prevented\n an unlink() from working because of an open handle.\n \n \n * Creating the alternates file used a path-specification that didn't work\n on Windows.\n \n \n * Renaming a file to the same name failed, but is probably related to the\n unlink() error mentioned above.\n \n \n\nUpdates in V5:\n\n * Responding to multiple items of feedback. Thanks Philip, Junio, and\n Ramsay!\n \n \n * Used the test coverage report to find holes in the test coverage. While\n adding tests, I found a bug in octopus merges. The fix is in the rewrite\n of \"deduplicate_commits()\" as \"sort_and_scan_merged_commits()\" and\n covered by the new tests.\n \n \n\nUpdates in V6:\n\n * Rebased onto ds/close-object-store and resolved conflicts around\n close_commit_graph().\n \n \n * Updated path normalization to be resilient to double-slashes and trailing\n slashes.\n \n \n * Added a prepare_alt_odb() call in load_commit_graph_one() for\n cross-alternate graph loads during 'verify' subcommands.\n \n \n\nThanks, -Stolee\n\n[1] \nhttps://github.com/git/git/commit/43d356180556180b4ef6ac232a14498a5bb2b446\ncommit-graph write: don't die if the existing graph is corrupt\n\nDerrick Stolee (18):\n commit-graph: document commit-graph chains\n commit-graph: prepare for commit-graph chains\n commit-graph: rename commit_compare to oid_compare\n commit-graph: load commit-graph chains\n commit-graph: add base graphs chunk\n commit-graph: rearrange chunk count logic\n commit-graph: write commit-graph chains\n commit-graph: add --split option to builtin\n commit-graph: merge commit-graph chains\n commit-graph: allow cross-alternate chains\n commit-graph: expire commit-graph files\n commit-graph: create options for split files\n commit-graph: verify chains with --shallow mode\n commit-graph: clean up chains after flattened write\n commit-graph: test octopus merges with --split\n commit-graph: test --split across alternate without --split\n commit-graph: normalize commit-graph filenames\n commit-graph: test verify across alternates\n\n Documentation/git-commit-graph.txt | 26 +-\n .../technical/commit-graph-format.txt | 11 +-\n Documentation/technical/commit-graph.txt | 195 +++++\n builtin/commit-graph.c | 58 +-\n builtin/commit.c | 2 +-\n builtin/gc.c | 3 +-\n commit-graph.c | 823 ++++++++++++++++--\n commit-graph.h | 25 +-\n t/t5318-commit-graph.sh | 2 +-\n t/t5324-split-commit-graph.sh | 343 ++++++++\n 10 files changed, 1414 insertions(+), 74 deletions(-)\n create mode 100755 t/t5324-split-commit-graph.sh\n\n\nbase-commit: 2d511cfc0bfe1d2b98ba8b272ddd9ba83e84e5f8\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-184%2Fderrickstolee%2Fgraph%2Fincremental-v6\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-184/derrickstolee/graph/incremental-v6\nPull-Request: https://github.com/gitgitgadget/git/pull/184\n\nRange-diff vs v5:\n\n 1: 7a0bfaaa6d = 1: 7c3c0df3cd commit-graph: document commit-graph chains\n 2: ce139d80df ! 2: a06428d749 commit-graph: prepare for commit-graph chains\n @@ -44,11 +44,11 @@\n +\tfree_commit_graph(g);\n +}\n +\n - void close_commit_graph(struct repository *r)\n + void close_commit_graph(struct raw_object_store *o)\n {\n --\tfree_commit_graph(r->objects->commit_graph);\n -+\tclose_commit_graph_one(r->objects->commit_graph);\n - \tr->objects->commit_graph = NULL;\n +-\tfree_commit_graph(o->commit_graph);\n ++\tclose_commit_graph_one(o->commit_graph);\n + \to->commit_graph = NULL;\n }\n \n @@\n 3: 2470d2b548 = 3: 62af96e62b commit-graph: rename commit_compare to oid_compare\n 4: fc3423046b = 4: 0b236c27eb commit-graph: load commit-graph chains\n 5: d14c79f9d5 = 5: 3c1bfcf30b commit-graph: add base graphs chunk\n 6: 5238bbbec3 = 6: 6595617bf1 commit-graph: rearrange chunk count logic\n 7: 02b0359571 ! 7: ee8c09fc5e commit-graph: write commit-graph chains\n @@ -263,7 +263,7 @@\n \t\tUNLEAK(ctx->graph_name);\n \t\terror(_(\"unable to create leading directories of %s\"),\n @@\n - \t\treturn errno;\n + \t\treturn -1;\n \t}\n \n -\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n @@ -340,7 +340,7 @@\n +\t\tctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 2] = new_base_hash;\n +\t}\n +\n - \tclose_commit_graph(ctx->r);\n + \tclose_commit_graph(ctx->r->objects);\n -\tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n +\tfinalize_hashfile(f, file_hash.hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n +\n @@ -481,8 +481,8 @@\n #define COMMIT_GRAPH_PROGRESS (1 << 1)\n +#define COMMIT_GRAPH_SPLIT (1 << 2)\n \n - int write_commit_graph_reachable(const char *obj_dir, unsigned int flags);\n - int write_commit_graph(const char *obj_dir,\n + /*\n + * The write_commit_graph* methods return zero on success\n \n diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n --- a/t/t5318-commit-graph.sh\n 8: a0330ebd2d = 8: dfca87d146 commit-graph: add --split option to builtin\n 9: 28eccfa52b = 9: cae2eb935a commit-graph: merge commit-graph chains\n 10: 2093bab5b1 ! 10: 65b1cc6ae9 commit-graph: allow cross-alternate chains\n @@ -97,6 +97,8 @@\n -\tfor (i = 0; i < count && valid; i++) {\n -\t\tchar *graph_name;\n -\t\tstruct commit_graph *g;\n ++\tprepare_alt_odb(r);\n ++\n +\tfor (i = 0; i < count; i++) {\n +\t\tstruct object_directory *odb;\n \n 11: 554880e3d7 = 11: 2d2977b7df commit-graph: expire commit-graph files\n 12: 66be8b03a8 ! 12: 71f3e9fda9 commit-graph: create options for split files\n @@ -119,24 +119,29 @@\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\n @@\n + \n \tread_replace_refs = 0;\n \n - \tif (opts.reachable)\n +-\tif (opts.reachable)\n -\t\treturn write_commit_graph_reachable(opts.obj_dir, flags);\n -+\t\treturn write_commit_graph_reachable(opts.obj_dir, flags, &split_opts);\n ++\tif (opts.reachable) {\n ++\t\tif (write_commit_graph_reachable(opts.obj_dir, flags, &split_opts))\n ++\t\t\treturn 1;\n ++\t\treturn 0;\n ++\t}\n \n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n @@\n - \tresult = write_commit_graph(opts.obj_dir,\n - \t\t\t\t pack_indexes,\n - \t\t\t\t commit_hex,\n --\t\t\t\t flags);\n -+\t\t\t\t flags,\n -+\t\t\t\t &split_opts);\n + \tif (write_commit_graph(opts.obj_dir,\n + \t\t\t pack_indexes,\n + \t\t\t commit_hex,\n +-\t\t\t flags))\n ++\t\t\t flags,\n ++\t\t\t &split_opts))\n + \t\tresult = 1;\n \n \tUNLEAK(lines);\n - \treturn result;\n \n diff --git a/builtin/commit.c b/builtin/commit.c\n --- a/builtin/commit.c\n @@ -275,13 +280,19 @@\n #define COMMIT_GRAPH_PROGRESS (1 << 1)\n #define COMMIT_GRAPH_SPLIT (1 << 2)\n \n --int write_commit_graph_reachable(const char *obj_dir, unsigned int flags);\n +struct split_commit_graph_opts {\n +\tint size_multiple;\n +\tint max_commits;\n +\ttimestamp_t expire_time;\n +};\n +\n + /*\n + * The write_commit_graph* methods return zero on success\n + * and a negative value on failure. Note that if the repository\n + * is not compatible with the commit-graph feature, then the\n + * methods will return 0 without writing a commit-graph.\n + */\n +-int write_commit_graph_reachable(const char *obj_dir, unsigned int flags);\n +int write_commit_graph_reachable(const char *obj_dir, unsigned int flags,\n +\t\t\t\t const struct split_commit_graph_opts *split_opts);\n int write_commit_graph(const char *obj_dir,\n 13: 9fec4f9a36 ! 13: 5e56632b8b commit-graph: verify chains with --shallow mode\n @@ -194,7 +194,7 @@\n +\n +int verify_commit_graph(struct repository *r, struct commit_graph *g, int flags);\n \n - void close_commit_graph(struct repository *);\n + void close_commit_graph(struct raw_object_store *);\n void free_commit_graph(struct commit_graph *);\n \n diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\n 14: 795ea36ff4 = 14: a4aa473c55 commit-graph: clean up chains after flattened write\n 15: 101792b92d = 15: 4877166d2f commit-graph: test octopus merges with --split\n 16: 84a3ff7c61 = 16: be4b83fdaf commit-graph: test --split across alternate without --split\n -: ---------- > 17: fba62e7a42 commit-graph: normalize commit-graph filenames\n -: ---------- > 18: bad9f13cc2 commit-graph: test verify across alternates\n\n-- \ngitgitgadget\n"},{"id":"377432","messageId":"0b236c27eba4e0109f499d75da9473ccfd193487.1560881661.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 04/18] commit-graph: load commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-18T18:14:25Z","receivedAt":"2019-06-18T18:14:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nPrepare the logic for reading a chain of commit-graphs.\n\nFirst, look for a file at $OBJDIR/info/commit-graph. If it exists,\nthen use that file and stop.\n\nNext, look for the chain file at $OBJDIR/info/commit-graphs/commit-graph-chain.\nIf this file exists, then load the hash values as line-separated values in that\nfile and load $OBJDIR/info/commit-graphs/graph-{hash[i]}.graph for each hash[i]\nin that file. The file is given in order, so the first hash corresponds to the\n\"base\" file and the final hash corresponds to the \"tip\" file.\n\nThis implementation assumes that all of the graph-{hash}.graph files are in\nthe same object directory as the commit-graph-chain file. This will be updated\nin a future change. This change is purposefully simple so we can isolate the\ndifferent concerns.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 112 ++++++++++++++++++++++++++++++++++++++++++++++---\n 1 file changed, 106 insertions(+), 6 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 96b07a674e..f7dfc6aecd 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -45,6 +45,19 @@ char *get_commit_graph_filename(const char *obj_dir)\n \treturn xstrfmt(\"%s/info/commit-graph\", obj_dir);\n }\n \n+static char *get_split_graph_filename(const char *obj_dir,\n+\t\t\t\t const char *oid_hex)\n+{\n+\treturn xstrfmt(\"%s/info/commit-graphs/graph-%s.graph\",\n+\t\t obj_dir,\n+\t\t oid_hex);\n+}\n+\n+static char *get_chain_filename(const char *obj_dir)\n+{\n+\treturn xstrfmt(\"%s/info/commit-graphs/commit-graph-chain\", obj_dir);\n+}\n+\n static uint8_t oid_version(void)\n {\n \treturn 1;\n@@ -286,18 +299,105 @@ static struct commit_graph *load_commit_graph_one(const char *graph_file)\n \treturn load_commit_graph_one_fd_st(fd, &st);\n }\n \n+static struct commit_graph *load_commit_graph_v1(struct repository *r, const char *obj_dir)\n+{\n+\tchar *graph_name = get_commit_graph_filename(obj_dir);\n+\tstruct commit_graph *g = load_commit_graph_one(graph_name);\n+\tfree(graph_name);\n+\n+\treturn g;\n+}\n+\n+static int add_graph_to_chain(struct commit_graph *g,\n+\t\t\t struct commit_graph *chain,\n+\t\t\t struct object_id *oids,\n+\t\t\t int n)\n+{\n+\tstruct commit_graph *cur_g = chain;\n+\n+\twhile (n) {\n+\t\tn--;\n+\t\tcur_g = cur_g->base_graph;\n+\t}\n+\n+\tg->base_graph = chain;\n+\n+\tif (chain)\n+\t\tg->num_commits_in_base = chain->num_commits + chain->num_commits_in_base;\n+\n+\treturn 1;\n+}\n+\n+static struct commit_graph *load_commit_graph_chain(struct repository *r, const char *obj_dir)\n+{\n+\tstruct commit_graph *graph_chain = NULL;\n+\tstruct strbuf line = STRBUF_INIT;\n+\tstruct stat st;\n+\tstruct object_id *oids;\n+\tint i = 0, valid = 1, count;\n+\tchar *chain_name = get_chain_filename(obj_dir);\n+\tFILE *fp;\n+\tint stat_res;\n+\n+\tfp = fopen(chain_name, \"r\");\n+\tstat_res = stat(chain_name, &st);\n+\tfree(chain_name);\n+\n+\tif (!fp ||\n+\t stat_res ||\n+\t st.st_size <= the_hash_algo->hexsz)\n+\t\treturn NULL;\n+\n+\tcount = st.st_size / (the_hash_algo->hexsz + 1);\n+\toids = xcalloc(count, sizeof(struct object_id));\n+\n+\tfor (i = 0; i < count && valid; i++) {\n+\t\tchar *graph_name;\n+\t\tstruct commit_graph *g;\n+\n+\t\tif (strbuf_getline_lf(&line, fp) == EOF)\n+\t\t\tbreak;\n+\n+\t\tif (get_oid_hex(line.buf, &oids[i])) {\n+\t\t\twarning(_(\"invalid commit-graph chain: line '%s' not a hash\"),\n+\t\t\t\tline.buf);\n+\t\t\tvalid = 0;\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\tgraph_name = get_split_graph_filename(obj_dir, line.buf);\n+\t\tg = load_commit_graph_one(graph_name);\n+\t\tfree(graph_name);\n+\n+\t\tif (g && add_graph_to_chain(g, graph_chain, oids, i))\n+\t\t\tgraph_chain = g;\n+\t\telse\n+\t\t\tvalid = 0;\n+\t}\n+\n+\tfree(oids);\n+\tfclose(fp);\n+\n+\treturn graph_chain;\n+}\n+\n+static struct commit_graph *read_commit_graph_one(struct repository *r, const char *obj_dir)\n+{\n+\tstruct commit_graph *g = load_commit_graph_v1(r, obj_dir);\n+\n+\tif (!g)\n+\t\tg = load_commit_graph_chain(r, obj_dir);\n+\n+\treturn g;\n+}\n+\n static void prepare_commit_graph_one(struct repository *r, const char *obj_dir)\n {\n-\tchar *graph_name;\n \n \tif (r->objects->commit_graph)\n \t\treturn;\n \n-\tgraph_name = get_commit_graph_filename(obj_dir);\n-\tr->objects->commit_graph =\n-\t\tload_commit_graph_one(graph_name);\n-\n-\tFREE_AND_NULL(graph_name);\n+\tr->objects->commit_graph = read_commit_graph_one(r, obj_dir);\n }\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"377433","messageId":"3c1bfcf30b6935c5dddabedab579f4e7195a060c.1560881661.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 05/18] commit-graph: add base graphs chunk","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-18T18:14:26Z","receivedAt":"2019-06-18T18:14:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTo quickly verify a commit-graph chain is valid on load, we will\nread from the new \"Base Graphs Chunk\" of each file in the chain.\nThis will prevent accidentally loading incorrect data from manually\nediting the commit-graph-chain file or renaming graph-{hash}.graph\nfiles.\n\nThe commit_graph struct already had an object_id struct \"oid\", but\nit was never initialized or used. Add a line to read the hash from\nthe end of the commit-graph file and into the oid member.\n\nSigned-off-by: Derrick Stolee \n---\n .../technical/commit-graph-format.txt | 11 ++++++++--\n commit-graph.c | 22 +++++++++++++++++++\n commit-graph.h | 1 +\n 3 files changed, 32 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex 16452a0504..a4f17441ae 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -44,8 +44,9 @@ HEADER:\n \n 1-byte number (C) of \"chunks\"\n \n- 1-byte (reserved for later use)\n- Current clients should ignore this value.\n+ 1-byte number (B) of base commit-graphs\n+ We infer the length (H*B) of the Base Graphs chunk\n+ from this value.\n \n CHUNK LOOKUP:\n \n@@ -92,6 +93,12 @@ CHUNK DATA:\n positions for the parents until reaching a value with the most-significant\n bit on. The other bits correspond to the position of the last parent.\n \n+ Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]\n+ This list of H-byte hashes describe a set of B commit-graph files that\n+ form a commit-graph chain. The graph position for the ith commit in this\n+ file's OID Lookup chunk is equal to i plus the number of commits in all\n+ base graphs. If B is non-zero, this chunk must exist.\n+\n TRAILER:\n \n \tH-byte HASH-checksum of all of the above.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f7dfc6aecd..6f7961d071 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -22,6 +22,7 @@\n #define GRAPH_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n #define GRAPH_CHUNKID_DATA 0x43444154 /* \"CDAT\" */\n #define GRAPH_CHUNKID_EXTRAEDGES 0x45444745 /* \"EDGE\" */\n+#define GRAPH_CHUNKID_BASE 0x42415345 /* \"BASE\" */\n \n #define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n \n@@ -262,6 +263,12 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t\t\telse\n \t\t\t\tgraph->chunk_extra_edges = data + chunk_offset;\n \t\t\tbreak;\n+\n+\t\tcase GRAPH_CHUNKID_BASE:\n+\t\t\tif (graph->chunk_base_graphs)\n+\t\t\t\tchunk_repeated = 1;\n+\t\t\telse\n+\t\t\t\tgraph->chunk_base_graphs = data + chunk_offset;\n \t\t}\n \n \t\tif (chunk_repeated) {\n@@ -280,6 +287,8 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t\tlast_chunk_offset = chunk_offset;\n \t}\n \n+\thashcpy(graph->oid.hash, graph->data + graph->data_len - graph->hash_len);\n+\n \tif (verify_commit_graph_lite(graph))\n \t\treturn NULL;\n \n@@ -315,8 +324,21 @@ static int add_graph_to_chain(struct commit_graph *g,\n {\n \tstruct commit_graph *cur_g = chain;\n \n+\tif (n && !g->chunk_base_graphs) {\n+\t\twarning(_(\"commit-graph has no base graphs chunk\"));\n+\t\treturn 0;\n+\t}\n+\n \twhile (n) {\n \t\tn--;\n+\n+\t\tif (!cur_g ||\n+\t\t !oideq(&oids[n], &cur_g->oid) ||\n+\t\t !hasheq(oids[n].hash, g->chunk_base_graphs + g->hash_len * n)) {\n+\t\t\twarning(_(\"commit-graph chain does not match\"));\n+\t\t\treturn 0;\n+\t\t}\n+\n \t\tcur_g = cur_g->base_graph;\n \t}\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex 5819910a5b..6e7d42cf32 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -55,6 +55,7 @@ struct commit_graph {\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_commit_data;\n \tconst unsigned char *chunk_extra_edges;\n+\tconst unsigned char *chunk_base_graphs;\n };\n \n struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st);\n-- \ngitgitgadget\n\n"},{"id":"377434","messageId":"62af96e62bc5a6ab4af6f6cbabbed5edbe586fc0.1560881661.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 03/18] commit-graph: rename commit_compare to oid_compare","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-18T18:14:24Z","receivedAt":"2019-06-18T18:14:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe helper function commit_compare() actually compares object_id\nstructs, not commits. A future change to commit-graph.c will need\nto sort commit structs, so rename this function in advance.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8f5c09363c..96b07a674e 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -770,7 +770,7 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n \t}\n }\n \n-static int commit_compare(const void *_a, const void *_b)\n+static int oid_compare(const void *_a, const void *_b)\n {\n \tconst struct object_id *a = (const struct object_id *)_a;\n \tconst struct object_id *b = (const struct object_id *)_b;\n@@ -1039,7 +1039,7 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n \t\t\t_(\"Counting distinct commits in commit graph\"),\n \t\t\tctx->oids.nr);\n \tdisplay_progress(ctx->progress, 0); /* TODO: Measure QSORT() progress */\n-\tQSORT(ctx->oids.list, ctx->oids.nr, commit_compare);\n+\tQSORT(ctx->oids.list, ctx->oids.nr, oid_compare);\n \n \tfor (i = 1; i < ctx->oids.nr; i++) {\n \t\tdisplay_progress(ctx->progress, i + 1);\n-- \ngitgitgadget\n\n"},{"id":"377435","messageId":"dfca87d146b05a23f1d89a8b2bb34c65be2f4699.1560881661.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 08/18] commit-graph: add --split option to builtin","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-18T18:14:28Z","receivedAt":"2019-06-18T18:14:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd a new \"--split\" option to the 'git commit-graph write' subcommand. This\noption allows the optional behavior of writing a commit-graph chain.\n\nThe current behavior will add a tip commit-graph containing any commits that\nare not in the existing commit-graph or commit-graph chain. Later changes\nwill allow merging the chain and expiring out-dated files.\n\nAdd a new test script (t5324-split-commit-graph.sh) that demonstrates this\nbehavior.\n\nHelped-by: Johannes Schindelin \nSigned-off-by: Derrick Stolee \n---\n builtin/commit-graph.c | 10 ++-\n commit-graph.c | 14 ++--\n t/t5324-split-commit-graph.sh | 122 ++++++++++++++++++++++++++++++++++\n 3 files changed, 138 insertions(+), 8 deletions(-)\n create mode 100755 t/t5324-split-commit-graph.sh\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex d8efa5bab2..ff8bc3c8b9 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -10,7 +10,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph verify [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -25,7 +25,7 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -35,9 +35,9 @@ static struct opts_commit_graph {\n \tint stdin_packs;\n \tint stdin_commits;\n \tint append;\n+\tint split;\n } opts;\n \n-\n static int graph_verify(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n@@ -156,6 +156,8 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"start walk at commits listed by stdin\")),\n \t\tOPT_BOOL(0, \"append\", &opts.append,\n \t\t\tN_(\"include all commits already in the commit-graph file\")),\n+\t\tOPT_BOOL(0, \"split\", &opts.split,\n+\t\t\tN_(\"allow writing an incremental commit-graph file\")),\n \t\tOPT_END(),\n \t};\n \n@@ -169,6 +171,8 @@ static int graph_write(int argc, const char **argv)\n \t\topts.obj_dir = get_object_directory();\n \tif (opts.append)\n \t\tflags |= COMMIT_GRAPH_APPEND;\n+\tif (opts.split)\n+\t\tflags |= COMMIT_GRAPH_SPLIT;\n \n \tread_replace_refs = 0;\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex f0698b0599..1224309e5f 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1472,12 +1472,16 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t}\n \n \t\tif (ctx->base_graph_name) {\n-\t\t\tresult = rename(ctx->base_graph_name,\n-\t\t\t\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2]);\n+\t\t\tconst char *dest = ctx->commit_graph_filenames_after[\n+\t\t\t\t\t\tctx->num_commit_graphs_after - 2];\n \n-\t\t\tif (result) {\n-\t\t\t\terror(_(\"failed to rename base commit-graph file\"));\n-\t\t\t\treturn -1;\n+\t\t\tif (strcmp(ctx->base_graph_name, dest)) {\n+\t\t\t\tresult = rename(ctx->base_graph_name, dest);\n+\n+\t\t\t\tif (result) {\n+\t\t\t\t\terror(_(\"failed to rename base commit-graph file\"));\n+\t\t\t\t\treturn -1;\n+\t\t\t\t}\n \t\t\t}\n \t\t} else {\n \t\t\tchar *graph_name = get_commit_graph_filename(ctx->obj_dir);\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nnew file mode 100755\nindex 0000000000..ccd24bd22b\n--- /dev/null\n+++ b/t/t5324-split-commit-graph.sh\n@@ -0,0 +1,122 @@\n+#!/bin/sh\n+\n+test_description='split commit graph'\n+. ./test-lib.sh\n+\n+GIT_TEST_COMMIT_GRAPH=0\n+\n+test_expect_success 'setup repo' '\n+\tgit init &&\n+\tgit config core.commitGraph true &&\n+\tinfodir=\".git/objects/info\" &&\n+\tgraphdir=\"$infodir/commit-graphs\" &&\n+\ttest_oid_init\n+'\n+\n+graph_read_expect() {\n+\tNUM_BASE=0\n+\tif test ! -z $2\n+\tthen\n+\t\tNUM_BASE=$2\n+\tfi\n+\tcat >expect <<- EOF\n+\theader: 43475048 1 1 3 $NUM_BASE\n+\tnum_commits: $1\n+\tchunks: oid_fanout oid_lookup commit_metadata\n+\tEOF\n+\tgit commit-graph read >output &&\n+\ttest_cmp expect output\n+}\n+\n+test_expect_success 'create commits and write commit-graph' '\n+\tfor i in $(test_seq 3)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i || return 1\n+\tdone &&\n+\tgit commit-graph write --reachable &&\n+\ttest_path_is_file $infodir/commit-graph &&\n+\tgraph_read_expect 3\n+'\n+\n+graph_git_two_modes() {\n+\tgit -c core.commitGraph=true $1 >output\n+\tgit -c core.commitGraph=false $1 >expect\n+\ttest_cmp expect output\n+}\n+\n+graph_git_behavior() {\n+\tMSG=$1\n+\tBRANCH=$2\n+\tCOMPARE=$3\n+\ttest_expect_success \"check normal git operations: $MSG\" '\n+\t\tgraph_git_two_modes \"log --oneline $BRANCH\" &&\n+\t\tgraph_git_two_modes \"log --topo-order $BRANCH\" &&\n+\t\tgraph_git_two_modes \"log --graph $COMPARE..$BRANCH\" &&\n+\t\tgraph_git_two_modes \"branch -vv\" &&\n+\t\tgraph_git_two_modes \"merge-base -a $BRANCH $COMPARE\"\n+\t'\n+}\n+\n+graph_git_behavior 'graph exists' commits/3 commits/1\n+\n+verify_chain_files_exist() {\n+\tfor hash in $(cat $1/commit-graph-chain)\n+\tdo\n+\t\ttest_path_is_file $1/graph-$hash.graph || return 1\n+\tdone\n+}\n+\n+test_expect_success 'add more commits, and write a new base graph' '\n+\tgit reset --hard commits/1 &&\n+\tfor i in $(test_seq 4 5)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i || return 1\n+\tdone &&\n+\tgit reset --hard commits/2 &&\n+\tfor i in $(test_seq 6 10)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i || return 1\n+\tdone &&\n+\tgit reset --hard commits/2 &&\n+\tgit merge commits/4 &&\n+\tgit branch merge/1 &&\n+\tgit reset --hard commits/4 &&\n+\tgit merge commits/6 &&\n+\tgit branch merge/2 &&\n+\tgit commit-graph write --reachable &&\n+\tgraph_read_expect 12\n+'\n+\n+test_expect_success 'add three more commits, write a tip graph' '\n+\tgit reset --hard commits/3 &&\n+\tgit merge merge/1 &&\n+\tgit merge commits/5 &&\n+\tgit merge merge/2 &&\n+\tgit branch merge/3 &&\n+\tgit commit-graph write --reachable --split &&\n+\ttest_path_is_missing $infodir/commit-graph &&\n+\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\tls $graphdir/graph-*.graph >graph-files &&\n+\ttest_line_count = 2 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n+graph_git_behavior 'split commit-graph: merge 3 vs 2' merge/3 merge/2\n+\n+test_expect_success 'add one commit, write a tip graph' '\n+\ttest_commit 11 &&\n+\tgit branch commits/11 &&\n+\tgit commit-graph write --reachable --split &&\n+\ttest_path_is_missing $infodir/commit-graph &&\n+\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\tls $graphdir/graph-*.graph >graph-files &&\n+\ttest_line_count = 3 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n+graph_git_behavior 'three-layer commit-graph: commit 11 vs 6' commits/11 commits/6\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"377436","messageId":"cae2eb935a9b5005ed7ee63379f0ca95c2aae60e.1560881661.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 09/18] commit-graph: merge commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-18T18:14:29Z","receivedAt":"2019-06-18T18:14:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen searching for a commit in a commit-graph chain of G graphs with N\ncommits, the search takes O(G log N) time. If we always add a new tip\ngraph with every write, the linear G term will start to dominate and\nslow the lookup process.\n\nTo keep lookups fast, but also keep most incremental writes fast, create\na strategy for merging levels of the commit-graph chain. The strategy is\ndetailed in the commit-graph design document, but is summarized by these\ntwo conditions:\n\n 1. If the number of commits we are adding is more than half the number\n of commits in the graph below, then merge with that graph.\n\n 2. If we are writing more than 64,000 commits into a single graph,\n then merge with all lower graphs.\n\nThe numeric values in the conditions above are currently constant, but\ncan become config options in a future update.\n\nAs we merge levels of the commit-graph chain, check that the commits\nstill exist in the repository. A garbage-collection operation may have\nremoved those commits from the object store and we do not want to\npersist them in the commit-graph chain. This is a non-issue if the\n'git gc' process wrote a new, single-level commit-graph file.\n\nAfter we merge levels, the old graph-{hash}.graph files are no longer\nreferenced by the commit-graph-chain file. We will expire these files in\na future change.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 80 ++++++++++\n commit-graph.c | 180 ++++++++++++++++++-----\n t/t5324-split-commit-graph.sh | 13 ++\n 3 files changed, 240 insertions(+), 33 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex 1dca3bd8fe..d9c6253b0a 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -186,6 +186,86 @@ positions to refer to their parents, which may be in `graph-{hash1}.graph` or\n its containment in the intervals [0, X0), [X0, X0 + X1), [X0 + X1, X0 + X1 +\n X2).\n \n+Each commit-graph file (except the base, `graph-{hash0}.graph`) contains data\n+specifying the hashes of all files in the lower layers. In the above example,\n+`graph-{hash1}.graph` contains `{hash0}` while `graph-{hash2}.graph` contains\n+`{hash0}` and `{hash1}`.\n+\n+## Merging commit-graph files\n+\n+If we only added a new commit-graph file on every write, we would run into a\n+linear search problem through many commit-graph files. Instead, we use a merge\n+strategy to decide when the stack should collapse some number of levels.\n+\n+The diagram below shows such a collapse. As a set of new commits are added, it\n+is determined by the merge strategy that the files should collapse to\n+`graph-{hash1}`. Thus, the new commits, the commits in `graph-{hash2}` and\n+the commits in `graph-{hash1}` should be combined into a new `graph-{hash3}`\n+file.\n+\n+\t\t\t +---------------------+\n+\t\t\t | |\n+\t\t\t | (new commits) |\n+\t\t\t | |\n+\t\t\t +---------------------+\n+\t\t\t | |\n+ +-----------------------+ +---------------------+\n+ | graph-{hash2} |->| |\n+ +-----------------------+ +---------------------+\n+\t | | |\n+ +-----------------------+ +---------------------+\n+ | | | |\n+ | graph-{hash1} |->| |\n+ | | | |\n+ +-----------------------+ +---------------------+\n+\t | tmp_graphXXX\n+ +-----------------------+\n+ | |\n+ | |\n+ | |\n+ | graph-{hash0} |\n+ | |\n+ | |\n+ | |\n+ +-----------------------+\n+\n+During this process, the commits to write are combined, sorted and we write the\n+contents to a temporary file, all while holding a `commit-graph-chain.lock`\n+lock-file. When the file is flushed, we rename it to `graph-{hash3}`\n+according to the computed `{hash3}`. Finally, we write the new chain data to\n+`commit-graph-chain.lock`:\n+\n+```\n+\t{hash3}\n+\t{hash0}\n+```\n+\n+We then close the lock-file.\n+\n+## Merge Strategy\n+\n+When writing a set of commits that do not exist in the commit-graph stack of\n+height N, we default to creating a new file at level N + 1. We then decide to\n+merge with the Nth level if one of two conditions hold:\n+\n+ 1. The expected file size for level N + 1 is at least half the file size for\n+ level N.\n+\n+ 2. Level N + 1 contains more than 64,0000 commits.\n+\n+This decision cascades down the levels: when we merge a level we create a new\n+set of commits that then compares to the next level.\n+\n+The first condition bounds the number of levels to be logarithmic in the total\n+number of commits. The second condition bounds the total number of commits in\n+a `graph-{hashN}` file and not in the `commit-graph` file, preventing\n+significant performance issues when the stack merges and another process only\n+partially reads the previous stack.\n+\n+The merge strategy values (2 for the size multiple, 64,000 for the maximum\n+number of commits) could be extracted into config settings for full\n+flexibility.\n+\n Related Links\n -------------\n [0] https://bugs.chromium.org/p/git/issues/detail?id=8\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 1224309e5f..fb3100921c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1298,36 +1298,6 @@ static int write_graph_chunk_base(struct hashfile *f,\n \treturn 0;\n }\n \n-static void init_commit_graph_chain(struct write_commit_graph_context *ctx)\n-{\n-\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n-\tuint32_t i;\n-\n-\tctx->new_base_graph = g;\n-\tctx->base_graph_name = xstrdup(g->filename);\n-\tctx->new_num_commits_in_base = g->num_commits + g->num_commits_in_base;\n-\n-\tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n-\n-\tALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);\n-\tALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);\n-\n-\tfor (i = 0; i < ctx->num_commit_graphs_before - 1; i++)\n-\t\tctx->commit_graph_filenames_after[i] = xstrdup(ctx->commit_graph_filenames_before[i]);\n-\n-\tif (ctx->num_commit_graphs_before)\n-\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_before - 1] =\n-\t\t\tget_split_graph_filename(ctx->obj_dir, oid_to_hex(&g->oid));\n-\n-\ti = ctx->num_commit_graphs_before - 1;\n-\n-\twhile (g) {\n-\t\tctx->commit_graph_hash_after[i] = xstrdup(oid_to_hex(&g->oid));\n-\t\ti--;\n-\t\tg = g->base_graph;\n-\t}\n-}\n-\n static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n@@ -1509,6 +1479,145 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \treturn 0;\n }\n \n+static int split_strategy_max_commits = 64000;\n+static float split_strategy_size_mult = 2.0f;\n+\n+static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n+{\n+\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\tuint32_t num_commits = ctx->commits.nr;\n+\tuint32_t i;\n+\n+\tg = ctx->r->objects->commit_graph;\n+\tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n+\n+\twhile (g && (g->num_commits <= split_strategy_size_mult * num_commits ||\n+\t\t num_commits > split_strategy_max_commits)) {\n+\t\tnum_commits += g->num_commits;\n+\t\tg = g->base_graph;\n+\n+\t\tctx->num_commit_graphs_after--;\n+\t}\n+\n+\tctx->new_base_graph = g;\n+\n+\tALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);\n+\tALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);\n+\n+\tfor (i = 0; i < ctx->num_commit_graphs_after &&\n+\t\t i < ctx->num_commit_graphs_before; i++)\n+\t\tctx->commit_graph_filenames_after[i] = xstrdup(ctx->commit_graph_filenames_before[i]);\n+\n+\ti = ctx->num_commit_graphs_before - 1;\n+\tg = ctx->r->objects->commit_graph;\n+\n+\twhile (g) {\n+\t\tif (i < ctx->num_commit_graphs_after)\n+\t\t\tctx->commit_graph_hash_after[i] = xstrdup(oid_to_hex(&g->oid));\n+\n+\t\ti--;\n+\t\tg = g->base_graph;\n+\t}\n+}\n+\n+static void merge_commit_graph(struct write_commit_graph_context *ctx,\n+\t\t\t struct commit_graph *g)\n+{\n+\tuint32_t i;\n+\tuint32_t offset = g->num_commits_in_base;\n+\n+\tALLOC_GROW(ctx->commits.list, ctx->commits.nr + g->num_commits, ctx->commits.alloc);\n+\n+\tfor (i = 0; i < g->num_commits; i++) {\n+\t\tstruct object_id oid;\n+\t\tstruct commit *result;\n+\n+\t\tdisplay_progress(ctx->progress, i + 1);\n+\n+\t\tload_oid_from_graph(g, i + offset, &oid);\n+\n+\t\t/* only add commits if they still exist in the repo */\n+\t\tresult = lookup_commit_reference_gently(ctx->r, &oid, 1);\n+\n+\t\tif (result) {\n+\t\t\tctx->commits.list[ctx->commits.nr] = result;\n+\t\t\tctx->commits.nr++;\n+\t\t}\n+\t}\n+}\n+\n+static int commit_compare(const void *_a, const void *_b)\n+{\n+\tconst struct commit *a = *(const struct commit **)_a;\n+\tconst struct commit *b = *(const struct commit **)_b;\n+\treturn oidcmp(&a->object.oid, &b->object.oid);\n+}\n+\n+static void sort_and_scan_merged_commits(struct write_commit_graph_context *ctx)\n+{\n+\tuint32_t i, num_parents;\n+\tstruct commit_list *parent;\n+\n+\tif (ctx->report_progress)\n+\t\tctx->progress = start_delayed_progress(\n+\t\t\t\t\t_(\"Scanning merged commits\"),\n+\t\t\t\t\tctx->commits.nr);\n+\n+\tQSORT(ctx->commits.list, ctx->commits.nr, commit_compare);\n+\n+\tctx->num_extra_edges = 0;\n+\tfor (i = 0; i < ctx->commits.nr; i++) {\n+\t\tdisplay_progress(ctx->progress, i);\n+\n+\t\tif (i && oideq(&ctx->commits.list[i - 1]->object.oid,\n+\t\t\t &ctx->commits.list[i]->object.oid)) {\n+\t\t\tdie(_(\"unexpected duplicate commit id %s\"),\n+\t\t\t oid_to_hex(&ctx->commits.list[i]->object.oid));\n+\t\t} else {\n+\t\t\tnum_parents = 0;\n+\t\t\tfor (parent = ctx->commits.list[i]->parents; parent; parent = parent->next)\n+\t\t\t\tnum_parents++;\n+\n+\t\t\tif (num_parents > 2)\n+\t\t\t\tctx->num_extra_edges += num_parents - 2;\n+\t\t}\n+\t}\n+\n+\tstop_progress(&ctx->progress);\n+}\n+\n+static void merge_commit_graphs(struct write_commit_graph_context *ctx)\n+{\n+\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\tuint32_t current_graph_number = ctx->num_commit_graphs_before;\n+\tstruct strbuf progress_title = STRBUF_INIT;\n+\n+\twhile (g && current_graph_number >= ctx->num_commit_graphs_after) {\n+\t\tcurrent_graph_number--;\n+\n+\t\tif (ctx->report_progress) {\n+\t\t\tstrbuf_addstr(&progress_title, _(\"Merging commit-graph\"));\n+\t\t\tctx->progress = start_delayed_progress(progress_title.buf, 0);\n+\t\t}\n+\n+\t\tmerge_commit_graph(ctx, g);\n+\t\tstop_progress(&ctx->progress);\n+\t\tstrbuf_release(&progress_title);\n+\n+\t\tg = g->base_graph;\n+\t}\n+\n+\tif (g) {\n+\t\tctx->new_base_graph = g;\n+\t\tctx->new_num_commits_in_base = g->num_commits + g->num_commits_in_base;\n+\t}\n+\n+\tif (ctx->new_base_graph)\n+\t\tctx->base_graph_name = xstrdup(ctx->new_base_graph->filename);\n+\n+\tsort_and_scan_merged_commits(ctx);\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -1554,6 +1663,9 @@ int write_commit_graph(const char *obj_dir,\n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n \n+\tif (ctx->split && ctx->oids.alloc > split_strategy_max_commits)\n+\t\tctx->oids.alloc = split_strategy_max_commits;\n+\n \tif (ctx->append) {\n \t\tprepare_commit_graph_one(ctx->r, ctx->obj_dir);\n \t\tif (ctx->r->objects->commit_graph)\n@@ -1607,9 +1719,11 @@ int write_commit_graph(const char *obj_dir,\n \tif (!ctx->commits.nr)\n \t\tgoto cleanup;\n \n-\tif (ctx->split)\n-\t\tinit_commit_graph_chain(ctx);\n-\telse\n+\tif (ctx->split) {\n+\t\tsplit_graph_merge_strategy(ctx);\n+\n+\t\tmerge_commit_graphs(ctx);\n+\t} else\n \t\tctx->num_commit_graphs_after = 1;\n \n \tcompute_generation_numbers(ctx);\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex ccd24bd22b..5cb5663a30 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -119,4 +119,17 @@ test_expect_success 'add one commit, write a tip graph' '\n \n graph_git_behavior 'three-layer commit-graph: commit 11 vs 6' commits/11 commits/6\n \n+test_expect_success 'add one commit, write a merged graph' '\n+\ttest_commit 12 &&\n+\tgit branch commits/12 &&\n+\tgit commit-graph write --reachable --split &&\n+\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\tls $graphdir/graph-*.graph >graph-files &&\n+\ttest_line_count = 4 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n+graph_git_behavior 'merged commit-graph: commit 12 vs 6' commits/12 commits/6\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"377437","messageId":"6595617bf19f7172cb9983f52a8faff4e463f795.1560881661.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 06/18] commit-graph: rearrange chunk count logic","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-18T18:14:27Z","receivedAt":"2019-06-18T18:14:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe number of chunks in a commit-graph file can change depending on\nwhether we need the Extra Edges Chunk. We are going to add more optional\nchunks, and it will be helpful to rearrange this logic around the chunk\ncount before doing so.\n\nSpecifically, we need to finalize the number of chunks before writing\nthe commit-graph header. Further, we also need to fill out the chunk\nlookup table dynamically and using \"num_chunks\" as we add optional\nchunks is useful for adding optional chunks in the future.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 35 +++++++++++++++++++++--------------\n 1 file changed, 21 insertions(+), 14 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 6f7961d071..f2163e109f 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1213,7 +1213,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tuint64_t chunk_offsets[5];\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n-\tint num_chunks = ctx->num_extra_edges ? 4 : 3;\n+\tint num_chunks = 3;\n \n \tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n \tif (safe_create_leading_directories(ctx->graph_name)) {\n@@ -1226,27 +1226,34 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n \tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n \n-\thashwrite_be32(f, GRAPH_SIGNATURE);\n-\n-\thashwrite_u8(f, GRAPH_VERSION);\n-\thashwrite_u8(f, oid_version());\n-\thashwrite_u8(f, num_chunks);\n-\thashwrite_u8(f, 0); /* unused padding byte */\n-\n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n \tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n-\tif (ctx->num_extra_edges)\n-\t\tchunk_ids[3] = GRAPH_CHUNKID_EXTRAEDGES;\n-\telse\n-\t\tchunk_ids[3] = 0;\n-\tchunk_ids[4] = 0;\n+\tif (ctx->num_extra_edges) {\n+\t\tchunk_ids[num_chunks] = GRAPH_CHUNKID_EXTRAEDGES;\n+\t\tnum_chunks++;\n+\t}\n+\n+\tchunk_ids[num_chunks] = 0;\n \n \tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n \tchunk_offsets[2] = chunk_offsets[1] + hashsz * ctx->commits.nr;\n \tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * ctx->commits.nr;\n-\tchunk_offsets[4] = chunk_offsets[3] + 4 * ctx->num_extra_edges;\n+\n+\tnum_chunks = 3;\n+\tif (ctx->num_extra_edges) {\n+\t\tchunk_offsets[num_chunks + 1] = chunk_offsets[num_chunks] +\n+\t\t\t\t\t\t4 * ctx->num_extra_edges;\n+\t\tnum_chunks++;\n+\t}\n+\n+\thashwrite_be32(f, GRAPH_SIGNATURE);\n+\n+\thashwrite_u8(f, GRAPH_VERSION);\n+\thashwrite_u8(f, oid_version());\n+\thashwrite_u8(f, num_chunks);\n+\thashwrite_u8(f, 0);\n \n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tuint32_t chunk_write[3];\n-- \ngitgitgadget\n\n"},{"id":"377438","messageId":"ee8c09fc5e6ce4b790e006e79458309b1e1e0acd.1560881661.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 07/18] commit-graph: write commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-18T18:14:27Z","receivedAt":"2019-06-18T18:14:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nExtend write_commit_graph() to write a commit-graph chain when given the\nCOMMIT_GRAPH_SPLIT flag.\n\nThis implementation is purposefully simplistic in how it creates a new\nchain. The commits not already in the chain are added to a new tip\ncommit-graph file.\n\nMuch of the logic around writing a graph-{hash}.graph file and updating\nthe commit-graph-chain file is the same as the commit-graph file case.\nHowever, there are several places where we need to do some extra logic\nin the split case.\n\nTrack the list of graph filenames before and after the planned write.\nThis will be more important when we start merging graph files, but it\nalso allows us to upgrade our commit-graph file to the appropriate\ngraph-{hash}.graph file when we upgrade to a chain of commit-graphs.\n\nNote that we use the eighth byte of the commit-graph header to store the\nnumber of base graph files. This determines the length of the base\ngraphs chunk.\n\nA subtle change of behavior with the new logic is that we do not write a\ncommit-graph if we our commit list is empty. This extends to the typical\ncase, which is reflected in t5318-commit-graph.sh.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 286 ++++++++++++++++++++++++++++++++++++++--\n commit-graph.h | 2 +\n t/t5318-commit-graph.sh | 2 +-\n 3 files changed, 278 insertions(+), 12 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f2163e109f..f0698b0599 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -300,12 +300,18 @@ static struct commit_graph *load_commit_graph_one(const char *graph_file)\n \n \tstruct stat st;\n \tint fd;\n+\tstruct commit_graph *g;\n \tint open_ok = open_commit_graph(graph_file, &fd, &st);\n \n \tif (!open_ok)\n \t\treturn NULL;\n \n-\treturn load_commit_graph_one_fd_st(fd, &st);\n+\tg = load_commit_graph_one_fd_st(fd, &st);\n+\n+\tif (g)\n+\t\tg->filename = xstrdup(graph_file);\n+\n+\treturn g;\n }\n \n static struct commit_graph *load_commit_graph_v1(struct repository *r, const char *obj_dir)\n@@ -730,8 +736,19 @@ struct write_commit_graph_context {\n \tstruct progress *progress;\n \tint progress_done;\n \tuint64_t progress_cnt;\n+\n+\tchar *base_graph_name;\n+\tint num_commit_graphs_before;\n+\tint num_commit_graphs_after;\n+\tchar **commit_graph_filenames_before;\n+\tchar **commit_graph_filenames_after;\n+\tchar **commit_graph_hash_after;\n+\tuint32_t new_num_commits_in_base;\n+\tstruct commit_graph *new_base_graph;\n+\n \tunsigned append:1,\n-\t\t report_progress:1;\n+\t\t report_progress:1,\n+\t\t split:1;\n };\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n@@ -801,6 +818,16 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t commit_to_sha1);\n \n+\t\t\tif (edge_value >= 0)\n+\t\t\t\tedge_value += ctx->new_num_commits_in_base;\n+\t\t\telse {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (find_commit_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n+\t\t\t\t\tedge_value = pos;\n+\t\t\t}\n+\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n \t\t\t\t oid_to_hex(&parent->item->object.oid),\n@@ -821,6 +848,17 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\t\t\t ctx->commits.list,\n \t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t commit_to_sha1);\n+\n+\t\t\tif (edge_value >= 0)\n+\t\t\t\tedge_value += ctx->new_num_commits_in_base;\n+\t\t\telse {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (find_commit_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n+\t\t\t\t\tedge_value = pos;\n+\t\t\t}\n+\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n \t\t\t\t oid_to_hex(&parent->item->object.oid),\n@@ -878,6 +916,16 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n \t\t\t\t\t\t ctx->commits.nr,\n \t\t\t\t\t\t commit_to_sha1);\n \n+\t\t\tif (edge_value >= 0)\n+\t\t\t\tedge_value += ctx->new_num_commits_in_base;\n+\t\t\telse {\n+\t\t\t\tuint32_t pos;\n+\t\t\t\tif (find_commit_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n+\t\t\t\t\tedge_value = pos;\n+\t\t\t}\n+\n \t\t\tif (edge_value < 0)\n \t\t\t\tBUG(\"missing parent %s for commit %s\",\n \t\t\t\t oid_to_hex(&parent->item->object.oid),\n@@ -969,7 +1017,13 @@ static void close_reachable(struct write_commit_graph_context *ctx)\n \t\tdisplay_progress(ctx->progress, i + 1);\n \t\tcommit = lookup_commit(ctx->r, &ctx->oids.list[i]);\n \n-\t\tif (commit && !parse_commit_no_graph(commit))\n+\t\tif (!commit)\n+\t\t\tcontinue;\n+\t\tif (ctx->split) {\n+\t\t\tif (!parse_commit(commit) &&\n+\t\t\t commit->graph_pos == COMMIT_NOT_FROM_GRAPH)\n+\t\t\t\tadd_missing_parents(ctx, commit);\n+\t\t} else if (!parse_commit_no_graph(commit))\n \t\t\tadd_missing_parents(ctx, commit);\n \t}\n \tstop_progress(&ctx->progress);\n@@ -1165,8 +1219,16 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n \n \tfor (i = 1; i < ctx->oids.nr; i++) {\n \t\tdisplay_progress(ctx->progress, i + 1);\n-\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n+\t\tif (!oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i])) {\n+\t\t\tif (ctx->split) {\n+\t\t\t\tstruct commit *c = lookup_commit(ctx->r, &ctx->oids.list[i]);\n+\n+\t\t\t\tif (!c || c->graph_pos != COMMIT_NOT_FROM_GRAPH)\n+\t\t\t\t\tcontinue;\n+\t\t\t}\n+\n \t\t\tcount_distinct++;\n+\t\t}\n \t}\n \tstop_progress(&ctx->progress);\n \n@@ -1189,7 +1251,13 @@ static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n \t\tif (i > 0 && oideq(&ctx->oids.list[i - 1], &ctx->oids.list[i]))\n \t\t\tcontinue;\n \n+\t\tALLOC_GROW(ctx->commits.list, ctx->commits.nr + 1, ctx->commits.alloc);\n \t\tctx->commits.list[ctx->commits.nr] = lookup_commit(ctx->r, &ctx->oids.list[i]);\n+\n+\t\tif (ctx->split &&\n+\t\t ctx->commits.list[ctx->commits.nr]->graph_pos != COMMIT_NOT_FROM_GRAPH)\n+\t\t\tcontinue;\n+\n \t\tparse_commit_no_graph(ctx->commits.list[ctx->commits.nr]);\n \n \t\tfor (parent = ctx->commits.list[ctx->commits.nr]->parents;\n@@ -1204,18 +1272,86 @@ static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n+static int write_graph_chunk_base_1(struct hashfile *f,\n+\t\t\t\t struct commit_graph *g)\n+{\n+\tint num = 0;\n+\n+\tif (!g)\n+\t\treturn 0;\n+\n+\tnum = write_graph_chunk_base_1(f, g->base_graph);\n+\thashwrite(f, g->oid.hash, the_hash_algo->rawsz);\n+\treturn num + 1;\n+}\n+\n+static int write_graph_chunk_base(struct hashfile *f,\n+\t\t\t\t struct write_commit_graph_context *ctx)\n+{\n+\tint num = write_graph_chunk_base_1(f, ctx->new_base_graph);\n+\n+\tif (num != ctx->num_commit_graphs_after - 1) {\n+\t\terror(_(\"failed to write correct number of base graph ids\"));\n+\t\treturn -1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void init_commit_graph_chain(struct write_commit_graph_context *ctx)\n+{\n+\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n+\tuint32_t i;\n+\n+\tctx->new_base_graph = g;\n+\tctx->base_graph_name = xstrdup(g->filename);\n+\tctx->new_num_commits_in_base = g->num_commits + g->num_commits_in_base;\n+\n+\tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n+\n+\tALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);\n+\tALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);\n+\n+\tfor (i = 0; i < ctx->num_commit_graphs_before - 1; i++)\n+\t\tctx->commit_graph_filenames_after[i] = xstrdup(ctx->commit_graph_filenames_before[i]);\n+\n+\tif (ctx->num_commit_graphs_before)\n+\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_before - 1] =\n+\t\t\tget_split_graph_filename(ctx->obj_dir, oid_to_hex(&g->oid));\n+\n+\ti = ctx->num_commit_graphs_before - 1;\n+\n+\twhile (g) {\n+\t\tctx->commit_graph_hash_after[i] = xstrdup(oid_to_hex(&g->oid));\n+\t\ti--;\n+\t\tg = g->base_graph;\n+\t}\n+}\n+\n static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n+\tint fd;\n \tstruct hashfile *f;\n \tstruct lock_file lk = LOCK_INIT;\n-\tuint32_t chunk_ids[5];\n-\tuint64_t chunk_offsets[5];\n+\tuint32_t chunk_ids[6];\n+\tuint64_t chunk_offsets[6];\n \tconst unsigned hashsz = the_hash_algo->rawsz;\n \tstruct strbuf progress_title = STRBUF_INIT;\n \tint num_chunks = 3;\n+\tstruct object_id file_hash;\n+\n+\tif (ctx->split) {\n+\t\tstruct strbuf tmp_file = STRBUF_INIT;\n+\n+\t\tstrbuf_addf(&tmp_file,\n+\t\t\t \"%s/info/commit-graphs/tmp_graph_XXXXXX\",\n+\t\t\t ctx->obj_dir);\n+\t\tctx->graph_name = strbuf_detach(&tmp_file, NULL);\n+\t} else {\n+\t\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n+\t}\n \n-\tctx->graph_name = get_commit_graph_filename(ctx->obj_dir);\n \tif (safe_create_leading_directories(ctx->graph_name)) {\n \t\tUNLEAK(ctx->graph_name);\n \t\terror(_(\"unable to create leading directories of %s\"),\n@@ -1223,8 +1359,23 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\treturn -1;\n \t}\n \n-\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n-\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n+\tif (ctx->split) {\n+\t\tchar *lock_name = get_chain_filename(ctx->obj_dir);\n+\n+\t\thold_lock_file_for_update(&lk, lock_name, LOCK_DIE_ON_ERROR);\n+\n+\t\tfd = git_mkstemp_mode(ctx->graph_name, 0444);\n+\t\tif (fd < 0) {\n+\t\t\terror(_(\"unable to create '%s'\"), ctx->graph_name);\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tf = hashfd(fd, ctx->graph_name);\n+\t} else {\n+\t\thold_lock_file_for_update(&lk, ctx->graph_name, LOCK_DIE_ON_ERROR);\n+\t\tfd = lk.tempfile->fd;\n+\t\tf = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);\n+\t}\n \n \tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n \tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n@@ -1233,6 +1384,10 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\tchunk_ids[num_chunks] = GRAPH_CHUNKID_EXTRAEDGES;\n \t\tnum_chunks++;\n \t}\n+\tif (ctx->num_commit_graphs_after > 1) {\n+\t\tchunk_ids[num_chunks] = GRAPH_CHUNKID_BASE;\n+\t\tnum_chunks++;\n+\t}\n \n \tchunk_ids[num_chunks] = 0;\n \n@@ -1247,13 +1402,18 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\t\t\t\t4 * ctx->num_extra_edges;\n \t\tnum_chunks++;\n \t}\n+\tif (ctx->num_commit_graphs_after > 1) {\n+\t\tchunk_offsets[num_chunks + 1] = chunk_offsets[num_chunks] +\n+\t\t\t\t\t\thashsz * (ctx->num_commit_graphs_after - 1);\n+\t\tnum_chunks++;\n+\t}\n \n \thashwrite_be32(f, GRAPH_SIGNATURE);\n \n \thashwrite_u8(f, GRAPH_VERSION);\n \thashwrite_u8(f, oid_version());\n \thashwrite_u8(f, num_chunks);\n-\thashwrite_u8(f, 0);\n+\thashwrite_u8(f, ctx->num_commit_graphs_after - 1);\n \n \tfor (i = 0; i <= num_chunks; i++) {\n \t\tuint32_t chunk_write[3];\n@@ -1279,11 +1439,67 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \twrite_graph_chunk_data(f, hashsz, ctx);\n \tif (ctx->num_extra_edges)\n \t\twrite_graph_chunk_extra_edges(f, ctx);\n+\tif (ctx->num_commit_graphs_after > 1 &&\n+\t write_graph_chunk_base(f, ctx)) {\n+\t\treturn -1;\n+\t}\n \tstop_progress(&ctx->progress);\n \tstrbuf_release(&progress_title);\n \n+\tif (ctx->split && ctx->base_graph_name && ctx->num_commit_graphs_after > 1) {\n+\t\tchar *new_base_hash = xstrdup(oid_to_hex(&ctx->new_base_graph->oid));\n+\t\tchar *new_base_name = get_split_graph_filename(ctx->obj_dir, new_base_hash);\n+\n+\t\tfree(ctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2]);\n+\t\tfree(ctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 2]);\n+\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2] = new_base_name;\n+\t\tctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 2] = new_base_hash;\n+\t}\n+\n \tclose_commit_graph(ctx->r->objects);\n-\tfinalize_hashfile(f, NULL, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n+\tfinalize_hashfile(f, file_hash.hash, CSUM_HASH_IN_STREAM | CSUM_FSYNC);\n+\n+\tif (ctx->split) {\n+\t\tFILE *chainf = fdopen_lock_file(&lk, \"w\");\n+\t\tchar *final_graph_name;\n+\t\tint result;\n+\n+\t\tclose(fd);\n+\n+\t\tif (!chainf) {\n+\t\t\terror(_(\"unable to open commit-graph chain file\"));\n+\t\t\treturn -1;\n+\t\t}\n+\n+\t\tif (ctx->base_graph_name) {\n+\t\t\tresult = rename(ctx->base_graph_name,\n+\t\t\t\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2]);\n+\n+\t\t\tif (result) {\n+\t\t\t\terror(_(\"failed to rename base commit-graph file\"));\n+\t\t\t\treturn -1;\n+\t\t\t}\n+\t\t} else {\n+\t\t\tchar *graph_name = get_commit_graph_filename(ctx->obj_dir);\n+\t\t\tunlink(graph_name);\n+\t\t}\n+\n+\t\tctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 1] = xstrdup(oid_to_hex(&file_hash));\n+\t\tfinal_graph_name = get_split_graph_filename(ctx->obj_dir,\n+\t\t\t\t\tctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 1]);\n+\t\tctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 1] = final_graph_name;\n+\n+\t\tresult = rename(ctx->graph_name, final_graph_name);\n+\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++)\n+\t\t\tfprintf(lk.tempfile->fp, \"%s\\n\", ctx->commit_graph_hash_after[i]);\n+\n+\t\tif (result) {\n+\t\t\terror(_(\"failed to rename temporary commit-graph file\"));\n+\t\t\treturn -1;\n+\t\t}\n+\t}\n+\n \tcommit_lock_file(&lk);\n \n \treturn 0;\n@@ -1306,6 +1522,30 @@ int write_commit_graph(const char *obj_dir,\n \tctx->obj_dir = obj_dir;\n \tctx->append = flags & COMMIT_GRAPH_APPEND ? 1 : 0;\n \tctx->report_progress = flags & COMMIT_GRAPH_PROGRESS ? 1 : 0;\n+\tctx->split = flags & COMMIT_GRAPH_SPLIT ? 1 : 0;\n+\n+\tif (ctx->split) {\n+\t\tstruct commit_graph *g;\n+\t\tprepare_commit_graph(ctx->r);\n+\n+\t\tg = ctx->r->objects->commit_graph;\n+\n+\t\twhile (g) {\n+\t\t\tctx->num_commit_graphs_before++;\n+\t\t\tg = g->base_graph;\n+\t\t}\n+\n+\t\tif (ctx->num_commit_graphs_before) {\n+\t\t\tALLOC_ARRAY(ctx->commit_graph_filenames_before, ctx->num_commit_graphs_before);\n+\t\t\ti = ctx->num_commit_graphs_before;\n+\t\t\tg = ctx->r->objects->commit_graph;\n+\n+\t\t\twhile (g) {\n+\t\t\t\tctx->commit_graph_filenames_before[--i] = xstrdup(g->filename);\n+\t\t\t\tg = g->base_graph;\n+\t\t\t}\n+\t\t}\n+\t}\n \n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n@@ -1360,6 +1600,14 @@ int write_commit_graph(const char *obj_dir,\n \t\tgoto cleanup;\n \t}\n \n+\tif (!ctx->commits.nr)\n+\t\tgoto cleanup;\n+\n+\tif (ctx->split)\n+\t\tinit_commit_graph_chain(ctx);\n+\telse\n+\t\tctx->num_commit_graphs_after = 1;\n+\n \tcompute_generation_numbers(ctx);\n \n \tres = write_commit_graph_file(ctx);\n@@ -1368,6 +1616,21 @@ int write_commit_graph(const char *obj_dir,\n \tfree(ctx->graph_name);\n \tfree(ctx->commits.list);\n \tfree(ctx->oids.list);\n+\n+\tif (ctx->commit_graph_filenames_after) {\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++) {\n+\t\t\tfree(ctx->commit_graph_filenames_after[i]);\n+\t\t\tfree(ctx->commit_graph_hash_after[i]);\n+\t\t}\n+\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_before; i++)\n+\t\t\tfree(ctx->commit_graph_filenames_before[i]);\n+\n+\t\tfree(ctx->commit_graph_filenames_after);\n+\t\tfree(ctx->commit_graph_filenames_before);\n+\t\tfree(ctx->commit_graph_hash_after);\n+\t}\n+\n \tfree(ctx);\n \n \treturn res;\n@@ -1555,5 +1818,6 @@ void free_commit_graph(struct commit_graph *g)\n \t\tg->data = NULL;\n \t\tclose(g->graph_fd);\n \t}\n+\tfree(g->filename);\n \tfree(g);\n }\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 6e7d42cf32..c321834533 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -47,6 +47,7 @@ struct commit_graph {\n \tunsigned char num_chunks;\n \tuint32_t num_commits;\n \tstruct object_id oid;\n+\tchar *filename;\n \n \tuint32_t num_commits_in_base;\n \tstruct commit_graph *base_graph;\n@@ -71,6 +72,7 @@ int generation_numbers_enabled(struct repository *r);\n \n #define COMMIT_GRAPH_APPEND (1 << 0)\n #define COMMIT_GRAPH_PROGRESS (1 << 1)\n+#define COMMIT_GRAPH_SPLIT (1 << 2)\n \n /*\n * The write_commit_graph* methods return zero on success\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 3b6fd0d728..063f906b3e 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -20,7 +20,7 @@ test_expect_success 'verify graph with no graph file' '\n test_expect_success 'write graph with no packs' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit-graph write --object-dir . &&\n-\ttest_path_is_file info/commit-graph\n+\ttest_path_is_missing info/commit-graph\n '\n \n test_expect_success 'close with correct error on bad input' '\n-- \ngitgitgadget\n\n"},{"id":"377439","messageId":"65b1cc6ae98fb801e43780439795e319396c1ce0.1560881661.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 10/18] commit-graph: allow cross-alternate chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-18T18:14:30Z","receivedAt":"2019-06-18T18:14:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn an environment like a fork network, it is helpful to have a\ncommit-graph chain that spans both the base repo and the fork repo. The\nfork is usually a small set of data on top of the large repo, but\nsometimes the fork is much larger. For example, git-for-windows/git has\nalmost double the number of commits as git/git because it rebases its\ncommits on every major version update.\n\nTo allow cross-alternate commit-graph chains, we need a few pieces:\n\n1. When looking for a graph-{hash}.graph file, check all alternates.\n\n2. When merging commit-graph chains, do not merge across alternates.\n\n3. When writing a new commit-graph chain based on a commit-graph file\n in another object directory, do not allow success if the base file\n has of the name \"commit-graph\" instead of\n \"commit-graphs/graph-{hash}.graph\".\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 40 +++++++++++++++++\n commit-graph.c | 56 +++++++++++++++++++-----\n commit-graph.h | 1 +\n t/t5324-split-commit-graph.sh | 37 ++++++++++++++++\n 4 files changed, 123 insertions(+), 11 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex d9c6253b0a..473032e476 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -266,6 +266,42 @@ The merge strategy values (2 for the size multiple, 64,000 for the maximum\n number of commits) could be extracted into config settings for full\n flexibility.\n \n+## Chains across multiple object directories\n+\n+In a repo with alternates, we look for the `commit-graph-chain` file starting\n+in the local object directory and then in each alternate. The first file that\n+exists defines our chain. As we look for the `graph-{hash}` files for\n+each `{hash}` in the chain file, we follow the same pattern for the host\n+directories.\n+\n+This allows commit-graphs to be split across multiple forks in a fork network.\n+The typical case is a large \"base\" repo with many smaller forks.\n+\n+As the base repo advances, it will likely update and merge its commit-graph\n+chain more frequently than the forks. If a fork updates their commit-graph after\n+the base repo, then it should \"reparent\" the commit-graph chain onto the new\n+chain in the base repo. When reading each `graph-{hash}` file, we track\n+the object directory containing it. During a write of a new commit-graph file,\n+we check for any changes in the source object directory and read the\n+`commit-graph-chain` file for that source and create a new file based on those\n+files. During this \"reparent\" operation, we necessarily need to collapse all\n+levels in the fork, as all of the files are invalid against the new base file.\n+\n+It is crucial to be careful when cleaning up \"unreferenced\" `graph-{hash}.graph`\n+files in this scenario. It falls to the user to define the proper settings for\n+their custom environment:\n+\n+ 1. When merging levels in the base repo, the unreferenced files may still be\n+ referenced by chains from fork repos.\n+\n+ 2. The expiry time should be set to a length of time such that every fork has\n+ time to recompute their commit-graph chain to \"reparent\" onto the new base\n+ file(s).\n+\n+ 3. If the commit-graph chain is updated in the base, the fork will not have\n+ access to the new chain until its chain is updated to reference those files.\n+ (This may change in the future [5].)\n+\n Related Links\n -------------\n [0] https://bugs.chromium.org/p/git/issues/detail?id=8\n@@ -292,3 +328,7 @@ Related Links\n \n [4] https://public-inbox.org/git/20180108154822.54829-1-git@jeffhostetler.com/T/#u\n A patch to remove the ahead-behind calculation from 'status'.\n+\n+[5] https://public-inbox.org/git/f27db281-abad-5043-6d71-cbb083b1c877@gmail.com/\n+ A discussion of a \"two-dimensional graph position\" that can allow reading\n+ multiple commit-graph chains at the same time.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex fb3100921c..fba705bc51 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -320,6 +320,9 @@ static struct commit_graph *load_commit_graph_v1(struct repository *r, const cha\n \tstruct commit_graph *g = load_commit_graph_one(graph_name);\n \tfree(graph_name);\n \n+\tif (g)\n+\t\tg->obj_dir = obj_dir;\n+\n \treturn g;\n }\n \n@@ -379,9 +382,10 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r, const\n \tcount = st.st_size / (the_hash_algo->hexsz + 1);\n \toids = xcalloc(count, sizeof(struct object_id));\n \n-\tfor (i = 0; i < count && valid; i++) {\n-\t\tchar *graph_name;\n-\t\tstruct commit_graph *g;\n+\tprepare_alt_odb(r);\n+\n+\tfor (i = 0; i < count; i++) {\n+\t\tstruct object_directory *odb;\n \n \t\tif (strbuf_getline_lf(&line, fp) == EOF)\n \t\t\tbreak;\n@@ -393,14 +397,29 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r, const\n \t\t\tbreak;\n \t\t}\n \n-\t\tgraph_name = get_split_graph_filename(obj_dir, line.buf);\n-\t\tg = load_commit_graph_one(graph_name);\n-\t\tfree(graph_name);\n+\t\tvalid = 0;\n+\t\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n+\t\t\tchar *graph_name = get_split_graph_filename(odb->path, line.buf);\n+\t\t\tstruct commit_graph *g = load_commit_graph_one(graph_name);\n \n-\t\tif (g && add_graph_to_chain(g, graph_chain, oids, i))\n-\t\t\tgraph_chain = g;\n-\t\telse\n-\t\t\tvalid = 0;\n+\t\t\tfree(graph_name);\n+\n+\t\t\tif (g) {\n+\t\t\t\tg->obj_dir = odb->path;\n+\n+\t\t\t\tif (add_graph_to_chain(g, graph_chain, oids, i)) {\n+\t\t\t\t\tgraph_chain = g;\n+\t\t\t\t\tvalid = 1;\n+\t\t\t\t}\n+\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (!valid) {\n+\t\t\twarning(_(\"unable to find all commit-graph files\"));\n+\t\t\tbreak;\n+\t\t}\n \t}\n \n \tfree(oids);\n@@ -1418,7 +1437,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \n \tif (ctx->split && ctx->base_graph_name && ctx->num_commit_graphs_after > 1) {\n \t\tchar *new_base_hash = xstrdup(oid_to_hex(&ctx->new_base_graph->oid));\n-\t\tchar *new_base_name = get_split_graph_filename(ctx->obj_dir, new_base_hash);\n+\t\tchar *new_base_name = get_split_graph_filename(ctx->new_base_graph->obj_dir, new_base_hash);\n \n \t\tfree(ctx->commit_graph_filenames_after[ctx->num_commit_graphs_after - 2]);\n \t\tfree(ctx->commit_graph_hash_after[ctx->num_commit_graphs_after - 2]);\n@@ -1493,6 +1512,9 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \n \twhile (g && (g->num_commits <= split_strategy_size_mult * num_commits ||\n \t\t num_commits > split_strategy_max_commits)) {\n+\t\tif (strcmp(g->obj_dir, ctx->obj_dir))\n+\t\t\tbreak;\n+\n \t\tnum_commits += g->num_commits;\n \t\tg = g->base_graph;\n \n@@ -1501,6 +1523,18 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \n \tctx->new_base_graph = g;\n \n+\tif (ctx->num_commit_graphs_after == 2) {\n+\t\tchar *old_graph_name = get_commit_graph_filename(g->obj_dir);\n+\n+\t\tif (!strcmp(g->filename, old_graph_name) &&\n+\t\t strcmp(g->obj_dir, ctx->obj_dir)) {\n+\t\t\tctx->num_commit_graphs_after = 1;\n+\t\t\tctx->new_base_graph = NULL;\n+\t\t}\n+\n+\t\tfree(old_graph_name);\n+\t}\n+\n \tALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after);\n \tALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after);\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex c321834533..802d35254f 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -48,6 +48,7 @@ struct commit_graph {\n \tuint32_t num_commits;\n \tstruct object_id oid;\n \tchar *filename;\n+\tconst char *obj_dir;\n \n \tuint32_t num_commits_in_base;\n \tstruct commit_graph *base_graph;\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 5cb5663a30..46f0832f68 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -90,6 +90,21 @@ test_expect_success 'add more commits, and write a new base graph' '\n \tgraph_read_expect 12\n '\n \n+test_expect_success 'fork and fail to base a chain on a commit-graph file' '\n+\ttest_when_finished rm -rf fork &&\n+\tgit clone . fork &&\n+\t(\n+\t\tcd fork &&\n+\t\trm .git/objects/info/commit-graph &&\n+\t\techo \"$(pwd)/../.git/objects\" >.git/objects/info/alternates &&\n+\t\ttest_commit new-commit &&\n+\t\tgit commit-graph write --reachable --split &&\n+\t\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n+\t\tverify_chain_files_exist $graphdir\n+\t)\n+'\n+\n test_expect_success 'add three more commits, write a tip graph' '\n \tgit reset --hard commits/3 &&\n \tgit merge merge/1 &&\n@@ -132,4 +147,26 @@ test_expect_success 'add one commit, write a merged graph' '\n \n graph_git_behavior 'merged commit-graph: commit 12 vs 6' commits/12 commits/6\n \n+test_expect_success 'create fork and chain across alternate' '\n+\tgit clone . fork &&\n+\t(\n+\t\tcd fork &&\n+\t\tgit config core.commitGraph true &&\n+\t\trm -rf $graphdir &&\n+\t\techo \"$(pwd)/../.git/objects\" >.git/objects/info/alternates &&\n+\t\ttest_commit 13 &&\n+\t\tgit branch commits/13 &&\n+\t\tgit commit-graph write --reachable --split &&\n+\t\ttest_path_is_file $graphdir/commit-graph-chain &&\n+\t\ttest_line_count = 3 $graphdir/commit-graph-chain &&\n+\t\tls $graphdir/graph-*.graph >graph-files &&\n+\t\ttest_line_count = 1 graph-files &&\n+\t\tgit -c core.commitGraph=true rev-list HEAD >expect &&\n+\t\tgit -c core.commitGraph=false rev-list HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+graph_git_behavior 'alternate: commit 13 vs 6' commits/13 commits/6\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"377440","messageId":"a06428d749198c180fe0cade50251e0c6b048658.1560881661.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 02/18] commit-graph: prepare for commit-graph chains","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-18T18:14:24Z","receivedAt":"2019-06-18T18:14:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nTo prepare for a chain of commit-graph files, augment the\ncommit_graph struct to point to a base commit_graph. As we load\ncommits from the graph, we may actually want to read from a base\nfile according to the graph position.\n\nThe \"graph position\" of a commit is given by concatenating the\nlexicographic commit orders from each of the commit-graph files in\nthe chain. This means that we must distinguish two values:\n\n * lexicographic index : the position within the lexicographic\n order in a single commit-graph file.\n\n * graph position: the position within the concatenated order\n of multiple commit-graph files\n\nGiven the lexicographic index of a commit in a graph, we can\ncompute the graph position by adding the number of commits in\nthe lower-level graphs. To find the lexicographic index of\na commit, we subtract the number of commits in lower-level graphs.\n\nWhile here, change insert_parent_or_die() to take a uint32_t\nposition, as that is the type used by its only caller and that\nmakes more sense with the limits in the commit-graph format.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 89 +++++++++++++++++++++++++++++++++++++++++++-------\n commit-graph.h | 3 ++\n 2 files changed, 81 insertions(+), 11 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 76d189de45..8f5c09363c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -359,9 +359,18 @@ int generation_numbers_enabled(struct repository *r)\n \treturn !!first_generation;\n }\n \n+static void close_commit_graph_one(struct commit_graph *g)\n+{\n+\tif (!g)\n+\t\treturn;\n+\n+\tclose_commit_graph_one(g->base_graph);\n+\tfree_commit_graph(g);\n+}\n+\n void close_commit_graph(struct raw_object_store *o)\n {\n-\tfree_commit_graph(o->commit_graph);\n+\tclose_commit_graph_one(o->commit_graph);\n \to->commit_graph = NULL;\n }\n \n@@ -371,18 +380,38 @@ static int bsearch_graph(struct commit_graph *g, struct object_id *oid, uint32_t\n \t\t\t g->chunk_oid_lookup, g->hash_len, pos);\n }\n \n+static void load_oid_from_graph(struct commit_graph *g,\n+\t\t\t\tuint32_t pos,\n+\t\t\t\tstruct object_id *oid)\n+{\n+\tuint32_t lex_index;\n+\n+\twhile (g && pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tif (!g)\n+\t\tBUG(\"NULL commit-graph\");\n+\n+\tif (pos >= g->num_commits + g->num_commits_in_base)\n+\t\tdie(_(\"invalid commit position. commit-graph is likely corrupt\"));\n+\n+\tlex_index = pos - g->num_commits_in_base;\n+\n+\thashcpy(oid->hash, g->chunk_oid_lookup + g->hash_len * lex_index);\n+}\n+\n static struct commit_list **insert_parent_or_die(struct repository *r,\n \t\t\t\t\t\t struct commit_graph *g,\n-\t\t\t\t\t\t uint64_t pos,\n+\t\t\t\t\t\t uint32_t pos,\n \t\t\t\t\t\t struct commit_list **pptr)\n {\n \tstruct commit *c;\n \tstruct object_id oid;\n \n-\tif (pos >= g->num_commits)\n-\t\tdie(\"invalid parent position %\"PRIu64, pos);\n+\tif (pos >= g->num_commits + g->num_commits_in_base)\n+\t\tdie(\"invalid parent position %\"PRIu32, pos);\n \n-\thashcpy(oid.hash, g->chunk_oid_lookup + g->hash_len * pos);\n+\tload_oid_from_graph(g, pos, &oid);\n \tc = lookup_commit(r, &oid);\n \tif (!c)\n \t\tdie(_(\"could not find commit %s\"), oid_to_hex(&oid));\n@@ -392,7 +421,14 @@ static struct commit_list **insert_parent_or_die(struct repository *r,\n \n static void fill_commit_graph_info(struct commit *item, struct commit_graph *g, uint32_t pos)\n {\n-\tconst unsigned char *commit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * pos;\n+\tconst unsigned char *commit_data;\n+\tuint32_t lex_index;\n+\n+\twhile (pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tlex_index = pos - g->num_commits_in_base;\n+\tcommit_data = g->chunk_commit_data + GRAPH_DATA_WIDTH * lex_index;\n \titem->graph_pos = pos;\n \titem->generation = get_be32(commit_data + g->hash_len + 8) >> 2;\n }\n@@ -405,10 +441,25 @@ static int fill_commit_in_graph(struct repository *r,\n \tuint32_t *parent_data_ptr;\n \tuint64_t date_low, date_high;\n \tstruct commit_list **pptr;\n-\tconst unsigned char *commit_data = g->chunk_commit_data + (g->hash_len + 16) * pos;\n+\tconst unsigned char *commit_data;\n+\tuint32_t lex_index;\n \n-\titem->object.parsed = 1;\n+\twhile (pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tif (pos >= g->num_commits + g->num_commits_in_base)\n+\t\tdie(_(\"invalid commit position. commit-graph is likely corrupt\"));\n+\n+\t/*\n+\t * Store the \"full\" position, but then use the\n+\t * \"local\" position for the rest of the calculation.\n+\t */\n \titem->graph_pos = pos;\n+\tlex_index = pos - g->num_commits_in_base;\n+\n+\tcommit_data = g->chunk_commit_data + (g->hash_len + 16) * lex_index;\n+\n+\titem->object.parsed = 1;\n \n \titem->maybe_tree = NULL;\n \n@@ -452,7 +503,18 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \t\t*pos = item->graph_pos;\n \t\treturn 1;\n \t} else {\n-\t\treturn bsearch_graph(g, &(item->object.oid), pos);\n+\t\tstruct commit_graph *cur_g = g;\n+\t\tuint32_t lex_index;\n+\n+\t\twhile (cur_g && !bsearch_graph(cur_g, &(item->object.oid), &lex_index))\n+\t\t\tcur_g = cur_g->base_graph;\n+\n+\t\tif (cur_g) {\n+\t\t\t*pos = lex_index + cur_g->num_commits_in_base;\n+\t\t\treturn 1;\n+\t\t}\n+\n+\t\treturn 0;\n \t}\n }\n \n@@ -492,8 +554,13 @@ static struct tree *load_tree_for_commit(struct repository *r,\n \t\t\t\t\t struct commit *c)\n {\n \tstruct object_id oid;\n-\tconst unsigned char *commit_data = g->chunk_commit_data +\n-\t\t\t\t\t GRAPH_DATA_WIDTH * (c->graph_pos);\n+\tconst unsigned char *commit_data;\n+\n+\twhile (c->graph_pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tcommit_data = g->chunk_commit_data +\n+\t\t\tGRAPH_DATA_WIDTH * (c->graph_pos - g->num_commits_in_base);\n \n \thashcpy(oid.hash, commit_data);\n \tc->maybe_tree = lookup_tree(r, &oid);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 390c7f6961..5819910a5b 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -48,6 +48,9 @@ struct commit_graph {\n \tuint32_t num_commits;\n \tstruct object_id oid;\n \n+\tuint32_t num_commits_in_base;\n+\tstruct commit_graph *base_graph;\n+\n \tconst uint32_t *chunk_oid_fanout;\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_commit_data;\n-- \ngitgitgadget\n\n"},{"id":"377441","messageId":"a4aa473c55e5df8a61f63742a2e27a2483ab6291.1560881661.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 14/18] commit-graph: clean up chains after flattened write","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-18T18:14:33Z","receivedAt":"2019-06-18T18:14:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf we write a commit-graph file without the split option, then\nwe write to $OBJDIR/info/commit-graph and start to ignore\nthe chains in $OBJDIR/info/commit-graphs/.\n\nUnlink the commit-graph-chain file and expire the graph-{hash}.graph\nfiles in $OBJDIR/info/commit-graphs/ during every write.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 12 +++++++++---\n t/t5324-split-commit-graph.sh | 12 ++++++++++++\n 2 files changed, 21 insertions(+), 3 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f33f4fe009..3599ae664d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1687,6 +1687,12 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \n \tif (ctx->split_opts && ctx->split_opts->expire_time)\n \t\texpire_time -= ctx->split_opts->expire_time;\n+\tif (!ctx->split) {\n+\t\tchar *chain_file_name = get_chain_filename(ctx->obj_dir);\n+\t\tunlink(chain_file_name);\n+\t\tfree(chain_file_name);\n+\t\tctx->num_commit_graphs_after = 0;\n+\t}\n \n \tstrbuf_addstr(&path, ctx->obj_dir);\n \tstrbuf_addstr(&path, \"/info/commit-graphs\");\n@@ -1841,10 +1847,10 @@ int write_commit_graph(const char *obj_dir,\n \n \tres = write_commit_graph_file(ctx);\n \n-\tif (ctx->split) {\n+\tif (ctx->split)\n \t\tmark_commit_graphs(ctx);\n-\t\texpire_commit_graphs(ctx);\n-\t}\n+\n+\texpire_commit_graphs(ctx);\n \n cleanup:\n \tfree(ctx->graph_name);\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 3df90ae58f..e8df35c30b 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -216,6 +216,18 @@ test_expect_success 'test merge stragety constants' '\n \t)\n '\n \n+test_expect_success 'remove commit-graph-chain file after flattening' '\n+\tgit clone . flatten &&\n+\t(\n+\t\tcd flatten &&\n+\t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\t\tgit commit-graph write --reachable &&\n+\t\ttest_path_is_missing $graphdir/commit-graph-chain &&\n+\t\tls $graphdir >graph-files &&\n+\t\ttest_line_count = 0 graph-files\n+\t)\n+'\n+\n corrupt_file() {\n \tfile=$1\n \tpos=$2\n-- \ngitgitgadget\n\n"},{"id":"377442","messageId":"4877166d2ff74ff75f72a2c4d92b25bd3a6b343d.1560881661.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 15/18] commit-graph: test octopus merges with --split","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-18T18:14:34Z","receivedAt":"2019-06-18T18:14:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nOctopus merges require an extra chunk of data in the commit-graph\nfile format. Create a test that ensures the new --split option\ncontinues to work with an octopus merge. Specifically, ensure\nthat the octopus merge has parents across layers to truly check\nthat our graph position logic holds up correctly.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5324-split-commit-graph.sh | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex e8df35c30b..704def70bb 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -290,4 +290,15 @@ test_expect_success 'verify after commit-graph-chain corruption' '\n \t)\n '\n \n+test_expect_success 'add octopus merge' '\n+\tgit reset --hard commits/10 &&\n+\tgit merge commits/3 commits/4 &&\n+\tgit branch merge/octopus &&\n+\tgit commit-graph write --reachable --split &&\n+\tgit commit-graph verify &&\n+\ttest_line_count = 3 $graphdir/commit-graph-chain\n+'\n+\n+graph_git_behavior 'graph exists' merge/octopus commits/12\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"377443","messageId":"bad9f13cc27663051dd3d9eebdd61a6287129861.1560881661.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 18/18] commit-graph: test verify across alternates","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-18T18:14:36Z","receivedAt":"2019-06-18T18:14:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git commit-graph verify' subcommand loads a commit-graph from\na given object directory instead of using the standard method\nprepare_commit_graph(). During development of load_commit_graph_chain(),\na version did not include prepare_alt_odb() as it was previously\nrun by prepare_commit_graph() in most cases.\n\nAdd a test that prevents that mistake from happening again.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5324-split-commit-graph.sh | 19 +++++++++++++++++++\n 1 file changed, 19 insertions(+)\n\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex fc0d00751c..03f45a1ed9 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -237,6 +237,7 @@ corrupt_file() {\n \tfile=$1\n \tpos=$2\n \tdata=\"${3:-\\0}\"\n+\tchmod a+w \"$file\" &&\n \tprintf \"$data\" | dd of=\"$file\" bs=1 seek=\"$pos\" conv=notrunc\n }\n \n@@ -295,6 +296,24 @@ test_expect_success 'verify after commit-graph-chain corruption' '\n \t)\n '\n \n+test_expect_success 'verify across alternates' '\n+\tgit clone --no-hardlinks . verify-alt &&\n+\t(\n+\t\tcd verify-alt &&\n+\t\trm -rf $graphdir &&\n+\t\taltdir=\"$(pwd)/../.git/objects\" &&\n+\t\techo \"$altdir\" >.git/objects/info/alternates &&\n+\t\tgit commit-graph verify --object-dir=\"$altdir/\" &&\n+\t\ttest_commit extra &&\n+\t\tgit commit-graph write --reachable --split &&\n+\t\ttip_file=$graphdir/graph-$(tail -n 1 $graphdir/commit-graph-chain).graph &&\n+\t\tcorrupt_file \"$tip_file\" 100 \"\\01\" &&\n+\t\ttest_must_fail git commit-graph verify --shallow 2>test_err &&\n+\t\tgrep -v \"^+\" test_err >err &&\n+\t\ttest_i18ngrep \"commit-graph has incorrect fanout value\" err\n+\t)\n+'\n+\n test_expect_success 'add octopus merge' '\n \tgit reset --hard commits/10 &&\n \tgit merge commits/3 commits/4 &&\n-- \ngitgitgadget\n"},{"id":"377444","messageId":"2d2977b7df8214cace865cfebc95e1b83fd5f2cb.1560881661.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 11/18] commit-graph: expire commit-graph files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-18T18:14:31Z","receivedAt":"2019-06-18T18:14:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAs we merge commit-graph files in a commit-graph chain, we should clean\nup the files that are no longer used.\n\nThis change introduces an 'expiry_window' value to the context, which is\nalways zero (for now). We then check the modified time of each\ngraph-{hash}.graph file in the $OBJDIR/info/commit-graphs folder and\nunlink the files that are older than the expiry_window.\n\nSince this is always zero, this immediately clears all unused graph\nfiles. We will update the value to match a config setting in a future\nchange.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 15 ++++++\n commit-graph.c | 69 ++++++++++++++++++++++++\n t/t5324-split-commit-graph.sh | 2 +-\n 3 files changed, 85 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex 473032e476..aed4350a59 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -266,6 +266,21 @@ The merge strategy values (2 for the size multiple, 64,000 for the maximum\n number of commits) could be extracted into config settings for full\n flexibility.\n \n+## Deleting graph-{hash} files\n+\n+After a new tip file is written, some `graph-{hash}` files may no longer\n+be part of a chain. It is important to remove these files from disk, eventually.\n+The main reason to delay removal is that another process could read the\n+`commit-graph-chain` file before it is rewritten, but then look for the\n+`graph-{hash}` files after they are deleted.\n+\n+To allow holding old split commit-graphs for a while after they are unreferenced,\n+we update the modified times of the files when they become unreferenced. Then,\n+we scan the `$OBJDIR/info/commit-graphs/` directory for `graph-{hash}`\n+files whose modified times are older than a given expiry window. This window\n+defaults to zero, but can be changed using command-line arguments or a config\n+setting.\n+\n ## Chains across multiple object directories\n \n In a repo with alternates, we look for the `commit-graph-chain` file starting\ndiff --git a/commit-graph.c b/commit-graph.c\nindex fba705bc51..0cc2ceb349 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1652,6 +1652,70 @@ static void merge_commit_graphs(struct write_commit_graph_context *ctx)\n \tsort_and_scan_merged_commits(ctx);\n }\n \n+static void mark_commit_graphs(struct write_commit_graph_context *ctx)\n+{\n+\tuint32_t i;\n+\ttime_t now = time(NULL);\n+\n+\tfor (i = ctx->num_commit_graphs_after - 1; i < ctx->num_commit_graphs_before; i++) {\n+\t\tstruct stat st;\n+\t\tstruct utimbuf updated_time;\n+\n+\t\tstat(ctx->commit_graph_filenames_before[i], &st);\n+\n+\t\tupdated_time.actime = st.st_atime;\n+\t\tupdated_time.modtime = now;\n+\t\tutime(ctx->commit_graph_filenames_before[i], &updated_time);\n+\t}\n+}\n+\n+static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\tDIR *dir;\n+\tstruct dirent *de;\n+\tsize_t dirnamelen;\n+\ttime_t expire_time = time(NULL);\n+\n+\tstrbuf_addstr(&path, ctx->obj_dir);\n+\tstrbuf_addstr(&path, \"/info/commit-graphs\");\n+\tdir = opendir(path.buf);\n+\n+\tif (!dir) {\n+\t\tstrbuf_release(&path);\n+\t\treturn;\n+\t}\n+\n+\tstrbuf_addch(&path, '/');\n+\tdirnamelen = path.len;\n+\twhile ((de = readdir(dir)) != NULL) {\n+\t\tstruct stat st;\n+\t\tuint32_t i, found = 0;\n+\n+\t\tstrbuf_setlen(&path, dirnamelen);\n+\t\tstrbuf_addstr(&path, de->d_name);\n+\n+\t\tstat(path.buf, &st);\n+\n+\t\tif (st.st_mtime > expire_time)\n+\t\t\tcontinue;\n+\t\tif (path.len < 6 || strcmp(path.buf + path.len - 6, \".graph\"))\n+\t\t\tcontinue;\n+\n+\t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++) {\n+\t\t\tif (!strcmp(ctx->commit_graph_filenames_after[i],\n+\t\t\t\t path.buf)) {\n+\t\t\t\tfound = 1;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (!found)\n+\t\t\tunlink(path.buf);\n+\n+\t}\n+}\n+\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n@@ -1764,6 +1828,11 @@ int write_commit_graph(const char *obj_dir,\n \n \tres = write_commit_graph_file(ctx);\n \n+\tif (ctx->split) {\n+\t\tmark_commit_graphs(ctx);\n+\t\texpire_commit_graphs(ctx);\n+\t}\n+\n cleanup:\n \tfree(ctx->graph_name);\n \tfree(ctx->commits.list);\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 46f0832f68..76068ee407 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -141,7 +141,7 @@ test_expect_success 'add one commit, write a merged graph' '\n \ttest_path_is_file $graphdir/commit-graph-chain &&\n \ttest_line_count = 2 $graphdir/commit-graph-chain &&\n \tls $graphdir/graph-*.graph >graph-files &&\n-\ttest_line_count = 4 graph-files &&\n+\ttest_line_count = 2 graph-files &&\n \tverify_chain_files_exist $graphdir\n '\n \n-- \ngitgitgadget\n\n"},{"id":"377445","messageId":"5e56632b8b2ec768379a9ebc42a6e5525bca337a.1560881661.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 13/18] commit-graph: verify chains with --shallow mode","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-18T18:14:32Z","receivedAt":"2019-06-18T18:14:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIf we wrote a commit-graph chain, we only modified the tip file in\nthe chain. It is valuable to verify what we wrote, but not waste\ntime checking files we did not write.\n\nAdd a '--shallow' option to the 'git commit-graph verify' subcommand\nand check that it does not read the base graph in a two-file chain.\n\nMaking the verify subcommand read from a chain of commit-graphs takes\nsome rearranging of the builtin code.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 5 ++-\n builtin/commit-graph.c | 27 +++++++++----\n commit-graph.c | 15 ++++++--\n commit-graph.h | 6 ++-\n t/t5324-split-commit-graph.sh | 62 ++++++++++++++++++++++++++++++\n 5 files changed, 101 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 365e145e82..eb5e7865f0 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -10,7 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git commit-graph read' [--object-dir ]\n-'git commit-graph verify' [--object-dir ]\n+'git commit-graph verify' [--object-dir ] [--shallow]\n 'git commit-graph write' [--object-dir ]\n \n \n@@ -80,6 +80,9 @@ Used for debugging purposes.\n \n Read the commit-graph file and verify its contents against the object\n database. Used to check for corrupted data.\n++\n+With the `--shallow` option, only check the tip commit-graph file in\n+a chain of split commit-graphs.\n \n \n EXAMPLES\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 6c0b5b17e0..38027b83d9 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -5,17 +5,18 @@\n #include \"parse-options.h\"\n #include \"repository.h\"\n #include \"commit-graph.h\"\n+#include \"object-store.h\"\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n-\tN_(\"git commit-graph verify [--object-dir ]\"),\n+\tN_(\"git commit-graph verify [--object-dir ] [--shallow]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] \"),\n \tNULL\n };\n \n static const char * const builtin_commit_graph_verify_usage[] = {\n-\tN_(\"git commit-graph verify [--object-dir ]\"),\n+\tN_(\"git commit-graph verify [--object-dir ] [--shallow]\"),\n \tNULL\n };\n \n@@ -36,6 +37,7 @@ static struct opts_commit_graph {\n \tint stdin_commits;\n \tint append;\n \tint split;\n+\tint shallow;\n } opts;\n \n static int graph_verify(int argc, const char **argv)\n@@ -45,11 +47,14 @@ static int graph_verify(int argc, const char **argv)\n \tint open_ok;\n \tint fd;\n \tstruct stat st;\n+\tint flags = 0;\n \n \tstatic struct option builtin_commit_graph_verify_options[] = {\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n \t\t\t N_(\"dir\"),\n \t\t\t N_(\"The object directory to store the graph\")),\n+\t\tOPT_BOOL(0, \"shallow\", &opts.shallow,\n+\t\t\t N_(\"if the commit-graph is split, only verify the tip file\")),\n \t\tOPT_END(),\n \t};\n \n@@ -59,21 +64,27 @@ static int graph_verify(int argc, const char **argv)\n \n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n+\tif (opts.shallow)\n+\t\tflags |= COMMIT_GRAPH_VERIFY_SHALLOW;\n \n \tgraph_name = get_commit_graph_filename(opts.obj_dir);\n \topen_ok = open_commit_graph(graph_name, &fd, &st);\n-\tif (!open_ok && errno == ENOENT)\n-\t\treturn 0;\n-\tif (!open_ok)\n+\tif (!open_ok && errno != ENOENT)\n \t\tdie_errno(_(\"Could not open commit-graph '%s'\"), graph_name);\n-\tgraph = load_commit_graph_one_fd_st(fd, &st);\n+\n \tFREE_AND_NULL(graph_name);\n \n+\tif (open_ok)\n+\t\tgraph = load_commit_graph_one_fd_st(fd, &st);\n+\t else\n+\t\tgraph = read_commit_graph_one(the_repository, opts.obj_dir);\n+\n+\t/* Return failure if open_ok predicted success */\n \tif (!graph)\n-\t\treturn 1;\n+\t\treturn !!open_ok;\n \n \tUNLEAK(graph);\n-\treturn verify_commit_graph(the_repository, graph);\n+\treturn verify_commit_graph(the_repository, graph, flags);\n }\n \n static int graph_read(int argc, const char **argv)\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 315088d205..f33f4fe009 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -428,7 +428,7 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r, const\n \treturn graph_chain;\n }\n \n-static struct commit_graph *read_commit_graph_one(struct repository *r, const char *obj_dir)\n+struct commit_graph *read_commit_graph_one(struct repository *r, const char *obj_dir)\n {\n \tstruct commit_graph *g = load_commit_graph_v1(r, obj_dir);\n \n@@ -1887,7 +1887,7 @@ static void graph_report(const char *fmt, ...)\n #define GENERATION_ZERO_EXISTS 1\n #define GENERATION_NUMBER_EXISTS 2\n \n-int verify_commit_graph(struct repository *r, struct commit_graph *g)\n+int verify_commit_graph(struct repository *r, struct commit_graph *g, int flags)\n {\n \tuint32_t i, cur_fanout_pos = 0;\n \tstruct object_id prev_oid, cur_oid, checksum;\n@@ -1895,6 +1895,7 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \tstruct hashfile *f;\n \tint devnull;\n \tstruct progress *progress = NULL;\n+\tint local_error = 0;\n \n \tif (!g) {\n \t\tgraph_report(\"no commit-graph file loaded\");\n@@ -1989,6 +1990,9 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t\t\t\tbreak;\n \t\t\t}\n \n+\t\t\t/* parse parent in case it is in a base graph */\n+\t\t\tparse_commit_in_graph_one(r, g, graph_parents->item);\n+\n \t\t\tif (!oideq(&graph_parents->item->object.oid, &odb_parents->item->object.oid))\n \t\t\t\tgraph_report(_(\"commit-graph parent for %s is %s != %s\"),\n \t\t\t\t\t oid_to_hex(&cur_oid),\n@@ -2040,7 +2044,12 @@ int verify_commit_graph(struct repository *r, struct commit_graph *g)\n \t}\n \tstop_progress(&progress);\n \n-\treturn verify_commit_graph_error;\n+\tlocal_error = verify_commit_graph_error;\n+\n+\tif (!(flags & COMMIT_GRAPH_VERIFY_SHALLOW) && g->base_graph)\n+\t\tlocal_error |= verify_commit_graph(r, g->base_graph, flags);\n+\n+\treturn local_error;\n }\n \n void free_commit_graph(struct commit_graph *g)\ndiff --git a/commit-graph.h b/commit-graph.h\nindex a84c22a560..df9a3b20e4 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -61,7 +61,7 @@ struct commit_graph {\n };\n \n struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st);\n-\n+struct commit_graph *read_commit_graph_one(struct repository *r, const char *obj_dir);\n struct commit_graph *parse_commit_graph(void *graph_map, int fd,\n \t\t\t\t\tsize_t graph_size);\n \n@@ -95,7 +95,9 @@ int write_commit_graph(const char *obj_dir,\n \t\t unsigned int flags,\n \t\t const struct split_commit_graph_opts *split_opts);\n \n-int verify_commit_graph(struct repository *r, struct commit_graph *g);\n+#define COMMIT_GRAPH_VERIFY_SHALLOW\t(1 << 0)\n+\n+int verify_commit_graph(struct repository *r, struct commit_graph *g, int flags);\n \n void close_commit_graph(struct raw_object_store *);\n void free_commit_graph(struct commit_graph *);\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 1b699a543c..3df90ae58f 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -216,4 +216,66 @@ test_expect_success 'test merge stragety constants' '\n \t)\n '\n \n+corrupt_file() {\n+\tfile=$1\n+\tpos=$2\n+\tdata=\"${3:-\\0}\"\n+\tprintf \"$data\" | dd of=\"$file\" bs=1 seek=\"$pos\" conv=notrunc\n+}\n+\n+test_expect_success 'verify hashes along chain, even in shallow' '\n+\tgit clone --no-hardlinks . verify &&\n+\t(\n+\t\tcd verify &&\n+\t\tgit commit-graph verify &&\n+\t\tbase_file=$graphdir/graph-$(head -n 1 $graphdir/commit-graph-chain).graph &&\n+\t\tcorrupt_file \"$base_file\" 1760 \"\\01\" &&\n+\t\ttest_must_fail git commit-graph verify --shallow 2>test_err &&\n+\t\tgrep -v \"^+\" test_err >err &&\n+\t\ttest_i18ngrep \"incorrect checksum\" err\n+\t)\n+'\n+\n+test_expect_success 'verify --shallow does not check base contents' '\n+\tgit clone --no-hardlinks . verify-shallow &&\n+\t(\n+\t\tcd verify-shallow &&\n+\t\tgit commit-graph verify &&\n+\t\tbase_file=$graphdir/graph-$(head -n 1 $graphdir/commit-graph-chain).graph &&\n+\t\tcorrupt_file \"$base_file\" 1000 \"\\01\" &&\n+\t\tgit commit-graph verify --shallow &&\n+\t\ttest_must_fail git commit-graph verify 2>test_err &&\n+\t\tgrep -v \"^+\" test_err >err &&\n+\t\ttest_i18ngrep \"incorrect checksum\" err\n+\t)\n+'\n+\n+test_expect_success 'warn on base graph chunk incorrect' '\n+\tgit clone --no-hardlinks . base-chunk &&\n+\t(\n+\t\tcd base-chunk &&\n+\t\tgit commit-graph verify &&\n+\t\tbase_file=$graphdir/graph-$(tail -n 1 $graphdir/commit-graph-chain).graph &&\n+\t\tcorrupt_file \"$base_file\" 1376 \"\\01\" &&\n+\t\tgit commit-graph verify --shallow 2>test_err &&\n+\t\tgrep -v \"^+\" test_err >err &&\n+\t\ttest_i18ngrep \"commit-graph chain does not match\" err\n+\t)\n+'\n+\n+test_expect_success 'verify after commit-graph-chain corruption' '\n+\tgit clone --no-hardlinks . verify-chain &&\n+\t(\n+\t\tcd verify-chain &&\n+\t\tcorrupt_file \"$graphdir/commit-graph-chain\" 60 \"G\" &&\n+\t\tgit commit-graph verify 2>test_err &&\n+\t\tgrep -v \"^+\" test_err >err &&\n+\t\ttest_i18ngrep \"invalid commit-graph chain\" err &&\n+\t\tcorrupt_file \"$graphdir/commit-graph-chain\" 60 \"A\" &&\n+\t\tgit commit-graph verify 2>test_err &&\n+\t\tgrep -v \"^+\" test_err >err &&\n+\t\ttest_i18ngrep \"unable to find all commit-graph files\" err\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"377446","messageId":"71f3e9fda9305faebd88477b67ee6c1d2b106425.1560881661.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 12/18] commit-graph: create options for split files","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-18T18:14:32Z","receivedAt":"2019-06-18T18:14:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe split commit-graph feature is now fully implemented, but needs\nsome more run-time configurability. Allow direct callers to 'git\ncommit-graph write --split' to specify the values used in the\nmerge strategy and the expire time.\n\nUpdate the documentation to specify these values.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 21 ++++++++++-\n Documentation/technical/commit-graph.txt | 7 ++--\n builtin/commit-graph.c | 25 ++++++++++---\n builtin/commit.c | 2 +-\n builtin/gc.c | 3 +-\n commit-graph.c | 35 ++++++++++++------\n commit-graph.h | 12 +++++-\n t/t5324-split-commit-graph.sh | 47 ++++++++++++++++++++++++\n 8 files changed, 128 insertions(+), 24 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 624470e198..365e145e82 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -26,7 +26,7 @@ OPTIONS\n \tUse given directory for the location of packfiles and commit-graph\n \tfile. This parameter exists to specify the location of an alternate\n \tthat only has the objects directory, not a full `.git` directory. The\n-\tcommit-graph file is expected to be at `/info/commit-graph` and\n+\tcommit-graph file is expected to be in the `/info` directory and\n \tthe packfiles are expected to be in `/pack`.\n \n \n@@ -51,6 +51,25 @@ or `--stdin-packs`.)\n +\n With the `--append` option, include all commits that are present in the\n existing commit-graph file.\n++\n+With the `--split` option, write the commit-graph as a chain of multiple\n+commit-graph files stored in `/info/commit-graphs`. The new commits\n+not already in the commit-graph are added in a new \"tip\" file. This file\n+is merged with the existing file if the following merge conditions are\n+met:\n++\n+* If `--size-multiple=` is not specified, let `X` equal 2. If the new\n+tip file would have `N` commits and the previous tip has `M` commits and\n+`X` times `N` is greater than `M`, instead merge the two files into a\n+single file.\n++\n+* If `--max-commits=` is specified with `M` a positive integer, and the\n+new tip file would have more than `M` commits, then instead merge the new\n+tip with the previous tip.\n++\n+Finally, if `--expire-time=` is not specified, let `datetime`\n+be the current time. After writing the split commit-graph, delete all\n+unused commit-graph whose modified times are older than `datetime`.\n \n 'read'::\n \ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nindex aed4350a59..729fbcb32f 100644\n--- a/Documentation/technical/commit-graph.txt\n+++ b/Documentation/technical/commit-graph.txt\n@@ -248,10 +248,11 @@ When writing a set of commits that do not exist in the commit-graph stack of\n height N, we default to creating a new file at level N + 1. We then decide to\n merge with the Nth level if one of two conditions hold:\n \n- 1. The expected file size for level N + 1 is at least half the file size for\n- level N.\n+ 1. `--size-multiple=` is specified or X = 2, and the number of commits in\n+ level N is less than X times the number of commits in level N + 1.\n \n- 2. Level N + 1 contains more than 64,0000 commits.\n+ 2. `--max-commits=` is specified with non-zero C and the number of commits\n+ in level N + 1 is more than C commits.\n \n This decision cascades down the levels: when we merge a level we create a new\n set of commits that then compares to the next level.\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex ff8bc3c8b9..6c0b5b17e0 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -10,7 +10,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ]\"),\n \tN_(\"git commit-graph verify [--object-dir ]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] \"),\n \tNULL\n };\n \n@@ -25,7 +25,7 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] \"),\n \tNULL\n };\n \n@@ -135,6 +135,7 @@ static int graph_read(int argc, const char **argv)\n }\n \n extern int read_replace_refs;\n+static struct split_commit_graph_opts split_opts;\n \n static int graph_write(int argc, const char **argv)\n {\n@@ -158,9 +159,19 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"include all commits already in the commit-graph file\")),\n \t\tOPT_BOOL(0, \"split\", &opts.split,\n \t\t\tN_(\"allow writing an incremental commit-graph file\")),\n+\t\tOPT_INTEGER(0, \"max-commits\", &split_opts.max_commits,\n+\t\t\tN_(\"maximum number of commits in a non-base split commit-graph\")),\n+\t\tOPT_INTEGER(0, \"size-multiple\", &split_opts.size_multiple,\n+\t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n+\t\tOPT_EXPIRY_DATE(0, \"expire-time\", &split_opts.expire_time,\n+\t\t\tN_(\"maximum number of commits in a non-base split commit-graph\")),\n \t\tOPT_END(),\n \t};\n \n+\tsplit_opts.size_multiple = 2;\n+\tsplit_opts.max_commits = 0;\n+\tsplit_opts.expire_time = 0;\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\n@@ -176,8 +187,11 @@ static int graph_write(int argc, const char **argv)\n \n \tread_replace_refs = 0;\n \n-\tif (opts.reachable)\n-\t\treturn write_commit_graph_reachable(opts.obj_dir, flags);\n+\tif (opts.reachable) {\n+\t\tif (write_commit_graph_reachable(opts.obj_dir, flags, &split_opts))\n+\t\t\treturn 1;\n+\t\treturn 0;\n+\t}\n \n \tstring_list_init(&lines, 0);\n \tif (opts.stdin_packs || opts.stdin_commits) {\n@@ -197,7 +211,8 @@ static int graph_write(int argc, const char **argv)\n \tif (write_commit_graph(opts.obj_dir,\n \t\t\t pack_indexes,\n \t\t\t commit_hex,\n-\t\t\t flags))\n+\t\t\t flags,\n+\t\t\t &split_opts))\n \t\tresult = 1;\n \n \tUNLEAK(lines);\ndiff --git a/builtin/commit.c b/builtin/commit.c\nindex b001ef565d..9216e9c043 100644\n--- a/builtin/commit.c\n+++ b/builtin/commit.c\n@@ -1670,7 +1670,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)\n \t\t \"not exceeded, and then \\\"git reset HEAD\\\" to recover.\"));\n \n \tif (git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n-\t write_commit_graph_reachable(get_object_directory(), 0))\n+\t write_commit_graph_reachable(get_object_directory(), 0, NULL))\n \t\treturn 1;\n \n \trepo_rerere(the_repository, 0);\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 20c8f1bfe8..0aa8eac747 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -666,7 +666,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \n \tif (gc_write_commit_graph &&\n \t write_commit_graph_reachable(get_object_directory(),\n-\t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_PROGRESS : 0))\n+\t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_PROGRESS : 0,\n+\t\t\t\t\t NULL))\n \t\treturn 1;\n \n \tif (auto_gc && too_many_loose_objects())\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0cc2ceb349..315088d205 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -768,6 +768,8 @@ struct write_commit_graph_context {\n \tunsigned append:1,\n \t\t report_progress:1,\n \t\t split:1;\n+\n+\tconst struct split_commit_graph_opts *split_opts;\n };\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n@@ -1116,14 +1118,15 @@ static int add_ref_to_list(const char *refname,\n \treturn 0;\n }\n \n-int write_commit_graph_reachable(const char *obj_dir, unsigned int flags)\n+int write_commit_graph_reachable(const char *obj_dir, unsigned int flags,\n+\t\t\t\t const struct split_commit_graph_opts *split_opts)\n {\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n \tint result;\n \n \tfor_each_ref(add_ref_to_list, &list);\n \tresult = write_commit_graph(obj_dir, NULL, &list,\n-\t\t\t\t flags);\n+\t\t\t\t flags, split_opts);\n \n \tstring_list_clear(&list, 0);\n \treturn result;\n@@ -1498,20 +1501,25 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \treturn 0;\n }\n \n-static int split_strategy_max_commits = 64000;\n-static float split_strategy_size_mult = 2.0f;\n-\n static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n {\n \tstruct commit_graph *g = ctx->r->objects->commit_graph;\n \tuint32_t num_commits = ctx->commits.nr;\n \tuint32_t i;\n \n+\tint max_commits = 0;\n+\tint size_mult = 2;\n+\n+\tif (ctx->split_opts) {\n+\t\tmax_commits = ctx->split_opts->max_commits;\n+\t\tsize_mult = ctx->split_opts->size_multiple;\n+\t}\n+\n \tg = ctx->r->objects->commit_graph;\n \tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n \n-\twhile (g && (g->num_commits <= split_strategy_size_mult * num_commits ||\n-\t\t num_commits > split_strategy_max_commits)) {\n+\twhile (g && (g->num_commits <= size_mult * num_commits ||\n+\t\t (max_commits && num_commits > max_commits))) {\n \t\tif (strcmp(g->obj_dir, ctx->obj_dir))\n \t\t\tbreak;\n \n@@ -1675,7 +1683,10 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \tDIR *dir;\n \tstruct dirent *de;\n \tsize_t dirnamelen;\n-\ttime_t expire_time = time(NULL);\n+\ttimestamp_t expire_time = time(NULL);\n+\n+\tif (ctx->split_opts && ctx->split_opts->expire_time)\n+\t\texpire_time -= ctx->split_opts->expire_time;\n \n \tstrbuf_addstr(&path, ctx->obj_dir);\n \tstrbuf_addstr(&path, \"/info/commit-graphs\");\n@@ -1719,7 +1730,8 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t unsigned int flags)\n+\t\t unsigned int flags,\n+\t\t const struct split_commit_graph_opts *split_opts)\n {\n \tstruct write_commit_graph_context *ctx;\n \tuint32_t i, count_distinct = 0;\n@@ -1734,6 +1746,7 @@ int write_commit_graph(const char *obj_dir,\n \tctx->append = flags & COMMIT_GRAPH_APPEND ? 1 : 0;\n \tctx->report_progress = flags & COMMIT_GRAPH_PROGRESS ? 1 : 0;\n \tctx->split = flags & COMMIT_GRAPH_SPLIT ? 1 : 0;\n+\tctx->split_opts = split_opts;\n \n \tif (ctx->split) {\n \t\tstruct commit_graph *g;\n@@ -1761,8 +1774,8 @@ int write_commit_graph(const char *obj_dir,\n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n \n-\tif (ctx->split && ctx->oids.alloc > split_strategy_max_commits)\n-\t\tctx->oids.alloc = split_strategy_max_commits;\n+\tif (ctx->split && split_opts && ctx->oids.alloc > split_opts->max_commits)\n+\t\tctx->oids.alloc = split_opts->max_commits;\n \n \tif (ctx->append) {\n \t\tprepare_commit_graph_one(ctx->r, ctx->obj_dir);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 802d35254f..a84c22a560 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -75,17 +75,25 @@ int generation_numbers_enabled(struct repository *r);\n #define COMMIT_GRAPH_PROGRESS (1 << 1)\n #define COMMIT_GRAPH_SPLIT (1 << 2)\n \n+struct split_commit_graph_opts {\n+\tint size_multiple;\n+\tint max_commits;\n+\ttimestamp_t expire_time;\n+};\n+\n /*\n * The write_commit_graph* methods return zero on success\n * and a negative value on failure. Note that if the repository\n * is not compatible with the commit-graph feature, then the\n * methods will return 0 without writing a commit-graph.\n */\n-int write_commit_graph_reachable(const char *obj_dir, unsigned int flags);\n+int write_commit_graph_reachable(const char *obj_dir, unsigned int flags,\n+\t\t\t\t const struct split_commit_graph_opts *split_opts);\n int write_commit_graph(const char *obj_dir,\n \t\t struct string_list *pack_indexes,\n \t\t struct string_list *commit_hex,\n-\t\t unsigned int flags);\n+\t\t unsigned int flags,\n+\t\t const struct split_commit_graph_opts *split_opts);\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g);\n \ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 76068ee407..1b699a543c 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -169,4 +169,51 @@ test_expect_success 'create fork and chain across alternate' '\n \n graph_git_behavior 'alternate: commit 13 vs 6' commits/13 commits/6\n \n+test_expect_success 'test merge stragety constants' '\n+\tgit clone . merge-2 &&\n+\t(\n+\t\tcd merge-2 &&\n+\t\tgit config core.commitGraph true &&\n+\t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\t\ttest_commit 14 &&\n+\t\tgit commit-graph write --reachable --split --size-multiple=2 &&\n+\t\ttest_line_count = 3 $graphdir/commit-graph-chain\n+\n+\t) &&\n+\tgit clone . merge-10 &&\n+\t(\n+\t\tcd merge-10 &&\n+\t\tgit config core.commitGraph true &&\n+\t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\t\ttest_commit 14 &&\n+\t\tgit commit-graph write --reachable --split --size-multiple=10 &&\n+\t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n+\t\tls $graphdir/graph-*.graph >graph-files &&\n+\t\ttest_line_count = 1 graph-files\n+\t) &&\n+\tgit clone . merge-10-expire &&\n+\t(\n+\t\tcd merge-10-expire &&\n+\t\tgit config core.commitGraph true &&\n+\t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\t\ttest_commit 15 &&\n+\t\tgit commit-graph write --reachable --split --size-multiple=10 --expire-time=1980-01-01 &&\n+\t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n+\t\tls $graphdir/graph-*.graph >graph-files &&\n+\t\ttest_line_count = 3 graph-files\n+\t) &&\n+\tgit clone --no-hardlinks . max-commits &&\n+\t(\n+\t\tcd max-commits &&\n+\t\tgit config core.commitGraph true &&\n+\t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n+\t\ttest_commit 16 &&\n+\t\ttest_commit 17 &&\n+\t\tgit commit-graph write --reachable --split --max-commits=1 &&\n+\t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n+\t\tls $graphdir/graph-*.graph >graph-files &&\n+\t\ttest_line_count = 1 graph-files\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"377447","messageId":"be4b83fdaf75b6a8e139f78c8db405585fe6627d.1560881661.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 16/18] commit-graph: test --split across alternate without --split","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-18T18:14:35Z","receivedAt":"2019-06-18T18:14:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWe allow sharing commit-graph files across alternates. When we are\nwriting a split commit-graph, we allow adding tip graph files that\nare not in the alternate, but include commits from our local repo.\n\nHowever, if our alternate is not using the split commit-graph format,\nits file is at .git/objects/info/commit-graph and we are trying to\nwrite files in .git/objects/info/commit-graphs/graph-{hash}.graph.\n\nWe already have logic to ensure we do not merge across alternate\nboundaries, but we also cannot have a commit-graph chain to our\nalternate if uses the old filename structure.\n\nCreate a test that verifies we create a new split commit-graph\nwith only one level and we do not modify the existing commit-graph\nin the alternate.\n\nSigned-off-by: Derrick Stolee \n---\n t/t5324-split-commit-graph.sh | 15 +++++++++++++++\n 1 file changed, 15 insertions(+)\n\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 704def70bb..130f2baf44 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -301,4 +301,19 @@ test_expect_success 'add octopus merge' '\n \n graph_git_behavior 'graph exists' merge/octopus commits/12\n \n+test_expect_success 'split across alternate where alternate is not split' '\n+\tgit commit-graph write --reachable &&\n+\ttest_path_is_file .git/objects/info/commit-graph &&\n+\tcp .git/objects/info/commit-graph . &&\n+\tgit clone --no-hardlinks . alt-split &&\n+\t(\n+\t\tcd alt-split &&\n+\t\techo \"$(pwd)\"/../.git/objects >.git/objects/info/alternates &&\n+\t\ttest_commit 18 &&\n+\t\tgit commit-graph write --reachable --split &&\n+\t\ttest_line_count = 1 $graphdir/commit-graph-chain\n+\t) &&\n+\ttest_cmp commit-graph .git/objects/info/commit-graph\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"377448","messageId":"fba62e7a42df630653092265999fb4a6e5aab81d.1560881661.git.gitgitgadget@gmail.com","threadId":"51052","inReplyTo":"pull.184.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 17/18] commit-graph: normalize commit-graph filenames","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-18T18:14:36Z","receivedAt":"2019-06-18T18:14:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nWhen writing commit-graph files, we append path data to an\nobject directory, which may be specified by the user via the\n'--object-dir' option. If the user supplies a trailing slash,\nor some other alternative path format, the resulting path may\nbe usable for writing to the correct location. However, when\nexpiring graph files from the /info/commit-graphs\ndirectory during a write, we need to compare paths with exact\nstring matches.\n\nNormalize the commit-graph filenames to avoid ambiguity. This\ncreates extra allocations, but this is a constant multiple of\nthe number of commit-graph files, which should be a number in\nthe single digits.\n\nFurther normalize the object directory in the context. Due to\na comparison between g->obj_dir and ctx->obj_dir in\nsplit_graph_merge_strategy(), a trailing slash would prevent\nany merging of layers within the same object directory. The\ncheck is there to ensure we do not merge across alternates.\nUpdate the tests to include a case with this trailing slash\nproblem.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 30 +++++++++++++++++++++++-------\n t/t5324-split-commit-graph.sh | 7 ++++++-\n 2 files changed, 29 insertions(+), 8 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 3599ae664d..c91e6f0fb8 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -43,15 +43,23 @@\n \n char *get_commit_graph_filename(const char *obj_dir)\n {\n-\treturn xstrfmt(\"%s/info/commit-graph\", obj_dir);\n+\tchar *filename = xstrfmt(\"%s/info/commit-graph\", obj_dir);\n+\tchar *normalized = xmalloc(strlen(filename) + 1);\n+\tnormalize_path_copy(normalized, filename);\n+\tfree(filename);\n+\treturn normalized;\n }\n \n static char *get_split_graph_filename(const char *obj_dir,\n \t\t\t\t const char *oid_hex)\n {\n-\treturn xstrfmt(\"%s/info/commit-graphs/graph-%s.graph\",\n-\t\t obj_dir,\n-\t\t oid_hex);\n+\tchar *filename = xstrfmt(\"%s/info/commit-graphs/graph-%s.graph\",\n+\t\t\t\t obj_dir,\n+\t\t\t\t oid_hex);\n+\tchar *normalized = xmalloc(strlen(filename) + 1);\n+\tnormalize_path_copy(normalized, filename);\n+\tfree(filename);\n+\treturn normalized;\n }\n \n static char *get_chain_filename(const char *obj_dir)\n@@ -746,7 +754,7 @@ struct packed_oid_list {\n \n struct write_commit_graph_context {\n \tstruct repository *r;\n-\tconst char *obj_dir;\n+\tchar *obj_dir;\n \tchar *graph_name;\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n@@ -1729,7 +1737,6 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \n \t\tif (!found)\n \t\t\tunlink(path.buf);\n-\n \t}\n }\n \n@@ -1741,6 +1748,7 @@ int write_commit_graph(const char *obj_dir,\n {\n \tstruct write_commit_graph_context *ctx;\n \tuint32_t i, count_distinct = 0;\n+\tsize_t len;\n \tint res = 0;\n \n \tif (!commit_graph_compatible(the_repository))\n@@ -1748,7 +1756,14 @@ int write_commit_graph(const char *obj_dir,\n \n \tctx = xcalloc(1, sizeof(struct write_commit_graph_context));\n \tctx->r = the_repository;\n-\tctx->obj_dir = obj_dir;\n+\n+\t/* normalize object dir with no trailing slash */\n+\tctx->obj_dir = xmallocz(strlen(obj_dir) + 1);\n+\tnormalize_path_copy(ctx->obj_dir, obj_dir);\n+\tlen = strlen(ctx->obj_dir);\n+\tif (len && ctx->obj_dir[len - 1] == '/')\n+\t\tctx->obj_dir[len - 1] = 0;\n+\n \tctx->append = flags & COMMIT_GRAPH_APPEND ? 1 : 0;\n \tctx->report_progress = flags & COMMIT_GRAPH_PROGRESS ? 1 : 0;\n \tctx->split = flags & COMMIT_GRAPH_SPLIT ? 1 : 0;\n@@ -1856,6 +1871,7 @@ int write_commit_graph(const char *obj_dir,\n \tfree(ctx->graph_name);\n \tfree(ctx->commits.list);\n \tfree(ctx->oids.list);\n+\tfree(ctx->obj_dir);\n \n \tif (ctx->commit_graph_filenames_after) {\n \t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++) {\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 130f2baf44..fc0d00751c 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -163,7 +163,12 @@ test_expect_success 'create fork and chain across alternate' '\n \t\ttest_line_count = 1 graph-files &&\n \t\tgit -c core.commitGraph=true rev-list HEAD >expect &&\n \t\tgit -c core.commitGraph=false rev-list HEAD >actual &&\n-\t\ttest_cmp expect actual\n+\t\ttest_cmp expect actual &&\n+\t\ttest_commit 14 &&\n+\t\tgit commit-graph write --reachable --split --object-dir=.git/objects/ &&\n+\t\ttest_line_count = 3 $graphdir/commit-graph-chain &&\n+\t\tls $graphdir/graph-*.graph >graph-files &&\n+\t\ttest_line_count = 1 graph-files\n \t)\n '\n \n-- \ngitgitgadget\n\n"}]}