{"thread":{"id":"52725","subject":"[PATCH 0/3] builtin/commit-graph.c: new split/merge options","startedAt":"2020-01-31T00:28:18Z","lastAt":"2020-02-17T18:24:43Z","messageCount":58,"participants":["Taylor Blau","Derrick Stolee","Martin Ågren","SZEDER Gábor","Johannes Schindelin","Jeff King","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"390879","messageId":"cover.1580430057.git.me@ttaylorr.com","threadId":"52725","inReplyTo":null,"subject":"[PATCH 0/3] builtin/commit-graph.c: new split/merge options","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-01-31T00:28:14Z","receivedAt":"2020-01-31T00:28:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi,\n\nHere are another few patches that came out of working on GitHub's\ndeployment of incremental commit-graphs. These three patches introduce\ntwo new options: '--split[=]' and\n'--input='.\n\nThe former controls whether or not commit-graph's split machinery should\neither write an incremental commit graph, squash the chain of\nincrementals, or defer to the other options.\n\n(This comes from GitHub's desire to have more fine-grained control over\nthe commit-graph chain's behavior. We run short jobs after every push\nthat we would like to limit the running time of, and hence we do not\nwant to ever merge a long chain of incrementals unless we specifically\nopt into that.)\n\nThe latter of the two new options does two things:\n\n * It cleans up the many options that specify input sources (e.g.,\n '--stdin-commits', '--stdin-packs', '--reachable' and so on) under\n one unifying name.\n\n * It allows us to introduce a new argument '--input=none', to prevent\n walking each packfile when neither '--stdin-commits' nor\n '--stdin-packs' was given.\n\nTogether, these have the combined effect of being able to write the\nfollowing two new invocations:\n\n $ git commit-graph write --split=merge-all --input=none\n\n $ git commit-graph write --split=no-merge --input=stdin-packs\n\nto (1) merge the chain, and (2) write a single new incremental.\n\nThanks in advance for your review, as always.\n\nTaylor Blau (3):\n builtin/commit-graph.c: support '--split[=]'\n builtin/commit-graph.c: introduce '--input='\n builtin/commit-graph.c: support '--input=none'\n\n Documentation/git-commit-graph.txt | 55 ++++++++------\n builtin/commit-graph.c | 113 +++++++++++++++++++++++------\n commit-graph.c | 25 ++++---\n commit-graph.h | 10 ++-\n t/t5318-commit-graph.sh | 46 ++++++------\n t/t5324-split-commit-graph.sh | 85 +++++++++++++++++-----\n 6 files changed, 239 insertions(+), 95 deletions(-)\n\n--\n2.25.0.dirty\n"},{"id":"390880","messageId":"4f5bc19903f8a1f5b153b5665de378e743e12744.1580430057.git.me@ttaylorr.com","threadId":"52725","inReplyTo":"cover.1580430057.git.me@ttaylorr.com","subject":"[PATCH 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-01-31T00:28:17Z","receivedAt":"2020-01-31T00:28:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"With '--split', the commit-graph machinery writes new commits in another\nincremental commit-graph which is part of the existing chain, and\noptionally decides to condense the chain into a single commit-graph.\nThis is done to ensure that the aysmptotic behavior of looking up a\ncommit in an incremental chain is dominated by the number of\nincrementals in that chain. It can be controlled by the '--max-commits'\nand '--size-multiple' options.\n\nOn occasion, callers may want to ensure that 'git commit-graph write\n--split' always writes an incremental, and never spends effort\ncondensing the incremental chain [1]. Previously, this was possible by\npassing '--size-multiple=0', but this no longer the case following\n63020f175f (commit-graph: prefer default size_mult when given zero,\n2020-01-02).\n\nReintroduce a less-magical variant of the above with a new pair of\narguments to '--split': '--split=no-merge' and '--split=merge-all'. When\n'--split=no-merge' is given, the commit-graph machinery will never\ncondense an existing chain and will always write a new incremental.\nConversely, if '--split=merge-all' is given, any invocation including it\nwill always condense a chain if one exists. If '--split' is given with\nno arguments, it behaves as before and defers to '--size-multiple', and\nso on.\n\n[1]: This might occur when, for example, a server administrator running\nsome program after each push may want to ensure that each job runs\nproportional in time to the size of the push, and does not \"jump\" when\nthe commit-graph machinery decides to trigger a merge.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-commit-graph.txt | 18 +++++++++++-----\n builtin/commit-graph.c | 33 ++++++++++++++++++++++++++----\n commit-graph.c | 19 +++++++++--------\n commit-graph.h | 7 +++++++\n t/t5324-split-commit-graph.sh | 25 ++++++++++++++++++++++\n 5 files changed, 85 insertions(+), 17 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 28d1fee505..8d61ba9f56 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -57,11 +57,19 @@ or `--stdin-packs`.)\n With the `--append` option, include all commits that are present in the\n existing commit-graph file.\n +\n-With the `--split` option, write the commit-graph as a chain of multiple\n-commit-graph files stored in `/info/commit-graphs`. The new commits\n-not already in the commit-graph are added in a new \"tip\" file. This file\n-is merged with the existing file if the following merge conditions are\n-met:\n+With the `--split[=]` option, write the commit-graph as a\n+chain of multiple commit-graph files stored in\n+`/info/commit-graphs`. Commit-graph layers are merged based on the\n+strategy and other splitting options. The new commits not already in the\n+commit-graph are added in a new \"tip\" file. This file is merged with the\n+existing file if the following merge conditions are met:\n+* If `--split=merge-always` is specified, then a merge is always\n+conducted, and the remaining options are ignored. Conversely, if\n+`--split=no-merge` is specified, a merge is never performed, and the\n+remaining options are ignored. A bare `--split` defers to the remaining\n+options. (Note that merging a chain of commit graphs replaces the\n+existing chain with a length-1 chain where the first and only\n+incremental holds the entire graph).\n +\n * If `--size-multiple=` is not specified, let `X` equal 2. If the new\n tip file would have `N` commits and the previous tip has `M` commits and\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex de321c71ad..f03b46d627 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -9,7 +9,9 @@\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] [--[no-]progress] \"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] \"\n+\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n+\t \"[--[no-]progress] \"),\n \tNULL\n };\n \n@@ -19,7 +21,9 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] [--[no-]progress] \"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] \"\n+\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n+\t \"[--[no-]progress] \"),\n \tNULL\n };\n \n@@ -101,6 +105,25 @@ static int graph_verify(int argc, const char **argv)\n extern int read_replace_refs;\n static struct split_commit_graph_opts split_opts;\n \n+static int write_option_parse_split(const struct option *opt, const char *arg,\n+\t\t\t\t int unset)\n+{\n+\tenum commit_graph_split_flags *flags = opt->value;\n+\n+\topts.split = 1;\n+\tif (!arg)\n+\t\treturn 0;\n+\n+\tif (!strcmp(arg, \"merge-all\"))\n+\t\t*flags = COMMIT_GRAPH_SPLIT_MERGE_REQUIRED;\n+\telse if (!strcmp(arg, \"no-merge\"))\n+\t\t*flags = COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED;\n+\telse\n+\t\tdie(_(\"unrecognized --split argument, %s\"), arg);\n+\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tstruct string_list *pack_indexes = NULL;\n@@ -123,8 +146,10 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_BOOL(0, \"append\", &opts.append,\n \t\t\tN_(\"include all commits already in the commit-graph file\")),\n \t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n-\t\tOPT_BOOL(0, \"split\", &opts.split,\n-\t\t\tN_(\"allow writing an incremental commit-graph file\")),\n+\t\tOPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n+\t\t\tN_(\"allow writing an incremental commit-graph file\"),\n+\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n+\t\t\twrite_option_parse_split),\n \t\tOPT_INTEGER(0, \"max-commits\", &split_opts.max_commits,\n \t\t\tN_(\"maximum number of commits in a non-base split commit-graph\")),\n \t\tOPT_INTEGER(0, \"size-multiple\", &split_opts.size_multiple,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 6d34829f57..02e6ad9d1f 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1565,15 +1565,18 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \tnum_commits = ctx->commits.nr;\n \tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n \n-\twhile (g && (g->num_commits <= size_mult * num_commits ||\n-\t\t (max_commits && num_commits > max_commits))) {\n-\t\tif (g->odb != ctx->odb)\n-\t\t\tbreak;\n+\tif (ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n+\t\twhile (g && (g->num_commits <= size_mult * num_commits ||\n+\t\t\t (max_commits && num_commits > max_commits) ||\n+\t\t\t (ctx->split_opts->flags == COMMIT_GRAPH_SPLIT_MERGE_REQUIRED))) {\n+\t\t\tif (g->odb != ctx->odb)\n+\t\t\t\tbreak;\n \n-\t\tnum_commits += g->num_commits;\n-\t\tg = g->base_graph;\n+\t\t\tnum_commits += g->num_commits;\n+\t\t\tg = g->base_graph;\n \n-\t\tctx->num_commit_graphs_after--;\n+\t\t\tctx->num_commit_graphs_after--;\n+\t\t}\n \t}\n \n \tctx->new_base_graph = g;\n@@ -1881,7 +1884,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\tgoto cleanup;\n \t}\n \n-\tif (!ctx->commits.nr)\n+\tif (!ctx->commits.nr && (!ctx->split || ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_REQUIRED))\n \t\tgoto cleanup;\n \n \tif (ctx->split) {\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 7d9fc9c16a..dadcc03808 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -84,10 +84,17 @@ enum commit_graph_write_flags {\n \tCOMMIT_GRAPH_WRITE_CHECK_OIDS = (1 << 3)\n };\n \n+enum commit_graph_split_flags {\n+\tCOMMIT_GRAPH_SPLIT_UNSPECIFIED = 0,\n+\tCOMMIT_GRAPH_SPLIT_MERGE_REQUIRED = 1,\n+\tCOMMIT_GRAPH_SPLIT_MERGE_PROHIBITED = 2\n+};\n+\n struct split_commit_graph_opts {\n \tint size_multiple;\n \tint max_commits;\n \ttimestamp_t expire_time;\n+\tenum commit_graph_split_flags flags;\n };\n \n /*\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex c24823431f..a165b48afe 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -344,4 +344,29 @@ test_expect_success 'split across alternate where alternate is not split' '\n \ttest_cmp commit-graph .git/objects/info/commit-graph\n '\n \n+test_expect_success '--split=merge-all always merges incrementals' '\n+\ttest_when_finished rm -rf a b c &&\n+\trm -rf $graphdir $infodir/commit-graph &&\n+\tgit reset --hard commits/10 &&\n+\tgit rev-list -3 HEAD~4 >a &&\n+\tgit rev-list -2 HEAD~2 >b &&\n+\tgit rev-list -2 HEAD >c &&\n+\tgit commit-graph write --split=no-merge --stdin-commits a &&\n+\tgit rev-list HEAD >b &&\n+\tgit commit-graph write --split --stdin-commits '","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-01-31T00:28:19Z","receivedAt":"2020-01-31T00:28:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The 'write' mode of the 'commit-graph' supports input from a number of\ndifferent sources: pack indexes over stdin, commits over stdin, commits\nreachable from all references, and so on. Each of these options are\nspecified with a unique option: '--stdin-packs', '--stdin-commits', etc.\n\nSimilar to our replacement of 'git config [--]' with 'git config\n[--type=]' (c.f., fb0dc3bac1 (builtin/config.c: support\n`--type=` as preferred alias for `--`, 2018-04-18)), softly\ndeprecate '[--]' in favor of '[--input=]'.\n\nThis makes it more clear to implement new options that are combinations\nof other options (such as, for example, \"none\", a combination of the old\n\"--append\" and a new sentinel to specify to _not_ look in other packs,\nwhich we will implement in a future patch).\n\nUnfortunately, the new enumerated type is a bitfield, even though it\nmakes much more sense as '0, 1, 2, ...'. Even though *almost* all\noptions are pairwise exclusive, '--stdin-{packs,commits}' *is*\ncompatible with '--append'. For this reason, use a bitfield.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-commit-graph.txt | 26 +++++-----\n builtin/commit-graph.c | 77 ++++++++++++++++++++++--------\n t/t5318-commit-graph.sh | 46 +++++++++---------\n t/t5324-split-commit-graph.sh | 44 ++++++++---------\n 4 files changed, 114 insertions(+), 79 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 8d61ba9f56..cbf80226e9 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -41,21 +41,21 @@ COMMANDS\n \n Write a commit-graph file based on the commits found in packfiles.\n +\n-With the `--stdin-packs` option, generate the new commit graph by\n+With the `--input=stdin-packs` option, generate the new commit graph by\n walking objects only in the specified pack-indexes. (Cannot be combined\n-with `--stdin-commits` or `--reachable`.)\n+with `--input=stdin-commits` or `--input=reachable`.)\n +\n-With the `--stdin-commits` option, generate the new commit graph by\n-walking commits starting at the commits specified in stdin as a list\n+With the `--input=stdin-commits` option, generate the new commit graph\n+by walking commits starting at the commits specified in stdin as a list\n of OIDs in hex, one OID per line. (Cannot be combined with\n-`--stdin-packs` or `--reachable`.)\n+`--input=stdin-packs` or `--input=reachable`.)\n +\n-With the `--reachable` option, generate the new commit graph by walking\n-commits starting at all refs. (Cannot be combined with `--stdin-commits`\n-or `--stdin-packs`.)\n+With the `--input=reachable` option, generate the new commit graph by\n+walking commits starting at all refs. (Cannot be combined with\n+`--input=stdin-commits` or `--input=stdin-packs`.)\n +\n-With the `--append` option, include all commits that are present in the\n-existing commit-graph file.\n+With the `--input=append` option, include all commits that are present\n+in the existing commit-graph file.\n +\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n@@ -107,20 +107,20 @@ $ git commit-graph write\n using commits in ``.\n +\n ------------------------------------------------\n-$ echo | git commit-graph write --stdin-packs\n+$ echo | git commit-graph write --input=stdin-packs\n ------------------------------------------------\n \n * Write a commit-graph file containing all reachable commits.\n +\n ------------------------------------------------\n-$ git show-ref -s | git commit-graph write --stdin-commits\n+$ git show-ref -s | git commit-graph write --input=stdin-commits\n ------------------------------------------------\n \n * Write a commit-graph file containing all commits in the current\n commit-graph file along with those reachable from `HEAD`.\n +\n ------------------------------------------------\n-$ git rev-parse HEAD | git commit-graph write --stdin-commits --append\n+$ git rev-parse HEAD | git commit-graph write --input=stdin-commits --input=append\n ------------------------------------------------\n \n \ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex f03b46d627..03d815e652 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -10,7 +10,7 @@\n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n-\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n+\t \"[--split[=]] [--input=] \"\n \t \"[--[no-]progress] \"),\n \tNULL\n };\n@@ -22,22 +22,48 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n \n static const char * const builtin_commit_graph_write_usage[] = {\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n-\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n+\t \"[--split[=]] [--input=] \"\n \t \"[--[no-]progress] \"),\n \tNULL\n };\n \n+enum commit_graph_input {\n+\tCOMMIT_GRAPH_INPUT_REACHABLE = (1 << 1),\n+\tCOMMIT_GRAPH_INPUT_STDIN_PACKS = (1 << 2),\n+\tCOMMIT_GRAPH_INPUT_STDIN_COMMITS = (1 << 3),\n+\tCOMMIT_GRAPH_INPUT_APPEND = (1 << 4)\n+};\n+\n static struct opts_commit_graph {\n \tconst char *obj_dir;\n-\tint reachable;\n-\tint stdin_packs;\n-\tint stdin_commits;\n-\tint append;\n+\tenum commit_graph_input input;\n \tint split;\n \tint shallow;\n \tint progress;\n } opts;\n \n+static int option_parse_input(const struct option *opt, const char *arg,\n+\t\t\t int unset)\n+{\n+\tenum commit_graph_input *to = opt->value;\n+\tif (unset || !strcmp(arg, \"packs\")) {\n+\t\t*to = 0;\n+\t\treturn 0;\n+\t}\n+\n+\tif (!strcmp(arg, \"reachable\"))\n+\t\t*to |= COMMIT_GRAPH_INPUT_REACHABLE;\n+\telse if (!strcmp(arg, \"stdin-packs\"))\n+\t\t*to |= COMMIT_GRAPH_INPUT_STDIN_PACKS;\n+\telse if (!strcmp(arg, \"stdin-commits\"))\n+\t\t*to |= COMMIT_GRAPH_INPUT_STDIN_COMMITS;\n+\telse if (!strcmp(arg, \"append\"))\n+\t\t*to |= COMMIT_GRAPH_INPUT_APPEND;\n+\telse\n+\t\tdie(_(\"unrecognized --input source, %s\"), arg);\n+\treturn 0;\n+}\n+\n static struct object_directory *find_odb_or_die(struct repository *r,\n \t\t\t\t\t\tconst char *obj_dir)\n {\n@@ -137,14 +163,21 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n \t\t\tN_(\"dir\"),\n \t\t\tN_(\"The object directory to store the graph\")),\n-\t\tOPT_BOOL(0, \"reachable\", &opts.reachable,\n-\t\t\tN_(\"start walk at all refs\")),\n-\t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n-\t\t\tN_(\"scan pack-indexes listed by stdin for commits\")),\n-\t\tOPT_BOOL(0, \"stdin-commits\", &opts.stdin_commits,\n-\t\t\tN_(\"start walk at commits listed by stdin\")),\n-\t\tOPT_BOOL(0, \"append\", &opts.append,\n-\t\t\tN_(\"include all commits already in the commit-graph file\")),\n+\t\tOPT_CALLBACK(0, \"input\", &opts.input, NULL,\n+\t\t\tN_(\"include commits from this source in the graph\"),\n+\t\t\toption_parse_input),\n+\t\tOPT_BIT(0, \"reachable\", &opts.input,\n+\t\t\tN_(\"start walk at all refs\"),\n+\t\t\tCOMMIT_GRAPH_INPUT_REACHABLE),\n+\t\tOPT_BIT(0, \"stdin-packs\", &opts.input,\n+\t\t\tN_(\"scan pack-indexes listed by stdin for commits\"),\n+\t\t\tCOMMIT_GRAPH_INPUT_STDIN_PACKS),\n+\t\tOPT_BIT(0, \"stdin-commits\", &opts.input,\n+\t\t\tN_(\"start walk at commits listed by stdin\"),\n+\t\t\tCOMMIT_GRAPH_INPUT_STDIN_COMMITS),\n+\t\tOPT_BIT(0, \"append\", &opts.input,\n+\t\t\tN_(\"include all commits already in the commit-graph file\"),\n+\t\t\tCOMMIT_GRAPH_INPUT_APPEND),\n \t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n \t\tOPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n \t\t\tN_(\"allow writing an incremental commit-graph file\"),\n@@ -170,11 +203,13 @@ static int graph_write(int argc, const char **argv)\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\n \n-\tif (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)\n-\t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\n+\tif ((!!(opts.input & COMMIT_GRAPH_INPUT_REACHABLE) +\n+\t !!(opts.input & COMMIT_GRAPH_INPUT_STDIN_PACKS) +\n+\t !!(opts.input & COMMIT_GRAPH_INPUT_STDIN_COMMITS)) > 1)\n+\t\tdie(_(\"use at most one of --input=reachable, --input=stdin-commits, or --input=stdin-packs\"));\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n-\tif (opts.append)\n+\tif (opts.input & COMMIT_GRAPH_INPUT_APPEND)\n \t\tflags |= COMMIT_GRAPH_WRITE_APPEND;\n \tif (opts.split)\n \t\tflags |= COMMIT_GRAPH_WRITE_SPLIT;\n@@ -184,22 +219,22 @@ static int graph_write(int argc, const char **argv)\n \tread_replace_refs = 0;\n \todb = find_odb_or_die(the_repository, opts.obj_dir);\n \n-\tif (opts.reachable) {\n+\tif (opts.input & COMMIT_GRAPH_INPUT_REACHABLE) {\n \t\tif (write_commit_graph_reachable(odb, flags, &split_opts))\n \t\t\treturn 1;\n \t\treturn 0;\n \t}\n \n \tstring_list_init(&lines, 0);\n-\tif (opts.stdin_packs || opts.stdin_commits) {\n+\tif (opts.input & (COMMIT_GRAPH_INPUT_STDIN_PACKS | COMMIT_GRAPH_INPUT_STDIN_COMMITS)) {\n \t\tstruct strbuf buf = STRBUF_INIT;\n \n \t\twhile (strbuf_getline(&buf, stdin) != EOF)\n \t\t\tstring_list_append(&lines, strbuf_detach(&buf, NULL));\n \n-\t\tif (opts.stdin_packs)\n+\t\tif (opts.input & COMMIT_GRAPH_INPUT_STDIN_PACKS)\n \t\t\tpack_indexes = &lines;\n-\t\tif (opts.stdin_commits) {\n+\t\tif (opts.input & COMMIT_GRAPH_INPUT_STDIN_COMMITS) {\n \t\t\tcommit_hex = &lines;\n \t\t\tflags |= COMMIT_GRAPH_WRITE_CHECK_OIDS;\n \t\t}\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 0bf98b56ec..6533724bc5 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -23,10 +23,10 @@ test_expect_success 'write graph with no packs' '\n \ttest_path_is_missing $objdir/info/commit-graph\n '\n \n-test_expect_success 'exit with correct error on bad input to --stdin-packs' '\n+test_expect_success 'exit with correct error on bad input to --input=stdin-packs' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \techo doesnotexist >in &&\n-\ttest_expect_code 1 git commit-graph write --stdin-packs stderr &&\n+\ttest_expect_code 1 git commit-graph write --input=stdin-packs stderr &&\n \ttest_i18ngrep \"error adding pack\" stderr\n '\n \n@@ -40,12 +40,12 @@ test_expect_success 'create commits and repack' '\n \tgit repack\n '\n \n-test_expect_success 'exit with correct error on bad input to --stdin-commits' '\n+test_expect_success 'exit with correct error on bad input to --input=stdin-commits' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n-\techo HEAD | test_expect_code 1 git commit-graph write --stdin-commits 2>stderr &&\n+\techo HEAD | test_expect_code 1 git commit-graph write --input=stdin-commits 2>stderr &&\n \ttest_i18ngrep \"invalid commit object id\" stderr &&\n \t# valid tree OID, but not a commit OID\n-\tgit rev-parse HEAD^{tree} | test_expect_code 1 git commit-graph write --stdin-commits 2>stderr &&\n+\tgit rev-parse HEAD^{tree} | test_expect_code 1 git commit-graph write --input=stdin-commits 2>stderr &&\n \ttest_i18ngrep \"invalid commit object id\" stderr\n '\n \n@@ -227,7 +227,7 @@ graph_git_behavior 'cleared graph, commit 8 vs merge 2' full commits/8 merge/2\n \n test_expect_success 'build graph from latest pack with closure' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n-\tcat new-idx | git commit-graph write --stdin-packs &&\n+\tcat new-idx | git commit-graph write --input=stdin-packs &&\n \ttest_path_is_file $objdir/info/commit-graph &&\n \tgraph_read_expect \"9\" \"extra_edges\"\n '\n@@ -240,7 +240,7 @@ test_expect_success 'build graph from commits with closure' '\n \tgit tag -a -m \"merge\" tag/merge merge/2 &&\n \tgit rev-parse tag/merge >commits-in &&\n \tgit rev-parse merge/1 >>commits-in &&\n-\tcat commits-in | git commit-graph write --stdin-commits &&\n+\tcat commits-in | git commit-graph write --input=stdin-commits &&\n \ttest_path_is_file $objdir/info/commit-graph &&\n \tgraph_read_expect \"6\"\n '\n@@ -250,7 +250,7 @@ graph_git_behavior 'graph from commits, commit 8 vs merge 2' full commits/8 merg\n \n test_expect_success 'build graph from commits with append' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n-\tgit rev-parse merge/3 | git commit-graph write --stdin-commits --append &&\n+\tgit rev-parse merge/3 | git commit-graph write --input=stdin-commits --input=append &&\n \ttest_path_is_file $objdir/info/commit-graph &&\n \tgraph_read_expect \"10\" \"extra_edges\"\n '\n@@ -260,7 +260,7 @@ graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n \n test_expect_success 'build graph using --reachable' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n-\tgit commit-graph write --reachable &&\n+\tgit commit-graph write --input=reachable &&\n \ttest_path_is_file $objdir/info/commit-graph &&\n \tgraph_read_expect \"11\" \"extra_edges\"\n '\n@@ -301,14 +301,14 @@ test_expect_success 'perform fast-forward merge in full repo' '\n test_expect_success 'check that gc computes commit-graph' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n \tgit commit --allow-empty -m \"blank\" &&\n-\tgit commit-graph write --reachable &&\n+\tgit commit-graph write --input=reachable &&\n \tcp $objdir/info/commit-graph commit-graph-before-gc &&\n \tgit reset --hard HEAD~1 &&\n \tgit config gc.writeCommitGraph true &&\n \tgit gc &&\n \tcp $objdir/info/commit-graph commit-graph-after-gc &&\n \t! test_cmp_bin commit-graph-before-gc commit-graph-after-gc &&\n-\tgit commit-graph write --reachable &&\n+\tgit commit-graph write --input=reachable &&\n \ttest_cmp_bin commit-graph-after-gc $objdir/info/commit-graph\n '\n \n@@ -318,18 +318,18 @@ test_expect_success 'replace-objects invalidates commit-graph' '\n \tgit clone full replace &&\n \t(\n \t\tcd replace &&\n-\t\tgit commit-graph write --reachable &&\n+\t\tgit commit-graph write --input=reachable &&\n \t\ttest_path_is_file .git/objects/info/commit-graph &&\n \t\tgit replace HEAD~1 HEAD~2 &&\n \t\tgit -c core.commitGraph=false log >expect &&\n \t\tgit -c core.commitGraph=true log >actual &&\n \t\ttest_cmp expect actual &&\n-\t\tgit commit-graph write --reachable &&\n+\t\tgit commit-graph write --input=reachable &&\n \t\tgit -c core.commitGraph=false --no-replace-objects log >expect &&\n \t\tgit -c core.commitGraph=true --no-replace-objects log >actual &&\n \t\ttest_cmp expect actual &&\n \t\trm -rf .git/objects/info/commit-graph &&\n-\t\tgit commit-graph write --reachable &&\n+\t\tgit commit-graph write --input=reachable &&\n \t\ttest_path_is_file .git/objects/info/commit-graph\n \t)\n '\n@@ -340,7 +340,7 @@ test_expect_success 'commit grafts invalidate commit-graph' '\n \tgit clone full graft &&\n \t(\n \t\tcd graft &&\n-\t\tgit commit-graph write --reachable &&\n+\t\tgit commit-graph write --input=reachable &&\n \t\ttest_path_is_file .git/objects/info/commit-graph &&\n \t\tH1=$(git rev-parse --verify HEAD~1) &&\n \t\tH3=$(git rev-parse --verify HEAD~3) &&\n@@ -348,12 +348,12 @@ test_expect_success 'commit grafts invalidate commit-graph' '\n \t\tgit -c core.commitGraph=false log >expect &&\n \t\tgit -c core.commitGraph=true log >actual &&\n \t\ttest_cmp expect actual &&\n-\t\tgit commit-graph write --reachable &&\n+\t\tgit commit-graph write --input=reachable &&\n \t\tgit -c core.commitGraph=false --no-replace-objects log >expect &&\n \t\tgit -c core.commitGraph=true --no-replace-objects log >actual &&\n \t\ttest_cmp expect actual &&\n \t\trm -rf .git/objects/info/commit-graph &&\n-\t\tgit commit-graph write --reachable &&\n+\t\tgit commit-graph write --input=reachable &&\n \t\ttest_path_is_missing .git/objects/info/commit-graph\n \t)\n '\n@@ -364,10 +364,10 @@ test_expect_success 'replace-objects invalidates commit-graph' '\n \tgit clone --depth 2 \"file://$TRASH_DIRECTORY/full\" shallow &&\n \t(\n \t\tcd shallow &&\n-\t\tgit commit-graph write --reachable &&\n+\t\tgit commit-graph write --input=reachable &&\n \t\ttest_path_is_missing .git/objects/info/commit-graph &&\n \t\tgit fetch origin --unshallow &&\n-\t\tgit commit-graph write --reachable &&\n+\t\tgit commit-graph write --input=reachable &&\n \t\ttest_path_is_file .git/objects/info/commit-graph\n \t)\n '\n@@ -380,7 +380,7 @@ test_expect_success 'replace-objects invalidates commit-graph' '\n \n test_expect_success 'git commit-graph verify' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n-\tgit rev-parse commits/8 | git commit-graph write --stdin-commits &&\n+\tgit rev-parse commits/8 | git commit-graph write --input=stdin-commits &&\n \tgit commit-graph verify >output\n '\n \n@@ -591,7 +591,7 @@ test_expect_success 'setup non-the_repository tests' '\n \ttest_commit -C repo two &&\n \tgit -C repo config core.commitGraph true &&\n \tgit -C repo rev-parse two | \\\n-\t\tgit -C repo commit-graph write --stdin-commits\n+\t\tgit -C repo commit-graph write --input=stdin-commits\n '\n \n test_expect_success 'parse_commit_in_graph works for non-the_repository' '\n@@ -637,7 +637,7 @@ test_expect_success 'corrupt commit-graph write (broken parent)' '\n \t\tEOF\n \t\tbroken=\"$(git hash-object -w -t commit --literally broken)\" &&\n \t\tgit commit-tree -p \"$broken\" -m \"good commit\" \"$empty\" >good &&\n-\t\ttest_must_fail git commit-graph write --stdin-commits \\\n+\t\ttest_must_fail git commit-graph write --input=stdin-commits \\\n \t\t\ttest_err &&\n \t\ttest_i18ngrep \"unable to parse commit\" test_err\n \t)\n@@ -658,7 +658,7 @@ test_expect_success 'corrupt commit-graph write (missing tree)' '\n \t\tEOF\n \t\tbroken=\"$(git hash-object -w -t commit --literally broken)\" &&\n \t\tgit commit-tree -p \"$broken\" -m \"good\" \"$tree\" >good &&\n-\t\ttest_must_fail git commit-graph write --stdin-commits \\\n+\t\ttest_must_fail git commit-graph write --input=stdin-commits \\\n \t\t\ttest_err &&\n \t\ttest_i18ngrep \"unable to parse commit\" test_err\n \t)\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex a165b48afe..dd74295885 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -35,7 +35,7 @@ test_expect_success 'create commits and write commit-graph' '\n \t\ttest_commit $i &&\n \t\tgit branch commits/$i || return 1\n \tdone &&\n-\tgit commit-graph write --reachable &&\n+\tgit commit-graph write --input=reachable &&\n \ttest_path_is_file $infodir/commit-graph &&\n \tgraph_read_expect 3\n '\n@@ -87,7 +87,7 @@ test_expect_success 'add more commits, and write a new base graph' '\n \tgit reset --hard commits/4 &&\n \tgit merge commits/6 &&\n \tgit branch merge/2 &&\n-\tgit commit-graph write --reachable &&\n+\tgit commit-graph write --input=reachable &&\n \tgraph_read_expect 12\n '\n \n@@ -99,7 +99,7 @@ test_expect_success 'fork and fail to base a chain on a commit-graph file' '\n \t\trm .git/objects/info/commit-graph &&\n \t\techo \"$(pwd)/../.git/objects\" >.git/objects/info/alternates &&\n \t\ttest_commit new-commit &&\n-\t\tgit commit-graph write --reachable --split &&\n+\t\tgit commit-graph write --input=reachable --split &&\n \t\ttest_path_is_file $graphdir/commit-graph-chain &&\n \t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n \t\tverify_chain_files_exist $graphdir\n@@ -112,7 +112,7 @@ test_expect_success 'add three more commits, write a tip graph' '\n \tgit merge commits/5 &&\n \tgit merge merge/2 &&\n \tgit branch merge/3 &&\n-\tgit commit-graph write --reachable --split &&\n+\tgit commit-graph write --input=reachable --split &&\n \ttest_path_is_missing $infodir/commit-graph &&\n \ttest_path_is_file $graphdir/commit-graph-chain &&\n \tls $graphdir/graph-*.graph >graph-files &&\n@@ -125,7 +125,7 @@ graph_git_behavior 'split commit-graph: merge 3 vs 2' merge/3 merge/2\n test_expect_success 'add one commit, write a tip graph' '\n \ttest_commit 11 &&\n \tgit branch commits/11 &&\n-\tgit commit-graph write --reachable --split &&\n+\tgit commit-graph write --input=reachable --split &&\n \ttest_path_is_missing $infodir/commit-graph &&\n \ttest_path_is_file $graphdir/commit-graph-chain &&\n \tls $graphdir/graph-*.graph >graph-files &&\n@@ -138,7 +138,7 @@ graph_git_behavior 'three-layer commit-graph: commit 11 vs 6' commits/11 commits\n test_expect_success 'add one commit, write a merged graph' '\n \ttest_commit 12 &&\n \tgit branch commits/12 &&\n-\tgit commit-graph write --reachable --split &&\n+\tgit commit-graph write --input=reachable --split &&\n \ttest_path_is_file $graphdir/commit-graph-chain &&\n \ttest_line_count = 2 $graphdir/commit-graph-chain &&\n \tls $graphdir/graph-*.graph >graph-files &&\n@@ -157,7 +157,7 @@ test_expect_success 'create fork and chain across alternate' '\n \t\techo \"$(pwd)/../.git/objects\" >.git/objects/info/alternates &&\n \t\ttest_commit 13 &&\n \t\tgit branch commits/13 &&\n-\t\tgit commit-graph write --reachable --split &&\n+\t\tgit commit-graph write --input=reachable --split &&\n \t\ttest_path_is_file $graphdir/commit-graph-chain &&\n \t\ttest_line_count = 3 $graphdir/commit-graph-chain &&\n \t\tls $graphdir/graph-*.graph >graph-files &&\n@@ -166,7 +166,7 @@ test_expect_success 'create fork and chain across alternate' '\n \t\tgit -c core.commitGraph=false rev-list HEAD >actual &&\n \t\ttest_cmp expect actual &&\n \t\ttest_commit 14 &&\n-\t\tgit commit-graph write --reachable --split --object-dir=.git/objects/ &&\n+\t\tgit commit-graph write --input=reachable --split --object-dir=.git/objects/ &&\n \t\ttest_line_count = 3 $graphdir/commit-graph-chain &&\n \t\tls $graphdir/graph-*.graph >graph-files &&\n \t\ttest_line_count = 1 graph-files\n@@ -182,7 +182,7 @@ test_expect_success 'test merge stragety constants' '\n \t\tgit config core.commitGraph true &&\n \t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n \t\ttest_commit 14 &&\n-\t\tgit commit-graph write --reachable --split --size-multiple=2 &&\n+\t\tgit commit-graph write --input=reachable --split --size-multiple=2 &&\n \t\ttest_line_count = 3 $graphdir/commit-graph-chain\n \n \t) &&\n@@ -192,7 +192,7 @@ test_expect_success 'test merge stragety constants' '\n \t\tgit config core.commitGraph true &&\n \t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n \t\ttest_commit 14 &&\n-\t\tgit commit-graph write --reachable --split --size-multiple=10 &&\n+\t\tgit commit-graph write --input=reachable --split --size-multiple=10 &&\n \t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n \t\tls $graphdir/graph-*.graph >graph-files &&\n \t\ttest_line_count = 1 graph-files\n@@ -203,7 +203,7 @@ test_expect_success 'test merge stragety constants' '\n \t\tgit config core.commitGraph true &&\n \t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n \t\ttest_commit 15 &&\n-\t\tgit commit-graph write --reachable --split --size-multiple=10 --expire-time=1980-01-01 &&\n+\t\tgit commit-graph write --input=reachable --split --size-multiple=10 --expire-time=1980-01-01 &&\n \t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n \t\tls $graphdir/graph-*.graph >graph-files &&\n \t\ttest_line_count = 3 graph-files\n@@ -215,7 +215,7 @@ test_expect_success 'test merge stragety constants' '\n \t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n \t\ttest_commit 16 &&\n \t\ttest_commit 17 &&\n-\t\tgit commit-graph write --reachable --split --max-commits=1 &&\n+\t\tgit commit-graph write --input=reachable --split --max-commits=1 &&\n \t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n \t\tls $graphdir/graph-*.graph >graph-files &&\n \t\ttest_line_count = 1 graph-files\n@@ -227,7 +227,7 @@ test_expect_success 'remove commit-graph-chain file after flattening' '\n \t(\n \t\tcd flatten &&\n \t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n-\t\tgit commit-graph write --reachable &&\n+\t\tgit commit-graph write --input=reachable &&\n \t\ttest_path_is_missing $graphdir/commit-graph-chain &&\n \t\tls $graphdir >graph-files &&\n \t\ttest_line_count = 0 graph-files\n@@ -306,7 +306,7 @@ test_expect_success 'verify across alternates' '\n \t\techo \"$altdir\" >.git/objects/info/alternates &&\n \t\tgit commit-graph verify --object-dir=\"$altdir/\" &&\n \t\ttest_commit extra &&\n-\t\tgit commit-graph write --reachable --split &&\n+\t\tgit commit-graph write --input=reachable --split &&\n \t\ttip_file=$graphdir/graph-$(tail -n 1 $graphdir/commit-graph-chain).graph &&\n \t\tcorrupt_file \"$tip_file\" 100 \"\\01\" &&\n \t\ttest_must_fail git commit-graph verify --shallow 2>test_err &&\n@@ -319,7 +319,7 @@ test_expect_success 'add octopus merge' '\n \tgit reset --hard commits/10 &&\n \tgit merge commits/3 commits/4 &&\n \tgit branch merge/octopus &&\n-\tgit commit-graph write --reachable --split &&\n+\tgit commit-graph write --input=reachable --split &&\n \tgit commit-graph verify --progress 2>err &&\n \ttest_line_count = 3 err &&\n \ttest_i18ngrep ! warning err &&\n@@ -329,7 +329,7 @@ test_expect_success 'add octopus merge' '\n graph_git_behavior 'graph exists' merge/octopus commits/12\n \n test_expect_success 'split across alternate where alternate is not split' '\n-\tgit commit-graph write --reachable &&\n+\tgit commit-graph write --input=reachable &&\n \ttest_path_is_file .git/objects/info/commit-graph &&\n \tcp .git/objects/info/commit-graph . &&\n \tgit clone --no-hardlinks . alt-split &&\n@@ -338,7 +338,7 @@ test_expect_success 'split across alternate where alternate is not split' '\n \t\trm -f .git/objects/info/commit-graph &&\n \t\techo \"$(pwd)\"/../.git/objects >.git/objects/info/alternates &&\n \t\ttest_commit 18 &&\n-\t\tgit commit-graph write --reachable --split &&\n+\t\tgit commit-graph write --input=reachable --split &&\n \t\ttest_line_count = 1 $graphdir/commit-graph-chain\n \t) &&\n \ttest_cmp commit-graph .git/objects/info/commit-graph\n@@ -351,10 +351,10 @@ test_expect_success '--split=merge-all always merges incrementals' '\n \tgit rev-list -3 HEAD~4 >a &&\n \tgit rev-list -2 HEAD~2 >b &&\n \tgit rev-list -2 HEAD >c &&\n-\tgit commit-graph write --split=no-merge --stdin-commits a &&\n \tgit rev-list HEAD >b &&\n-\tgit commit-graph write --split --stdin-commits \n---\n Documentation/git-commit-graph.txt | 21 +++++++++++++--------\n builtin/commit-graph.c | 13 ++++++++++---\n commit-graph.c | 6 ++++--\n commit-graph.h | 3 ++-\n t/t5324-split-commit-graph.sh | 26 ++++++++++++++++++++++++++\n 5 files changed, 55 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex cbf80226e9..d380c42e82 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -39,24 +39,29 @@ COMMANDS\n --------\n 'write'::\n \n-Write a commit-graph file based on the commits found in packfiles.\n+Write a commit-graph file based on the commits specified:\n+* With the `--input=stdin-packs` option, generate the new commit graph\n+by walking objects only in the specified pack-indexes. (Cannot be\n+combined with `--input=stdin-commits` or `--input=reachable`.)\n +\n-With the `--input=stdin-packs` option, generate the new commit graph by\n-walking objects only in the specified pack-indexes. (Cannot be combined\n-with `--input=stdin-commits` or `--input=reachable`.)\n-+\n-With the `--input=stdin-commits` option, generate the new commit graph\n+* With the `--input=stdin-commits` option, generate the new commit graph\n by walking commits starting at the commits specified in stdin as a list\n of OIDs in hex, one OID per line. (Cannot be combined with\n `--input=stdin-packs` or `--input=reachable`.)\n +\n-With the `--input=reachable` option, generate the new commit graph by\n+* With the `--input=reachable` option, generate the new commit graph by\n walking commits starting at all refs. (Cannot be combined with\n `--input=stdin-commits` or `--input=stdin-packs`.)\n +\n-With the `--input=append` option, include all commits that are present\n+* With the `--input=append` option, include all commits that are present\n in the existing commit-graph file.\n +\n+* With the `--input=none` option, behave as if `input=append` were\n+given, but do not walk other packs to find additional commits.\n+\n+If none of the above options are given, then commits found in\n+packfiles are specified.\n++\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n `/info/commit-graphs`. Commit-graph layers are merged based on the\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 03d815e652..937b98e99e 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -10,7 +10,8 @@\n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n-\t \"[--split[=]] [--input=] \"\n+\t \"[--split[=]] \"\n+\t \"[--input=] \"\n \t \"[--[no-]progress] \"),\n \tNULL\n };\n@@ -22,7 +23,8 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n \n static const char * const builtin_commit_graph_write_usage[] = {\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n-\t \"[--split[=]] [--input=] \"\n+\t \"[--split[=]] \"\n+\t \"[--input=] \"\n \t \"[--[no-]progress] \"),\n \tNULL\n };\n@@ -31,7 +33,8 @@ enum commit_graph_input {\n \tCOMMIT_GRAPH_INPUT_REACHABLE = (1 << 1),\n \tCOMMIT_GRAPH_INPUT_STDIN_PACKS = (1 << 2),\n \tCOMMIT_GRAPH_INPUT_STDIN_COMMITS = (1 << 3),\n-\tCOMMIT_GRAPH_INPUT_APPEND = (1 << 4)\n+\tCOMMIT_GRAPH_INPUT_APPEND = (1 << 4),\n+\tCOMMIT_GRAPH_INPUT_NONE = (1 << 5)\n };\n \n static struct opts_commit_graph {\n@@ -59,6 +62,8 @@ static int option_parse_input(const struct option *opt, const char *arg,\n \t\t*to |= COMMIT_GRAPH_INPUT_STDIN_COMMITS;\n \telse if (!strcmp(arg, \"append\"))\n \t\t*to |= COMMIT_GRAPH_INPUT_APPEND;\n+\telse if (!strcmp(arg, \"none\"))\n+\t\t*to |= (COMMIT_GRAPH_INPUT_APPEND | COMMIT_GRAPH_INPUT_NONE);\n \telse\n \t\tdie(_(\"unrecognized --input source, %s\"), arg);\n \treturn 0;\n@@ -211,6 +216,8 @@ static int graph_write(int argc, const char **argv)\n \t\topts.obj_dir = get_object_directory();\n \tif (opts.input & COMMIT_GRAPH_INPUT_APPEND)\n \t\tflags |= COMMIT_GRAPH_WRITE_APPEND;\n+\tif (opts.input & COMMIT_GRAPH_INPUT_NONE)\n+\t\tflags |= COMMIT_GRAPH_WRITE_NO_INPUT;\n \tif (opts.split)\n \t\tflags |= COMMIT_GRAPH_WRITE_SPLIT;\n \tif (opts.progress)\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 02e6ad9d1f..a5d7624073 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -808,7 +808,8 @@ struct write_commit_graph_context {\n \tunsigned append:1,\n \t\t report_progress:1,\n \t\t split:1,\n-\t\t check_oids:1;\n+\t\t check_oids:1,\n+\t\t no_input:1;\n \n \tconst struct split_commit_graph_opts *split_opts;\n };\n@@ -1802,6 +1803,7 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->split = flags & COMMIT_GRAPH_WRITE_SPLIT ? 1 : 0;\n \tctx->check_oids = flags & COMMIT_GRAPH_WRITE_CHECK_OIDS ? 1 : 0;\n \tctx->split_opts = split_opts;\n+\tctx->no_input = flags & COMMIT_GRAPH_WRITE_NO_INPUT ? 1 : 0;\n \n \tif (ctx->split) {\n \t\tstruct commit_graph *g;\n@@ -1860,7 +1862,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\t\tgoto cleanup;\n \t}\n \n-\tif (!pack_indexes && !commit_hex)\n+\tif (!ctx->no_input && !pack_indexes && !commit_hex)\n \t\tfill_oids_from_all_packs(ctx);\n \n \tclose_reachable(ctx);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex dadcc03808..dd8c00a2d8 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -81,7 +81,8 @@ enum commit_graph_write_flags {\n \tCOMMIT_GRAPH_WRITE_PROGRESS = (1 << 1),\n \tCOMMIT_GRAPH_WRITE_SPLIT = (1 << 2),\n \t/* Make sure that each OID in the input is a valid commit OID. */\n-\tCOMMIT_GRAPH_WRITE_CHECK_OIDS = (1 << 3)\n+\tCOMMIT_GRAPH_WRITE_CHECK_OIDS = (1 << 3),\n+\tCOMMIT_GRAPH_WRITE_NO_INPUT = (1 << 4)\n };\n \n enum commit_graph_split_flags {\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex dd74295885..296b5a9185 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -369,4 +369,30 @@ test_expect_success '--split=no-merge always writes an incremental' '\n \ttest_line_count = 2 $graphdir/commit-graph-chain\n '\n \n+test_expect_success '--split=no-merge, --input=none writes nothing' '\n+\ttest_when_finished rm -rf a graphs.before graphs.after &&\n+\trm -rf $graphdir &&\n+\tgit reset --hard commits/2 &&\n+\tgit rev-list -1 HEAD~1 >a &&\n+\tgit commit-graph write --split=no-merge --input=stdin-commits graphs.before &&\n+\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n+\tgit commit-graph write --split --input=none &&\n+\tls $graphdir/graph-*.graph >graphs.after &&\n+\ttest_cmp graphs.before graphs.after\n+'\n+\n+test_expect_success '--split=merge-all, --input=none merges the chain' '\n+\ttest_when_finished rm -rf a b &&\n+\trm -rf $graphdir &&\n+\tgit reset --hard commits/2 &&\n+\tgit rev-list -1 HEAD~1 >a &&\n+\tgit rev-list -1 HEAD >b &&\n+\tgit commit-graph write --split=no-merge --input=stdin-commits Hi,\n>\n> Here are another few patches that came out of working on GitHub's\n> deployment of incremental commit-graphs. These three patches introduce\n> two new options: '--split[=]' and\n> '--input='.\n\nI should have mentioned: these patches are based on top of my\n'tb/commit-graph-use-odb' branch, which I sent to the list in:\n\n https://lore.kernel.org/git/cover.1580424766.git.me@ttaylorr.com\n\nI think that it makes sense to queue the above other topic first before\nthis one is applied, at least since these patches are based on that\nbranch. I can prepare a version that is based on 'master' if that is\npreferable, there are only a handful of conflicts to resolve around use\nof '->obj_dir' vs '->odb->path' in changing the order.\n\nThe two just felt a little too large and disjoint to send as one larger\nseries.\n\n> [...]\n\nThanks,\nTaylor\n"},{"id":"390902","messageId":"ecb789c0-1a6e-244d-1b7f-de136e0173d2@gmail.com","threadId":"52725","inReplyTo":"20200131003210.GA91635@syl.local","subject":"Re: [PATCH 0/3] builtin/commit-graph.c: new split/merge options","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-31T13:26:20Z","receivedAt":"2020-01-31T13:26:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/30/2020 7:32 PM, Taylor Blau wrote:\n> On Thu, Jan 30, 2020 at 04:28:14PM -0800, Taylor Blau wrote:\n>> Hi,\n>>\n>> Here are another few patches that came out of working on GitHub's\n>> deployment of incremental commit-graphs. These three patches introduce\n>> two new options: '--split[=]' and\n>> '--input='.\n> \n> I should have mentioned: these patches are based on top of my\n> 'tb/commit-graph-use-odb' branch, which I sent to the list in:\n> \n> https://lore.kernel.org/git/cover.1580424766.git.me@ttaylorr.com\n> \n> I think that it makes sense to queue the above other topic first before\n> this one is applied, at least since these patches are based on that\n> branch. I can prepare a version that is based on 'master' if that is\n> preferable, there are only a handful of conflicts to resolve around use\n> of '->obj_dir' vs '->odb->path' in changing the order.\n> \n> The two just felt a little too large and disjoint to send as one larger\n> series.\n\nI think the order you recommend is good, especially because the obj_dir\ncleanup is less likely to be controversial. Command-line interface changes\nshould have more time to cook.\n\nThanks,\n-Stolee\n\n"},{"id":"390905","messageId":"8a648a69-60bd-2b19-facb-1d7821661883@gmail.com","threadId":"52725","inReplyTo":"4f5bc19903f8a1f5b153b5665de378e743e12744.1580430057.git.me@ttaylorr.com","subject":"Re: [PATCH 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-31T14:19:19Z","receivedAt":"2020-01-31T14:19:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/30/2020 7:28 PM, Taylor Blau wrote:\n> With '--split', the commit-graph machinery writes new commits in another\n> incremental commit-graph which is part of the existing chain, and\n> optionally decides to condense the chain into a single commit-graph.\n> This is done to ensure that the aysmptotic behavior of looking up a\n> commit in an incremental chain is dominated by the number of\n> incrementals in that chain. It can be controlled by the '--max-commits'\n> and '--size-multiple' options.\n> \n> On occasion, callers may want to ensure that 'git commit-graph write\n> --split' always writes an incremental, and never spends effort\n> condensing the incremental chain [1]. Previously, this was possible by\n> passing '--size-multiple=0', but this no longer the case following\n> 63020f175f (commit-graph: prefer default size_mult when given zero,\n> 2020-01-02).\n> \n> Reintroduce a less-magical variant of the above with a new pair of\n> arguments to '--split': '--split=no-merge' and '--split=merge-all'. When\n> '--split=no-merge' is given, the commit-graph machinery will never\n> condense an existing chain and will always write a new incremental.\n> Conversely, if '--split=merge-all' is given, any invocation including it\n> will always condense a chain if one exists. If '--split' is given with\n> no arguments, it behaves as before and defers to '--size-multiple', and\n> so on.\n> \n> [1]: This might occur when, for example, a server administrator running\n> some program after each push may want to ensure that each job runs\n> proportional in time to the size of the push, and does not \"jump\" when\n> the commit-graph machinery decides to trigger a merge.\n> \n> Signed-off-by: Taylor Blau \n> ---\n> Documentation/git-commit-graph.txt | 18 +++++++++++-----\n> builtin/commit-graph.c | 33 ++++++++++++++++++++++++++----\n> commit-graph.c | 19 +++++++++--------\n> commit-graph.h | 7 +++++++\n> t/t5324-split-commit-graph.sh | 25 ++++++++++++++++++++++\n> 5 files changed, 85 insertions(+), 17 deletions(-)\n> \n> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> index 28d1fee505..8d61ba9f56 100644\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -57,11 +57,19 @@ or `--stdin-packs`.)\n> With the `--append` option, include all commits that are present in the\n> existing commit-graph file.\n> +\n> -With the `--split` option, write the commit-graph as a chain of multiple\n> -commit-graph files stored in `/info/commit-graphs`. The new commits\n> -not already in the commit-graph are added in a new \"tip\" file. This file\n> -is merged with the existing file if the following merge conditions are\n> -met:\n> +With the `--split[=]` option, write the commit-graph as a\n> +chain of multiple commit-graph files stored in\n> +`/info/commit-graphs`. Commit-graph layers are merged based on the\n> +strategy and other splitting options. The new commits not already in the\n> +commit-graph are added in a new \"tip\" file. This file is merged with the\n> +existing file if the following merge conditions are met:\n> +* If `--split=merge-always` is specified, then a merge is always\n> +conducted, and the remaining options are ignored. Conversely, if\n> +`--split=no-merge` is specified, a merge is never performed, and the\n> +remaining options are ignored. A bare `--split` defers to the remaining\n> +options. (Note that merging a chain of commit graphs replaces the\n> +existing chain with a length-1 chain where the first and only\n> +incremental holds the entire graph).\n> +\n> * If `--size-multiple=` is not specified, let `X` equal 2. If the new\n> tip file would have `N` commits and the previous tip has `M` commits and\n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index de321c71ad..f03b46d627 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -9,7 +9,9 @@\n> \n> static char const * const builtin_commit_graph_usage[] = {\n> \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n> -\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] [--[no-]progress] \"),\n> +\tN_(\"git commit-graph write [--object-dir ] [--append] \"\n> +\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n> +\t \"[--[no-]progress] \"),\n> \tNULL\n> };\n> \n> @@ -19,7 +21,9 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n> };\n> \n> static const char * const builtin_commit_graph_write_usage[] = {\n> -\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] [--[no-]progress] \"),\n> +\tN_(\"git commit-graph write [--object-dir ] [--append] \"\n> +\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n> +\t \"[--[no-]progress] \"),\n> \tNULL\n> };\n> \n> @@ -101,6 +105,25 @@ static int graph_verify(int argc, const char **argv)\n> extern int read_replace_refs;\n> static struct split_commit_graph_opts split_opts;\n> \n> +static int write_option_parse_split(const struct option *opt, const char *arg,\n> +\t\t\t\t int unset)\n> +{\n> +\tenum commit_graph_split_flags *flags = opt->value;\n> +\n> +\topts.split = 1;\n> +\tif (!arg)\n> +\t\treturn 0;\n\nThis allows `--split` to continue working as-is. But should we also\nset \"*flags = COMMIT_GRAPH_SPLIT_UNSPECIFIED\" here? This allows one\nto run \"git commit-graph write --split=no-merge --split\" (which could\nhappen if \"--split=no-merge\" is inside an alias).\n\n> +test_expect_success '--split=merge-all always merges incrementals' '\n> +\ttest_when_finished rm -rf a b c &&\n> +\trm -rf $graphdir $infodir/commit-graph &&\n> +\tgit reset --hard commits/10 &&\n> +\tgit rev-list -3 HEAD~4 >a &&\n> +\tgit rev-list -2 HEAD~2 >b &&\n> +\tgit rev-list -2 HEAD >c &&\n> +\tgit commit-graph write --split=no-merge --stdin-commits +\tgit commit-graph write --split=no-merge --stdin-commits +\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n> +\tgit commit-graph write --split=merge-all --stdin-commits +\ttest_line_count = 1 $graphdir/commit-graph-chain\n> +'\n> +\n> +test_expect_success '--split=no-merge always writes an incremental' '\n> +\ttest_when_finished rm -rf a b &&\n> +\trm -rf $graphdir &&\n> +\tgit reset --hard commits/2 &&\n> +\tgit rev-list HEAD~1 >a &&\n> +\tgit rev-list HEAD >b &&\n> +\tgit commit-graph write --split --stdin-commits +\tgit commit-graph write --split=no-merge --stdin-commits +\ttest_line_count = 2 $graphdir/commit-graph-chain\n> +'\n> +\n> test_done\n\nGood tests!\n\nThanks,\n-Stolee\n\n"},{"id":"390906","messageId":"5fac4f9c-b9e3-5e78-591b-abc62e73ac0e@gmail.com","threadId":"52725","inReplyTo":"c329a8ddc087c9b74f673005f8b1e8769816ab9a.1580430057.git.me@ttaylorr.com","subject":"Re: [PATCH 3/3] builtin/commit-graph.c: support '--input=none'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-31T14:40:19Z","receivedAt":"2020-01-31T14:40:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/30/2020 7:28 PM, Taylor Blau wrote:\n> In the previous commit, we introduced '--[no-]merge', and alluded to the\n> fact that '--merge' would be useful for callers who wish to always\n> trigger a merge of an incremental chain.\n> \n> There is a problem with the above approach, which is that there is no\n> way to specify to the commit-graph builtin that a caller only wants to\n> include commits already in the graph. One can specify '--input=append'\n> to include all commits in the existing graphs, but the absence of\n> '--input=stdin-{commits,packs}' causes the builtin to call\n> 'fill_oids_from_all_packs()'.\n> \n> Passing '--input=reachable' (as in 'git commit-graph write\n> --split=merge-all --input=reachable --input=append') works around this\n> issue by making '--input=reachable' effectively a no-op, but this can be\n> prohibitively expensive in large repositories, making it an undesirable\n> choice for some users.\n> \n> Teach '--input=none' as an option to behave as if '--input=append' were\n> given, but to consider no other sources in addition.\n\nThe code change looks good.\n\n> +test_expect_success '--split=no-merge, --input=none writes nothing' '\n> +\ttest_when_finished rm -rf a graphs.before graphs.after &&\n> +\trm -rf $graphdir &&\n> +\tgit reset --hard commits/2 &&\n> +\tgit rev-list -1 HEAD~1 >a &&\n> +\tgit commit-graph write --split=no-merge --input=stdin-commits +\tls $graphdir/graph-*.graph >graphs.before &&\n> +\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n> +\tgit commit-graph write --split --input=none &&\n> +\tls $graphdir/graph-*.graph >graphs.after &&\n> +\ttest_cmp graphs.before graphs.after\n> +'\n> +\n> +test_expect_success '--split=merge-all, --input=none merges the chain' '\n> +\ttest_when_finished rm -rf a b &&\n> +\trm -rf $graphdir &&\n> +\tgit reset --hard commits/2 &&\n> +\tgit rev-list -1 HEAD~1 >a &&\n> +\tgit rev-list -1 HEAD >b &&\n> +\tgit commit-graph write --split=no-merge --input=stdin-commits +\tgit commit-graph write --split=no-merge --input=stdin-commits +\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n> +\tgit commit-graph write --split=merge-all --input=none &&\n> +\ttest_line_count = 1 $graphdir/commit-graph-chain\n> +'\n\nAnd these tests demonstrate the value quite clearly. Thanks!\n\n-Stolee\n"},{"id":"390907","messageId":"846706e9-efe2-448d-67a3-a96638e9bcbc@gmail.com","threadId":"52725","inReplyTo":"8effe35bcd1dadee3a29d996f269353cf6e4982d.1580430057.git.me@ttaylorr.com","subject":"Re: [PATCH 2/3] builtin/commit-graph.c: introduce '--input='","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-31T14:40:23Z","receivedAt":"2020-01-31T14:40:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/30/2020 7:28 PM, Taylor Blau wrote:\n> The 'write' mode of the 'commit-graph' supports input from a number of\n> different sources: pack indexes over stdin, commits over stdin, commits\n> reachable from all references, and so on. Each of these options are\n> specified with a unique option: '--stdin-packs', '--stdin-commits', etc.\n> \n> Similar to our replacement of 'git config [--]' with 'git config\n> [--type=]' (c.f., fb0dc3bac1 (builtin/config.c: support\n> `--type=` as preferred alias for `--`, 2018-04-18)), softly\n> deprecate '[--]' in favor of '[--input=]'.\n> \n> This makes it more clear to implement new options that are combinations\n> of other options (such as, for example, \"none\", a combination of the old\n> \"--append\" and a new sentinel to specify to _not_ look in other packs,\n> which we will implement in a future patch).\n> \n> Unfortunately, the new enumerated type is a bitfield, even though it\n> makes much more sense as '0, 1, 2, ...'. Even though *almost* all\n> options are pairwise exclusive, '--stdin-{packs,commits}' *is*\n> compatible with '--append'. For this reason, use a bitfield.\n> \n> Signed-off-by: Taylor Blau \n> ---\n> Documentation/git-commit-graph.txt | 26 +++++-----\n> builtin/commit-graph.c | 77 ++++++++++++++++++++++--------\n> t/t5318-commit-graph.sh | 46 +++++++++---------\n> t/t5324-split-commit-graph.sh | 44 ++++++++---------\n> 4 files changed, 114 insertions(+), 79 deletions(-)\n> \n> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> index 8d61ba9f56..cbf80226e9 100644\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -41,21 +41,21 @@ COMMANDS\n> \n> Write a commit-graph file based on the commits found in packfiles.\n> +\n> -With the `--stdin-packs` option, generate the new commit graph by\n> +With the `--input=stdin-packs` option, generate the new commit graph by\n> walking objects only in the specified pack-indexes. (Cannot be combined\n> -with `--stdin-commits` or `--reachable`.)\n> +with `--input=stdin-commits` or `--input=reachable`.)\n> +\n> -With the `--stdin-commits` option, generate the new commit graph by\n> -walking commits starting at the commits specified in stdin as a list\n> +With the `--input=stdin-commits` option, generate the new commit graph\n> +by walking commits starting at the commits specified in stdin as a list\n> of OIDs in hex, one OID per line. (Cannot be combined with\n> -`--stdin-packs` or `--reachable`.)\n> +`--input=stdin-packs` or `--input=reachable`.)\n> +\n> -With the `--reachable` option, generate the new commit graph by walking\n> -commits starting at all refs. (Cannot be combined with `--stdin-commits`\n> -or `--stdin-packs`.)\n> +With the `--input=reachable` option, generate the new commit graph by\n> +walking commits starting at all refs. (Cannot be combined with\n> +`--input=stdin-commits` or `--input=stdin-packs`.)\n> +\n> -With the `--append` option, include all commits that are present in the\n> -existing commit-graph file.\n> +With the `--input=append` option, include all commits that are present\n> +in the existing commit-graph file.\n> +\n> With the `--split[=]` option, write the commit-graph as a\n> chain of multiple commit-graph files stored in\n> @@ -107,20 +107,20 @@ $ git commit-graph write\n> using commits in ``.\n> +\n> ------------------------------------------------\n> -$ echo | git commit-graph write --stdin-packs\n> +$ echo | git commit-graph write --input=stdin-packs\n> ------------------------------------------------\n> \n> * Write a commit-graph file containing all reachable commits.\n> +\n> ------------------------------------------------\n> -$ git show-ref -s | git commit-graph write --stdin-commits\n> +$ git show-ref -s | git commit-graph write --input=stdin-commits\n> ------------------------------------------------\n> \n> * Write a commit-graph file containing all commits in the current\n> commit-graph file along with those reachable from `HEAD`.\n> +\n> ------------------------------------------------\n> -$ git rev-parse HEAD | git commit-graph write --stdin-commits --append\n> +$ git rev-parse HEAD | git commit-graph write --input=stdin-commits --input=append\n> ------------------------------------------------\n> \n> \n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index f03b46d627..03d815e652 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -10,7 +10,7 @@\n> static char const * const builtin_commit_graph_usage[] = {\n> \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n> \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n> -\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n> +\t \"[--split[=]] [--input=] \"\n> \t \"[--[no-]progress] \"),\n> \tNULL\n> };\n> @@ -22,22 +22,48 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n> \n> static const char * const builtin_commit_graph_write_usage[] = {\n> \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n> -\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n> +\t \"[--split[=]] [--input=] \"\n> \t \"[--[no-]progress] \"),\n> \tNULL\n> };\n> \n> +enum commit_graph_input {\n> +\tCOMMIT_GRAPH_INPUT_REACHABLE = (1 << 1),\n> +\tCOMMIT_GRAPH_INPUT_STDIN_PACKS = (1 << 2),\n> +\tCOMMIT_GRAPH_INPUT_STDIN_COMMITS = (1 << 3),\n> +\tCOMMIT_GRAPH_INPUT_APPEND = (1 << 4)\n> +};\n> +\n> static struct opts_commit_graph {\n> \tconst char *obj_dir;\n> -\tint reachable;\n> -\tint stdin_packs;\n> -\tint stdin_commits;\n> -\tint append;\n> +\tenum commit_graph_input input;\n> \tint split;\n> \tint shallow;\n> \tint progress;\n> } opts;\n> \n> +static int option_parse_input(const struct option *opt, const char *arg,\n> +\t\t\t int unset)\n> +{\n> +\tenum commit_graph_input *to = opt->value;\n> +\tif (unset || !strcmp(arg, \"packs\")) {\n> +\t\t*to = 0;\n> +\t\treturn 0;\n> +\t}\n\nHere, you _do_ clear the bitfield, allowing \"--input=reachable --input\"\nto do the correct override. Thanks!\n\n> +\n> +\tif (!strcmp(arg, \"reachable\"))\n> +\t\t*to |= COMMIT_GRAPH_INPUT_REACHABLE;\n> +\telse if (!strcmp(arg, \"stdin-packs\"))\n> +\t\t*to |= COMMIT_GRAPH_INPUT_STDIN_PACKS;\n> +\telse if (!strcmp(arg, \"stdin-commits\"))\n> +\t\t*to |= COMMIT_GRAPH_INPUT_STDIN_COMMITS;\n> +\telse if (!strcmp(arg, \"append\"))\n> +\t\t*to |= COMMIT_GRAPH_INPUT_APPEND;\n> +\telse\n> +\t\tdie(_(\"unrecognized --input source, %s\"), arg);\n> +\treturn 0;\n> +}\n> +\n> static struct object_directory *find_odb_or_die(struct repository *r,\n> \t\t\t\t\t\tconst char *obj_dir)\n> {\n> @@ -137,14 +163,21 @@ static int graph_write(int argc, const char **argv)\n> \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n> \t\t\tN_(\"dir\"),\n> \t\t\tN_(\"The object directory to store the graph\")),\n> -\t\tOPT_BOOL(0, \"reachable\", &opts.reachable,\n> -\t\t\tN_(\"start walk at all refs\")),\n> -\t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n> -\t\t\tN_(\"scan pack-indexes listed by stdin for commits\")),\n> -\t\tOPT_BOOL(0, \"stdin-commits\", &opts.stdin_commits,\n> -\t\t\tN_(\"start walk at commits listed by stdin\")),\n> -\t\tOPT_BOOL(0, \"append\", &opts.append,\n> -\t\t\tN_(\"include all commits already in the commit-graph file\")),\n> +\t\tOPT_CALLBACK(0, \"input\", &opts.input, NULL,\n> +\t\t\tN_(\"include commits from this source in the graph\"),\n> +\t\t\toption_parse_input),\n> +\t\tOPT_BIT(0, \"reachable\", &opts.input,\n> +\t\t\tN_(\"start walk at all refs\"),\n> +\t\t\tCOMMIT_GRAPH_INPUT_REACHABLE),\n> +\t\tOPT_BIT(0, \"stdin-packs\", &opts.input,\n> +\t\t\tN_(\"scan pack-indexes listed by stdin for commits\"),\n> +\t\t\tCOMMIT_GRAPH_INPUT_STDIN_PACKS),\n> +\t\tOPT_BIT(0, \"stdin-commits\", &opts.input,\n> +\t\t\tN_(\"start walk at commits listed by stdin\"),\n> +\t\t\tCOMMIT_GRAPH_INPUT_STDIN_COMMITS),\n> +\t\tOPT_BIT(0, \"append\", &opts.input,\n> +\t\t\tN_(\"include all commits already in the commit-graph file\"),\n> +\t\t\tCOMMIT_GRAPH_INPUT_APPEND),\n\nSince you are rewriting how we interpret the deprecated options, perhaps we\nshould keep some tests around that call these versions? It would make the\ntest diff be a bit smaller. These options can be removed from the tests if/when\nwe actually remove the options.\n\n> @@ -351,10 +351,10 @@ test_expect_success '--split=merge-all always merges incrementals' '\n> \tgit rev-list -3 HEAD~4 >a &&\n> \tgit rev-list -2 HEAD~2 >b &&\n> \tgit rev-list -2 HEAD >c &&\n> -\tgit commit-graph write --split=no-merge --stdin-commits -\tgit commit-graph write --split=no-merge --stdin-commits +\tgit commit-graph write --split=no-merge --input=stdin-commits +\tgit commit-graph write --split=no-merge --input=stdin-commits \ttest_line_count = 2 $graphdir/commit-graph-chain &&\n> -\tgit commit-graph write --split=merge-all --stdin-commits +\tgit commit-graph write --split=merge-all --input=stdin-commits \ttest_line_count = 1 $graphdir/commit-graph-chain\n> '\n> \n> @@ -364,8 +364,8 @@ test_expect_success '--split=no-merge always writes an incremental' '\n> \tgit reset --hard commits/2 &&\n> \tgit rev-list HEAD~1 >a &&\n> \tgit rev-list HEAD >b &&\n> -\tgit commit-graph write --split --stdin-commits -\tgit commit-graph write --split=no-merge --stdin-commits +\tgit commit-graph write --split --input=stdin-commits +\tgit commit-graph write --split=no-merge --input=stdin-commits \ttest_line_count = 2 $graphdir/commit-graph-chain\n> '\n\nUpdating these new tests with the given options is good. Perhaps convert only one\nof the old tests for each of the stdin-packs, reachable, \"\", and \"append\" options?\n\nThanks,\n-Stolee\n"},{"id":"390908","messageId":"b873ab69-cdae-7f71-b28b-671b62cea5a8@gmail.com","threadId":"52725","inReplyTo":"cover.1580430057.git.me@ttaylorr.com","subject":"Re: [PATCH 0/3] builtin/commit-graph.c: new split/merge options","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-31T14:41:01Z","receivedAt":"2020-01-31T14:41:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/30/2020 7:28 PM, Taylor Blau wrote:\n> Hi,\n> \n> Here are another few patches that came out of working on GitHub's\n> deployment of incremental commit-graphs. These three patches introduce\n> two new options: '--split[=]' and\n> '--input='.\n> \n> The former controls whether or not commit-graph's split machinery should\n> either write an incremental commit graph, squash the chain of\n> incrementals, or defer to the other options.\n> \n> (This comes from GitHub's desire to have more fine-grained control over\n> the commit-graph chain's behavior. We run short jobs after every push\n> that we would like to limit the running time of, and hence we do not\n> want to ever merge a long chain of incrementals unless we specifically\n> opt into that.)\n\nI can imagine many scenarios that require the amount of work to be\npredictable, which the current merge strategies do not guarantee.\n\n> The latter of the two new options does two things:\n> \n> * It cleans up the many options that specify input sources (e.g.,\n> '--stdin-commits', '--stdin-packs', '--reachable' and so on) under\n> one unifying name.\n> \n> * It allows us to introduce a new argument '--input=none', to prevent\n> walking each packfile when neither '--stdin-commits' nor\n> '--stdin-packs' was given.\n\nI'm happy with these goals.\n\n> Together, these have the combined effect of being able to write the\n> following two new invocations:\n> \n> $ git commit-graph write --split=merge-all --input=none\n> \n> $ git commit-graph write --split=no-merge --input=stdin-packs\n> \n> to (1) merge the chain, and (2) write a single new incremental.\n\nThis is much cleaner than adding yet another option to the builtin,\nand allows more flexibility in future extensions.\n\nI have a few comments on the patches, but they are minor.\n\nThanks!\n-Stolee\n\n"},{"id":"390935","messageId":"CAN0heSo29+sf1352EfGv=qFsir-e=Ompja445bh5z3vpOfkaJA@mail.gmail.com","threadId":"52725","inReplyTo":"4f5bc19903f8a1f5b153b5665de378e743e12744.1580430057.git.me@ttaylorr.com","subject":"Re: [PATCH 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-01-31T19:27:27Z","receivedAt":"2020-01-31T19:27:41Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Fri, 31 Jan 2020 at 01:29, Taylor Blau wrote:\n> With '--split', the commit-graph machinery writes new commits in another\n> incremental commit-graph which is part of the existing chain, and\n> optionally decides to condense the chain into a single commit-graph.\n> This is done to ensure that the aysmptotic behavior of looking up a\n\nasymptotic\n\n> commit in an incremental chain is dominated by the number of\n> incrementals in that chain. It can be controlled by the '--max-commits'\n> and '--size-multiple' options.\n>\n> On occasion, callers may want to ensure that 'git commit-graph write\n> --split' always writes an incremental, and never spends effort\n> condensing the incremental chain [1]. Previously, this was possible by\n> passing '--size-multiple=0', but this no longer the case following\n> 63020f175f (commit-graph: prefer default size_mult when given zero,\n> 2020-01-02).\n>\n> Reintroduce a less-magical variant of the above with a new pair of\n> arguments to '--split': '--split=no-merge' and '--split=merge-all'. When\n> '--split=no-merge' is given, the commit-graph machinery will never\n> condense an existing chain and will always write a new incremental.\n> Conversely, if '--split=merge-all' is given, any invocation including it\n> will always condense a chain if one exists. If '--split' is given with\n> no arguments, it behaves as before and defers to '--size-multiple', and\n> so on.\n\nI understand your motivation for doing this -- it all seems quite sound\nto me. Not being too familiar with this commit-graph splitting and\nmerging, I had a hard time groking this terminology though. From what I\nunderstand, before this patch, `--split` means \"write the commit-graph\nusing the 'split' file-format / in a split fashion\". Ok, that makes\nsense.\n\nFrom seeing `--split=no-merge`, I have no idea how to even parse that.\nOf course I don't want to merge, I want to split! Well not split, but\nwrite split files.\n\nI think it would help me (and others like me) if we could somehow\nseparate \"I want to use 'split' files\" from \"and here's how I want you\nto decide on the merging\". That is, which \"strategy\" to use. Obviously,\ntalking about a \"merge strategy\" would be stupid and \"split strategy\"\nalso seems a bit odd. \"Coalescing strategy\"? \"Joining strategy\"?\n\nOr can you convince me otherwise? From which angle should I look at\nthis?\n\n> -With the `--split` option, write the commit-graph as a chain of multiple\n> -commit-graph files stored in `/info/commit-graphs`. The new commits\n> -not already in the commit-graph are added in a new \"tip\" file. This file\n> -is merged with the existing file if the following merge conditions are\n> -met:\n> +With the `--split[=]` option, write the commit-graph as a\n> +chain of multiple commit-graph files stored in\n> +`/info/commit-graphs`. Commit-graph layers are merged based on the\n> +strategy and other splitting options. The new commits not already in the\n> +commit-graph are added in a new \"tip\" file. This file is merged with the\n> +existing file if the following merge conditions are met:\n> +* If `--split=merge-always` is specified, then a merge is always\n> +conducted, and the remaining options are ignored. Conversely, if\n> +`--split=no-merge` is specified, a merge is never performed, and the\n> +remaining options are ignored. A bare `--split` defers to the remaining\n> +options. (Note that merging a chain of commit graphs replaces the\n> +existing chain with a length-1 chain where the first and only\n> +incremental holds the entire graph).\n\nTo better understand the background for this patch, I read the manpage\nas it stands today. From the section on `--split`, I got this\nimpression: Let's say that `--max-commits` is huge, so all that matters\nis the `--size-multiple`. Let's say it's two. If the current tip\ncontains three commits and we're about to write one with two, then 2*2 >\n3 so we will merge, i.e., write a tip file with five commits. Unless of\ncourse *that* is more than half the size of the file before. And so on.\nWe might just merge once, or maybe \"many\" files in an avalanche effect.\nEvery now and then, such avalanches will go all the way back to the\nfirst file.\n\nNow this says something different, namely that once we decide to merge,\nwe do it all the way back, no matter what.\n\nThe commit message of 1771be90c8 (\"commit-graph: merge commit-graph\nchains\", 2019-06-18) seems to support my original understanding, at\nleast for `--size-multiple`, but not `--max-commits`, curiously enough.\n\nCan you clarify?\n\n> - OPT_BOOL(0, \"split\", &opts.split,\n> - N_(\"allow writing an incremental commit-graph file\")),\n> + OPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n> + N_(\"allow writing an incremental commit-graph file\"),\n\nThis still sounds very boolean. Cramming in the \"strategy\" might be hard\n-- is this an argument in favor of having two separate options? ;-)\n\n> +enum commit_graph_split_flags {\n> + COMMIT_GRAPH_SPLIT_UNSPECIFIED = 0,\n> + COMMIT_GRAPH_SPLIT_MERGE_REQUIRED = 1,\n> + COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED = 2\n> +};\n\nI wonder if this should be \"MERGE_AUTO\" rather than \"UNSPECIFIED\". This\nis related to Stolee's comment, I think.\n\n\nMartin\n"},{"id":"390936","messageId":"CAN0heSqGe0HpAj+qoYkif4pyjaW5EBNsRce9OmxMaZXTSW7C9w@mail.gmail.com","threadId":"52725","inReplyTo":"8effe35bcd1dadee3a29d996f269353cf6e4982d.1580430057.git.me@ttaylorr.com","subject":"Re: [PATCH 2/3] builtin/commit-graph.c: introduce '--input='","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-01-31T19:34:41Z","receivedAt":"2020-01-31T19:34:55Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Fri, 31 Jan 2020 at 01:30, Taylor Blau wrote:\n> The 'write' mode of the 'commit-graph' supports input from a number of\n> different sources: pack indexes over stdin, commits over stdin, commits\n> reachable from all references, and so on. Each of these options are\n> specified with a unique option: '--stdin-packs', '--stdin-commits', etc.\n>\n> Similar to our replacement of 'git config [--]' with 'git config\n> [--type=]' (c.f., fb0dc3bac1 (builtin/config.c: support\n> `--type=` as preferred alias for `--`, 2018-04-18)), softly\n> deprecate '[--]' in favor of '[--input=]'.\n>\n> This makes it more clear to implement new options that are combinations\n> of other options (such as, for example, \"none\", a combination of the old\n> \"--append\" and a new sentinel to specify to _not_ look in other packs,\n> which we will implement in a future patch).\n\nMakes sense.\n\n> Unfortunately, the new enumerated type is a bitfield, even though it\n> makes much more sense as '0, 1, 2, ...'. Even though *almost* all\n> options are pairwise exclusive, '--stdin-{packs,commits}' *is*\n> compatible with '--append'. For this reason, use a bitfield.\n\n> -With the `--append` option, include all commits that are present in the\n> -existing commit-graph file.\n> +With the `--input=append` option, include all commits that are present\n> +in the existing commit-graph file.\n\nWould it be too crazy to call this `--input=existing` instead, and have\nit be the same as `--append`? I find that `--append` makes a lot of\nsense (it's a mode we can turn on or off), whereas \"input = append\"\nseems more odd.\n\nFrom the next commit message, we learn that a long `--input=append`\ntriggers `fill_oids_from_all_packs()`, which wouldn't match my expecting\nfrom \"--input=existing\". So...\n\nDoes this hint that we could leave `--append` alone? We'd have lots of\ndifferent inputs to choose from using `--input`, and an `--append` mode\non top of that. That would make your inputs truly mutually exclusive and\nyou don't need the bitfield anymore, as you mention above. Hmm?\n\nWould that mean that the falling back to `fill_oids_from_all_packs()`\nwould follow from \"is there an --input?\", as opposed to from \"is there\nan --input except --input=append?\"?\n\n(I don't know whether these inputs really *have* to be exclusive, or if\nthat's more of an implementation detail. That is, even without an\n\"append\" input, might we one day be able to handle more inputs at once?\nMaybe this is not the time to worry about that.)\n\n\nMartin\n"},{"id":"390937","messageId":"CAN0heSpfqMChHO=iNuUExrBsMPcpZ6K2zH68GkHO2ttEiSinSA@mail.gmail.com","threadId":"52725","inReplyTo":"c329a8ddc087c9b74f673005f8b1e8769816ab9a.1580430057.git.me@ttaylorr.com","subject":"Re: [PATCH 3/3] builtin/commit-graph.c: support '--input=none'","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-01-31T19:45:59Z","receivedAt":"2020-01-31T19:46:13Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Fri, 31 Jan 2020 at 01:30, Taylor Blau wrote:\n> In the previous commit, we introduced '--[no-]merge', and alluded to the\n> fact that '--merge' would be useful for callers who wish to always\n> trigger a merge of an incremental chain.\n\nHmmm. So it looks like you've already had similar thoughts as I did\nabout patch 1/3. At some point, you had a separate `--merge=...` option,\nthen later made that `--split=...`. :-) Could you say something about why\nyou changed your mind?\n\n> There is a problem with the above approach, which is that there is no\n> way to specify to the commit-graph builtin that a caller only wants to\n> include commits already in the graph. One can specify '--input=append'\n> to include all commits in the existing graphs, but the absence of\n> '--input=stdin-{commits,packs}' causes the builtin to call\n> 'fill_oids_from_all_packs()'.\n\n(Use one of those options with an empty stdin? Anyway, let's read on.)\n\n> Passing '--input=reachable' (as in 'git commit-graph write\n> --split=merge-all --input=reachable --input=append') works around this\n> issue by making '--input=reachable' effectively a no-op, but this can be\n> prohibitively expensive in large repositories, making it an undesirable\n> choice for some users.\n>\n> Teach '--input=none' as an option to behave as if '--input=append' were\n> given, but to consider no other sources in addition.\n\n`--input=none` almost makes me wonder if it should produce an empty\ncommit-graph. But there wouldn't be much point in that... I guess\nanother way of defining this would be that it \"uses no input, and\nimplies `--append`\".\n\n> This, in conjunction with the option introduced in the previous patch\n> offers the convenient way to force the commit-graph machinery to\n> condense a chain of incrementals without requiring any new commits:\n>\n> $ git commit-graph write --split=merge-all --input=none\n\nRight.\n\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -39,24 +39,29 @@ COMMANDS\n> --------\n> 'write'::\n>\n> -Write a commit-graph file based on the commits found in packfiles.\n> +Write a commit-graph file based on the commits specified:\n> +* With the `--input=stdin-packs` option, generate the new commit graph\n> +by walking objects only in the specified pack-indexes. (Cannot be\n> +combined with `--input=stdin-commits` or `--input=reachable`.)\n> +\n> -With the `--input=stdin-packs` option, generate the new commit graph by\n> -walking objects only in the specified pack-indexes. (Cannot be combined\n> -with `--input=stdin-commits` or `--input=reachable`.)\n> -+\n> -With the `--input=stdin-commits` option, generate the new commit graph\n> +* With the `--input=stdin-commits` option, generate the new commit graph\n> by walking commits starting at the commits specified in stdin as a list\n> of OIDs in hex, one OID per line. (Cannot be combined with\n> `--input=stdin-packs` or `--input=reachable`.)\n> +\n> -With the `--input=reachable` option, generate the new commit graph by\n> +* With the `--input=reachable` option, generate the new commit graph by\n> walking commits starting at all refs. (Cannot be combined with\n> `--input=stdin-commits` or `--input=stdin-packs`.)\n> +\n> -With the `--input=append` option, include all commits that are present\n> +* With the `--input=append` option, include all commits that are present\n> in the existing commit-graph file.\n\nDo these changes above really belong in this commit?\n\n> +* With the `--input=none` option, behave as if `input=append` were\n> +given, but do not walk other packs to find additional commits.\n> +\n> +If none of the above options are given, then commits found in\n> +packfiles are specified.\n\n\"specified\"? Plus, that also happens for `--input=append` right? (It\nreally seems like \"append\" is an odd one among all the inputs.)\n\n> N_(\"git commit-graph write [--object-dir ] [--append] \"\n> - \"[--split[=]] [--input=] \"\n> + \"[--split[=]] \"\n> + \"[--input=] \"\n> \"[--[no-]progress] \"),\n\nHmm, you've left \"--append\" the old way.\n\n\nMartin\n"},{"id":"390977","messageId":"20200131233434.GJ10482@szeder.dev","threadId":"52725","inReplyTo":"4f5bc19903f8a1f5b153b5665de378e743e12744.1580430057.git.me@ttaylorr.com","subject":"Re: [PATCH 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-01-31T23:34:34Z","receivedAt":"2020-01-31T23:34:41Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Jan 30, 2020 at 04:28:17PM -0800, Taylor Blau wrote:\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 6d34829f57..02e6ad9d1f 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -1565,15 +1565,18 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n> \tnum_commits = ctx->commits.nr;\n> \tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n> \n> -\twhile (g && (g->num_commits <= size_mult * num_commits ||\n> -\t\t (max_commits && num_commits > max_commits))) {\n> -\t\tif (g->odb != ctx->odb)\n> -\t\t\tbreak;\n> +\tif (ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n\nThis line segfaults in the tests 'fetch.writeCommitGraph' and\n'fetch.writeCommitGraph with submodules' in 't5510-fetch.sh', because\n'git fetch' doesn't pass a 'split_opts' to the commit-graph functions.\n\nThread 1 \"git\" received signal SIGSEGV, Segmentation fault.\n0x00000000005113dd in split_graph_merge_strategy (ctx=0x9ca930)\n at commit-graph.c:1568\n1568 if (ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n(gdb) p *ctx\n$1 = {r = 0x9ae2a0 , odb = 0x9c0df0, graph_name = 0x0, oids = {\n list = 0xa02660, nr = 12, alloc = 1024}, commits = {list = 0x9caa00, \n nr = 6, alloc = 6}, num_extra_edges = 0, approx_nr_objects = 0, \n progress = 0x0, progress_done = 0, progress_cnt = 0, base_graph_name = 0x0, \n num_commit_graphs_before = 0, num_commit_graphs_after = 1, \n commit_graph_filenames_before = 0x0, commit_graph_filenames_after = 0x0, \n commit_graph_hash_after = 0x0, new_num_commits_in_base = 0, \n new_base_graph = 0x0, append = 0, report_progress = 1, split = 1, \n check_oids = 0, split_opts = 0x0}\n ^^^^^^^^^^^^^^^^\n(gdb) bt\n#0 0x00000000005113dd in split_graph_merge_strategy (ctx=0x9ca930)\n at commit-graph.c:1568\n#1 0x0000000000512446 in write_commit_graph (odb=0x9c0df0, pack_indexes=0x0, \n commit_hex=0x7fffffffd550, \n flags=(COMMIT_GRAPH_WRITE_PROGRESS | COMMIT_GRAPH_WRITE_SPLIT), \n split_opts=0x0) at commit-graph.c:1891\n#2 0x000000000050fd86 in write_commit_graph_reachable (odb=0x9c0df0, \n flags=(COMMIT_GRAPH_WRITE_PROGRESS | COMMIT_GRAPH_WRITE_SPLIT), \n split_opts=0x0) at commit-graph.c:1174\n ^^^^^^^^^^^^^^\n#3 0x0000000000444ea4 in cmd_fetch (argc=0, argv=0x7fffffffd9b8, prefix=0x0)\n at builtin/fetch.c:1873\n#4 0x0000000000406154 in run_builtin (p=0x969a88 , argc=2, \n argv=0x7fffffffd9b0) at git.c:444\n#5 0x00000000004064a4 in handle_builtin (argc=2, argv=0x7fffffffd9b0)\n at git.c:674\n#6 0x000000000040674c in run_argv (argcp=0x7fffffffd84c, argv=0x7fffffffd840)\n at git.c:741\n#7 0x0000000000406bbd in cmd_main (argc=2, argv=0x7fffffffd9b0) at git.c:872\n#8 0x00000000004cfd4e in main (argc=5, argv=0x7fffffffd998)\n at common-main.c:52\n\nNote that this function split_graph_merge_strategy() does look at\nvarious fields in 'ctx->split_opts' a bit earlier, but those accesses\nare protected by a 'if (ctx->split_opts)' condition.\nexpire_commit_graphs() does the same.\n\n\n> +\t\twhile (g && (g->num_commits <= size_mult * num_commits ||\n> +\t\t\t (max_commits && num_commits > max_commits) ||\n> +\t\t\t (ctx->split_opts->flags == COMMIT_GRAPH_SPLIT_MERGE_REQUIRED))) {\n> +\t\t\tif (g->odb != ctx->odb)\n> +\t\t\t\tbreak;\n> \n> -\t\tnum_commits += g->num_commits;\n> -\t\tg = g->base_graph;\n> +\t\t\tnum_commits += g->num_commits;\n> +\t\t\tg = g->base_graph;\n> \n> -\t\tctx->num_commit_graphs_after--;\n> +\t\t\tctx->num_commit_graphs_after--;\n> +\t\t}\n> \t}\n> \n"},{"id":"390999","messageId":"nycvar.QRO.7.76.6.2002012223490.46@tvgsbejvaqbjf.bet","threadId":"52725","inReplyTo":"20200131233434.GJ10482@szeder.dev","subject":"Re: [PATCH 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2020-02-01T21:25:36Z","receivedAt":"2020-02-01T21:25:51Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 1 Feb 2020, SZEDER Gábor wrote:\n\n> On Thu, Jan 30, 2020 at 04:28:17PM -0800, Taylor Blau wrote:\n> > diff --git a/commit-graph.c b/commit-graph.c\n> > index 6d34829f57..02e6ad9d1f 100644\n> > --- a/commit-graph.c\n> > +++ b/commit-graph.c\n> > @@ -1565,15 +1565,18 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n> > \tnum_commits = ctx->commits.nr;\n> > \tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n> >\n> > -\twhile (g && (g->num_commits <= size_mult * num_commits ||\n> > -\t\t (max_commits && num_commits > max_commits))) {\n> > -\t\tif (g->odb != ctx->odb)\n> > -\t\t\tbreak;\n> > +\tif (ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n>\n> This line segfaults in the tests 'fetch.writeCommitGraph' and\n> 'fetch.writeCommitGraph with submodules' in 't5510-fetch.sh', because\n> 'git fetch' doesn't pass a 'split_opts' to the commit-graph functions.\n\nI noticed the same. This patch seems to fix it for me:\n\n-- snip --\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a5d7624073f..af5c58833cf 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1566,7 +1566,8 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \tnum_commits = ctx->commits.nr;\n \tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n\n-\tif (ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n+\tif (ctx->split_opts &&\n+\t ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n \t\twhile (g && (g->num_commits <= size_mult * num_commits ||\n \t\t\t (max_commits && num_commits > max_commits) ||\n \t\t\t (ctx->split_opts->flags == COMMIT_GRAPH_SPLIT_MERGE_REQUIRED))) {\n-- snap --\n\nFor your convenience, I also pushed this up as\n`tb/commit-graph-split-merge` to https://github.com/dscho/git\n\nThanks,\nDscho\n\n\n>\n> Thread 1 \"git\" received signal SIGSEGV, Segmentation fault.\n> 0x00000000005113dd in split_graph_merge_strategy (ctx=0x9ca930)\n> at commit-graph.c:1568\n> 1568 if (ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n> (gdb) p *ctx\n> $1 = {r = 0x9ae2a0 , odb = 0x9c0df0, graph_name = 0x0, oids = {\n> list = 0xa02660, nr = 12, alloc = 1024}, commits = {list = 0x9caa00,\n> nr = 6, alloc = 6}, num_extra_edges = 0, approx_nr_objects = 0,\n> progress = 0x0, progress_done = 0, progress_cnt = 0, base_graph_name = 0x0,\n> num_commit_graphs_before = 0, num_commit_graphs_after = 1,\n> commit_graph_filenames_before = 0x0, commit_graph_filenames_after = 0x0,\n> commit_graph_hash_after = 0x0, new_num_commits_in_base = 0,\n> new_base_graph = 0x0, append = 0, report_progress = 1, split = 1,\n> check_oids = 0, split_opts = 0x0}\n> ^^^^^^^^^^^^^^^^\n> (gdb) bt\n> #0 0x00000000005113dd in split_graph_merge_strategy (ctx=0x9ca930)\n> at commit-graph.c:1568\n> #1 0x0000000000512446 in write_commit_graph (odb=0x9c0df0, pack_indexes=0x0,\n> commit_hex=0x7fffffffd550,\n> flags=(COMMIT_GRAPH_WRITE_PROGRESS | COMMIT_GRAPH_WRITE_SPLIT),\n> split_opts=0x0) at commit-graph.c:1891\n> #2 0x000000000050fd86 in write_commit_graph_reachable (odb=0x9c0df0,\n> flags=(COMMIT_GRAPH_WRITE_PROGRESS | COMMIT_GRAPH_WRITE_SPLIT),\n> split_opts=0x0) at commit-graph.c:1174\n> ^^^^^^^^^^^^^^\n> #3 0x0000000000444ea4 in cmd_fetch (argc=0, argv=0x7fffffffd9b8, prefix=0x0)\n> at builtin/fetch.c:1873\n> #4 0x0000000000406154 in run_builtin (p=0x969a88 , argc=2,\n> argv=0x7fffffffd9b0) at git.c:444\n> #5 0x00000000004064a4 in handle_builtin (argc=2, argv=0x7fffffffd9b0)\n> at git.c:674\n> #6 0x000000000040674c in run_argv (argcp=0x7fffffffd84c, argv=0x7fffffffd840)\n> at git.c:741\n> #7 0x0000000000406bbd in cmd_main (argc=2, argv=0x7fffffffd9b0) at git.c:872\n> #8 0x00000000004cfd4e in main (argc=5, argv=0x7fffffffd998)\n> at common-main.c:52\n>\n> Note that this function split_graph_merge_strategy() does look at\n> various fields in 'ctx->split_opts' a bit earlier, but those accesses\n> are protected by a 'if (ctx->split_opts)' condition.\n> expire_commit_graphs() does the same.\n>\n>\n> > +\t\twhile (g && (g->num_commits <= size_mult * num_commits ||\n> > +\t\t\t (max_commits && num_commits > max_commits) ||\n> > +\t\t\t (ctx->split_opts->flags == COMMIT_GRAPH_SPLIT_MERGE_REQUIRED))) {\n> > +\t\t\tif (g->odb != ctx->odb)\n> > +\t\t\t\tbreak;\n> >\n> > -\t\tnum_commits += g->num_commits;\n> > -\t\tg = g->base_graph;\n> > +\t\t\tnum_commits += g->num_commits;\n> > +\t\t\tg = g->base_graph;\n> >\n> > -\t\tctx->num_commit_graphs_after--;\n> > +\t\t\tctx->num_commit_graphs_after--;\n> > +\t\t}\n> > \t}\n> >\n>\n"},{"id":"391032","messageId":"20200203104730.GK10482@szeder.dev","threadId":"52725","inReplyTo":"nycvar.QRO.7.76.6.2002012223490.46@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-02-03T10:47:30Z","receivedAt":"2020-02-03T10:47:39Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Sat, Feb 01, 2020 at 10:25:36PM +0100, Johannes Schindelin wrote:\n> Hi,\n> \n> On Sat, 1 Feb 2020, SZEDER Gábor wrote:\n> \n> > On Thu, Jan 30, 2020 at 04:28:17PM -0800, Taylor Blau wrote:\n> > > diff --git a/commit-graph.c b/commit-graph.c\n> > > index 6d34829f57..02e6ad9d1f 100644\n> > > --- a/commit-graph.c\n> > > +++ b/commit-graph.c\n> > > @@ -1565,15 +1565,18 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n> > > \tnum_commits = ctx->commits.nr;\n> > > \tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n> > >\n> > > -\twhile (g && (g->num_commits <= size_mult * num_commits ||\n> > > -\t\t (max_commits && num_commits > max_commits))) {\n> > > -\t\tif (g->odb != ctx->odb)\n> > > -\t\t\tbreak;\n> > > +\tif (ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n> >\n> > This line segfaults in the tests 'fetch.writeCommitGraph' and\n> > 'fetch.writeCommitGraph with submodules' in 't5510-fetch.sh', because\n> > 'git fetch' doesn't pass a 'split_opts' to the commit-graph functions.\n> \n> I noticed the same. This patch seems to fix it for me:\n> \n> -- snip --\n> diff --git a/commit-graph.c b/commit-graph.c\n> index a5d7624073f..af5c58833cf 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -1566,7 +1566,8 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n> \tnum_commits = ctx->commits.nr;\n> \tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n> \n> -\tif (ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n> +\tif (ctx->split_opts &&\n> +\t ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n> \t\twhile (g && (g->num_commits <= size_mult * num_commits ||\n> \t\t\t (max_commits && num_commits > max_commits) ||\n> \t\t\t (ctx->split_opts->flags == COMMIT_GRAPH_SPLIT_MERGE_REQUIRED))) {\n> -- snap --\n\nYeah, that's what I noted below, but I'm not sure that this is the\nright solution. Why doesn't cmd_fetch() call\nwrite_commit_graph_reachable() with a real 'split_opts' parameter in\nthe first place? Wouldn't it be better if it did?\n\n\n> > Thread 1 \"git\" received signal SIGSEGV, Segmentation fault.\n> > 0x00000000005113dd in split_graph_merge_strategy (ctx=0x9ca930)\n> > at commit-graph.c:1568\n> > 1568 if (ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n> > (gdb) p *ctx\n> > $1 = {r = 0x9ae2a0 , odb = 0x9c0df0, graph_name = 0x0, oids = {\n> > list = 0xa02660, nr = 12, alloc = 1024}, commits = {list = 0x9caa00,\n> > nr = 6, alloc = 6}, num_extra_edges = 0, approx_nr_objects = 0,\n> > progress = 0x0, progress_done = 0, progress_cnt = 0, base_graph_name = 0x0,\n> > num_commit_graphs_before = 0, num_commit_graphs_after = 1,\n> > commit_graph_filenames_before = 0x0, commit_graph_filenames_after = 0x0,\n> > commit_graph_hash_after = 0x0, new_num_commits_in_base = 0,\n> > new_base_graph = 0x0, append = 0, report_progress = 1, split = 1,\n> > check_oids = 0, split_opts = 0x0}\n> > ^^^^^^^^^^^^^^^^\n> > (gdb) bt\n> > #0 0x00000000005113dd in split_graph_merge_strategy (ctx=0x9ca930)\n> > at commit-graph.c:1568\n> > #1 0x0000000000512446 in write_commit_graph (odb=0x9c0df0, pack_indexes=0x0,\n> > commit_hex=0x7fffffffd550,\n> > flags=(COMMIT_GRAPH_WRITE_PROGRESS | COMMIT_GRAPH_WRITE_SPLIT),\n> > split_opts=0x0) at commit-graph.c:1891\n> > #2 0x000000000050fd86 in write_commit_graph_reachable (odb=0x9c0df0,\n> > flags=(COMMIT_GRAPH_WRITE_PROGRESS | COMMIT_GRAPH_WRITE_SPLIT),\n> > split_opts=0x0) at commit-graph.c:1174\n> > ^^^^^^^^^^^^^^\n> > #3 0x0000000000444ea4 in cmd_fetch (argc=0, argv=0x7fffffffd9b8, prefix=0x0)\n> > at builtin/fetch.c:1873\n> > #4 0x0000000000406154 in run_builtin (p=0x969a88 , argc=2,\n> > argv=0x7fffffffd9b0) at git.c:444\n> > #5 0x00000000004064a4 in handle_builtin (argc=2, argv=0x7fffffffd9b0)\n> > at git.c:674\n> > #6 0x000000000040674c in run_argv (argcp=0x7fffffffd84c, argv=0x7fffffffd840)\n> > at git.c:741\n> > #7 0x0000000000406bbd in cmd_main (argc=2, argv=0x7fffffffd9b0) at git.c:872\n> > #8 0x00000000004cfd4e in main (argc=5, argv=0x7fffffffd998)\n> > at common-main.c:52\n> >\n> > Note that this function split_graph_merge_strategy() does look at\n> > various fields in 'ctx->split_opts' a bit earlier, but those accesses\n> > are protected by a 'if (ctx->split_opts)' condition.\n> > expire_commit_graphs() does the same.\n> >\n> >\n> > > +\t\twhile (g && (g->num_commits <= size_mult * num_commits ||\n> > > +\t\t\t (max_commits && num_commits > max_commits) ||\n> > > +\t\t\t (ctx->split_opts->flags == COMMIT_GRAPH_SPLIT_MERGE_REQUIRED))) {\n> > > +\t\t\tif (g->odb != ctx->odb)\n> > > +\t\t\t\tbreak;\n> > >\n> > > -\t\tnum_commits += g->num_commits;\n> > > -\t\tg = g->base_graph;\n> > > +\t\t\tnum_commits += g->num_commits;\n> > > +\t\t\tg = g->base_graph;\n> > >\n> > > -\t\tctx->num_commit_graphs_after--;\n> > > +\t\t\tctx->num_commit_graphs_after--;\n> > > +\t\t}\n> > > \t}\n> > >\n> >\n\n"},{"id":"391037","messageId":"20200203111117.GC4153124@coredump.intra.peff.net","threadId":"52725","inReplyTo":"20200203104730.GK10482@szeder.dev","subject":"Re: [PATCH 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-02-03T11:11:17Z","receivedAt":"2020-02-03T11:11:19Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Feb 03, 2020 at 11:47:30AM +0100, SZEDER Gábor wrote:\n\n> > -- snip --\n> > diff --git a/commit-graph.c b/commit-graph.c\n> > index a5d7624073f..af5c58833cf 100644\n> > --- a/commit-graph.c\n> > +++ b/commit-graph.c\n> > @@ -1566,7 +1566,8 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n> > \tnum_commits = ctx->commits.nr;\n> > \tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n> > \n> > -\tif (ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n> > +\tif (ctx->split_opts &&\n> > +\t ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n> > \t\twhile (g && (g->num_commits <= size_mult * num_commits ||\n> > \t\t\t (max_commits && num_commits > max_commits) ||\n> > \t\t\t (ctx->split_opts->flags == COMMIT_GRAPH_SPLIT_MERGE_REQUIRED))) {\n> > -- snap --\n> \n> Yeah, that's what I noted below, but I'm not sure that this is the\n> right solution. Why doesn't cmd_fetch() call\n> write_commit_graph_reachable() with a real 'split_opts' parameter in\n> the first place? Wouldn't it be better if it did?\n\nIt used to provide a \"blank\" split_opts until 63020f175f (commit-graph:\nprefer default size_mult when given zero, 2020-01-02), which caused a\nbug. That bug was since fixed, but the idea was to keep things\nconvenient for callers.\n\nWhether that's a good idea or not I guess is up for debate, but it\ncertainly is what the commit-graph code has tried to provide for some\ntime. If we're not going to follow that in this new code, then we should\npresumably also rip out all of the other \"if (ctx->split_opts)\" lines.\n\n-Peff\n"},{"id":"391070","messageId":"20200204034714.GA5790@syl.local","threadId":"52725","inReplyTo":"8a648a69-60bd-2b19-facb-1d7821661883@gmail.com","subject":"Re: [PATCH 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-04T03:47:14Z","receivedAt":"2020-02-04T03:47:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jan 31, 2020 at 09:19:19AM -0500, Derrick Stolee wrote:\n> On 1/30/2020 7:28 PM, Taylor Blau wrote:\n> > With '--split', the commit-graph machinery writes new commits in another\n> > incremental commit-graph which is part of the existing chain, and\n> > optionally decides to condense the chain into a single commit-graph.\n> > This is done to ensure that the aysmptotic behavior of looking up a\n> > commit in an incremental chain is dominated by the number of\n> > incrementals in that chain. It can be controlled by the '--max-commits'\n> > and '--size-multiple' options.\n> >\n> > On occasion, callers may want to ensure that 'git commit-graph write\n> > --split' always writes an incremental, and never spends effort\n> > condensing the incremental chain [1]. Previously, this was possible by\n> > passing '--size-multiple=0', but this no longer the case following\n> > 63020f175f (commit-graph: prefer default size_mult when given zero,\n> > 2020-01-02).\n> >\n> > Reintroduce a less-magical variant of the above with a new pair of\n> > arguments to '--split': '--split=no-merge' and '--split=merge-all'. When\n> > '--split=no-merge' is given, the commit-graph machinery will never\n> > condense an existing chain and will always write a new incremental.\n> > Conversely, if '--split=merge-all' is given, any invocation including it\n> > will always condense a chain if one exists. If '--split' is given with\n> > no arguments, it behaves as before and defers to '--size-multiple', and\n> > so on.\n> >\n> > [1]: This might occur when, for example, a server administrator running\n> > some program after each push may want to ensure that each job runs\n> > proportional in time to the size of the push, and does not \"jump\" when\n> > the commit-graph machinery decides to trigger a merge.\n> >\n> > Signed-off-by: Taylor Blau \n> > ---\n> > Documentation/git-commit-graph.txt | 18 +++++++++++-----\n> > builtin/commit-graph.c | 33 ++++++++++++++++++++++++++----\n> > commit-graph.c | 19 +++++++++--------\n> > commit-graph.h | 7 +++++++\n> > t/t5324-split-commit-graph.sh | 25 ++++++++++++++++++++++\n> > 5 files changed, 85 insertions(+), 17 deletions(-)\n> >\n> > diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> > index 28d1fee505..8d61ba9f56 100644\n> > --- a/Documentation/git-commit-graph.txt\n> > +++ b/Documentation/git-commit-graph.txt\n> > @@ -57,11 +57,19 @@ or `--stdin-packs`.)\n> > With the `--append` option, include all commits that are present in the\n> > existing commit-graph file.\n> > +\n> > -With the `--split` option, write the commit-graph as a chain of multiple\n> > -commit-graph files stored in `/info/commit-graphs`. The new commits\n> > -not already in the commit-graph are added in a new \"tip\" file. This file\n> > -is merged with the existing file if the following merge conditions are\n> > -met:\n> > +With the `--split[=]` option, write the commit-graph as a\n> > +chain of multiple commit-graph files stored in\n> > +`/info/commit-graphs`. Commit-graph layers are merged based on the\n> > +strategy and other splitting options. The new commits not already in the\n> > +commit-graph are added in a new \"tip\" file. This file is merged with the\n> > +existing file if the following merge conditions are met:\n> > +* If `--split=merge-always` is specified, then a merge is always\n> > +conducted, and the remaining options are ignored. Conversely, if\n> > +`--split=no-merge` is specified, a merge is never performed, and the\n> > +remaining options are ignored. A bare `--split` defers to the remaining\n> > +options. (Note that merging a chain of commit graphs replaces the\n> > +existing chain with a length-1 chain where the first and only\n> > +incremental holds the entire graph).\n> > +\n> > * If `--size-multiple=` is not specified, let `X` equal 2. If the new\n> > tip file would have `N` commits and the previous tip has `M` commits and\n> > diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> > index de321c71ad..f03b46d627 100644\n> > --- a/builtin/commit-graph.c\n> > +++ b/builtin/commit-graph.c\n> > @@ -9,7 +9,9 @@\n> >\n> > static char const * const builtin_commit_graph_usage[] = {\n> > \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n> > -\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] [--[no-]progress] \"),\n> > +\tN_(\"git commit-graph write [--object-dir ] [--append] \"\n> > +\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n> > +\t \"[--[no-]progress] \"),\n> > \tNULL\n> > };\n> >\n> > @@ -19,7 +21,9 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n> > };\n> >\n> > static const char * const builtin_commit_graph_write_usage[] = {\n> > -\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] [--[no-]progress] \"),\n> > +\tN_(\"git commit-graph write [--object-dir ] [--append] \"\n> > +\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n> > +\t \"[--[no-]progress] \"),\n> > \tNULL\n> > };\n> >\n> > @@ -101,6 +105,25 @@ static int graph_verify(int argc, const char **argv)\n> > extern int read_replace_refs;\n> > static struct split_commit_graph_opts split_opts;\n> >\n> > +static int write_option_parse_split(const struct option *opt, const char *arg,\n> > +\t\t\t\t int unset)\n> > +{\n> > +\tenum commit_graph_split_flags *flags = opt->value;\n> > +\n> > +\topts.split = 1;\n> > +\tif (!arg)\n> > +\t\treturn 0;\n>\n> This allows `--split` to continue working as-is. But should we also\n> set \"*flags = COMMIT_GRAPH_SPLIT_UNSPECIFIED\" here? This allows one\n> to run \"git commit-graph write --split=no-merge --split\" (which could\n> happen if \"--split=no-merge\" is inside an alias).\n\nYeah, this is an oversight on my part. I think that we should set the\nsplit option to 'COMMIT_GRAPH_SPLIT_UNSPECIFIED' when '--split' is\ngiven, for exactly the reason you outlined above. Thanks for the\nsuggestion!\n\n> > +test_expect_success '--split=merge-all always merges incrementals' '\n> > +\ttest_when_finished rm -rf a b c &&\n> > +\trm -rf $graphdir $infodir/commit-graph &&\n> > +\tgit reset --hard commits/10 &&\n> > +\tgit rev-list -3 HEAD~4 >a &&\n> > +\tgit rev-list -2 HEAD~2 >b &&\n> > +\tgit rev-list -2 HEAD >c &&\n> > +\tgit commit-graph write --split=no-merge --stdin-commits > +\tgit commit-graph write --split=no-merge --stdin-commits > +\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n> > +\tgit commit-graph write --split=merge-all --stdin-commits > +\ttest_line_count = 1 $graphdir/commit-graph-chain\n> > +'\n> > +\n> > +test_expect_success '--split=no-merge always writes an incremental' '\n> > +\ttest_when_finished rm -rf a b &&\n> > +\trm -rf $graphdir &&\n> > +\tgit reset --hard commits/2 &&\n> > +\tgit rev-list HEAD~1 >a &&\n> > +\tgit rev-list HEAD >b &&\n> > +\tgit commit-graph write --split --stdin-commits > +\tgit commit-graph write --split=no-merge --stdin-commits > +\ttest_line_count = 2 $graphdir/commit-graph-chain\n> > +'\n> > +\n> > test_done\n>\n> Good tests!\n\nThanks :-).\n\n> Thanks,\n> -Stolee\n\nThanks,\nTaylor\n"},{"id":"391071","messageId":"20200204035821.GB5790@syl.local","threadId":"52725","inReplyTo":"20200203111117.GC4153124@coredump.intra.peff.net","subject":"Re: [PATCH 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-04T03:58:21Z","receivedAt":"2020-02-04T03:58:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Feb 03, 2020 at 06:11:17AM -0500, Jeff King wrote:\n> On Mon, Feb 03, 2020 at 11:47:30AM +0100, SZEDER Gábor wrote:\n>\n> > > -- snip --\n> > > diff --git a/commit-graph.c b/commit-graph.c\n> > > index a5d7624073f..af5c58833cf 100644\n> > > --- a/commit-graph.c\n> > > +++ b/commit-graph.c\n> > > @@ -1566,7 +1566,8 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n> > > \tnum_commits = ctx->commits.nr;\n> > > \tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n> > >\n> > > -\tif (ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n> > > +\tif (ctx->split_opts &&\n> > > +\t ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n> > > \t\twhile (g && (g->num_commits <= size_mult * num_commits ||\n> > > \t\t\t (max_commits && num_commits > max_commits) ||\n> > > \t\t\t (ctx->split_opts->flags == COMMIT_GRAPH_SPLIT_MERGE_REQUIRED))) {\n> > > -- snap --\n> >\n> > Yeah, that's what I noted below, but I'm not sure that this is the\n> > right solution. Why doesn't cmd_fetch() call\n> > write_commit_graph_reachable() with a real 'split_opts' parameter in\n> > the first place? Wouldn't it be better if it did?\n>\n> It used to provide a \"blank\" split_opts until 63020f175f (commit-graph:\n> prefer default size_mult when given zero, 2020-01-02), which caused a\n> bug. That bug was since fixed, but the idea was to keep things\n> convenient for callers.\n>\n> Whether that's a good idea or not I guess is up for debate, but it\n> certainly is what the commit-graph code has tried to provide for some\n> time. If we're not going to follow that in this new code, then we should\n> presumably also rip out all of the other \"if (ctx->split_opts)\" lines.\n\nI think that this seems like a good step that we should probably take,\nbut I don't think that it's necessary for the series at hand. The\npattern in this function is to define a local variable which has the\nsame value as in split_opts, or a reasonable default if split_opts is\nNULL (c.f., 'max_commits' and 'size_mult').\n\nSo, I think that a safe thing to do which prevents the segv and doesn't\nchange the pattern too much is to write:\n\n enum commit_graph_split_flags flags = COMMIT_GRAPH_SPLIT_MERGE_AUTO;\n if (ctx->split_opts) {\n /* ... */\n flags = ctx->split_opts->flags;\n }\n\n /* ... */\n\n if (flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n while ( ... ) { ... }\n }\n\nThis is adding another local variable, which seems like an odd thing to\ndo *every* time that we add another member to split_opts. So for that\nreason it seems like in the longer-term we should either force the\ncaller to pass in a blank, or do something else that doesn't require\nthis, but I think that the intermediate cost isn't too bad.\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"391072","messageId":"20200204035903.GC5790@syl.local","threadId":"52725","inReplyTo":"20200131233434.GJ10482@szeder.dev","subject":"Re: [PATCH 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-04T03:59:03Z","receivedAt":"2020-02-04T03:59:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Feb 01, 2020 at 12:34:34AM +0100, SZEDER Gábor wrote:\n> On Thu, Jan 30, 2020 at 04:28:17PM -0800, Taylor Blau wrote:\n> > diff --git a/commit-graph.c b/commit-graph.c\n> > index 6d34829f57..02e6ad9d1f 100644\n> > --- a/commit-graph.c\n> > +++ b/commit-graph.c\n> > @@ -1565,15 +1565,18 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n> > \tnum_commits = ctx->commits.nr;\n> > \tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n> >\n> > -\twhile (g && (g->num_commits <= size_mult * num_commits ||\n> > -\t\t (max_commits && num_commits > max_commits))) {\n> > -\t\tif (g->odb != ctx->odb)\n> > -\t\t\tbreak;\n> > +\tif (ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n>\n> This line segfaults in the tests 'fetch.writeCommitGraph' and\n> 'fetch.writeCommitGraph with submodules' in 't5510-fetch.sh', because\n> 'git fetch' doesn't pass a 'split_opts' to the commit-graph functions.\n\nThanks for pointing it out. I responded in more detail lower in the\nthread with the fix, but I appreciate your testing each patch. Clearly,\nI just should have run the full suite myself before sending!\n\n> Thread 1 \"git\" received signal SIGSEGV, Segmentation fault.\n> 0x00000000005113dd in split_graph_merge_strategy (ctx=0x9ca930)\n> at commit-graph.c:1568\n> 1568 if (ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n> (gdb) p *ctx\n> $1 = {r = 0x9ae2a0 , odb = 0x9c0df0, graph_name = 0x0, oids = {\n> list = 0xa02660, nr = 12, alloc = 1024}, commits = {list = 0x9caa00,\n> nr = 6, alloc = 6}, num_extra_edges = 0, approx_nr_objects = 0,\n> progress = 0x0, progress_done = 0, progress_cnt = 0, base_graph_name = 0x0,\n> num_commit_graphs_before = 0, num_commit_graphs_after = 1,\n> commit_graph_filenames_before = 0x0, commit_graph_filenames_after = 0x0,\n> commit_graph_hash_after = 0x0, new_num_commits_in_base = 0,\n> new_base_graph = 0x0, append = 0, report_progress = 1, split = 1,\n> check_oids = 0, split_opts = 0x0}\n> ^^^^^^^^^^^^^^^^\n> (gdb) bt\n> #0 0x00000000005113dd in split_graph_merge_strategy (ctx=0x9ca930)\n> at commit-graph.c:1568\n> #1 0x0000000000512446 in write_commit_graph (odb=0x9c0df0, pack_indexes=0x0,\n> commit_hex=0x7fffffffd550,\n> flags=(COMMIT_GRAPH_WRITE_PROGRESS | COMMIT_GRAPH_WRITE_SPLIT),\n> split_opts=0x0) at commit-graph.c:1891\n> #2 0x000000000050fd86 in write_commit_graph_reachable (odb=0x9c0df0,\n> flags=(COMMIT_GRAPH_WRITE_PROGRESS | COMMIT_GRAPH_WRITE_SPLIT),\n> split_opts=0x0) at commit-graph.c:1174\n> ^^^^^^^^^^^^^^\n> #3 0x0000000000444ea4 in cmd_fetch (argc=0, argv=0x7fffffffd9b8, prefix=0x0)\n> at builtin/fetch.c:1873\n> #4 0x0000000000406154 in run_builtin (p=0x969a88 , argc=2,\n> argv=0x7fffffffd9b0) at git.c:444\n> #5 0x00000000004064a4 in handle_builtin (argc=2, argv=0x7fffffffd9b0)\n> at git.c:674\n> #6 0x000000000040674c in run_argv (argcp=0x7fffffffd84c, argv=0x7fffffffd840)\n> at git.c:741\n> #7 0x0000000000406bbd in cmd_main (argc=2, argv=0x7fffffffd9b0) at git.c:872\n> #8 0x00000000004cfd4e in main (argc=5, argv=0x7fffffffd998)\n> at common-main.c:52\n>\n> Note that this function split_graph_merge_strategy() does look at\n> various fields in 'ctx->split_opts' a bit earlier, but those accesses\n> are protected by a 'if (ctx->split_opts)' condition.\n> expire_commit_graphs() does the same.\n>\n>\n> > +\t\twhile (g && (g->num_commits <= size_mult * num_commits ||\n> > +\t\t\t (max_commits && num_commits > max_commits) ||\n> > +\t\t\t (ctx->split_opts->flags == COMMIT_GRAPH_SPLIT_MERGE_REQUIRED))) {\n> > +\t\t\tif (g->odb != ctx->odb)\n> > +\t\t\t\tbreak;\n> >\n> > -\t\tnum_commits += g->num_commits;\n> > -\t\tg = g->base_graph;\n> > +\t\t\tnum_commits += g->num_commits;\n> > +\t\t\tg = g->base_graph;\n> >\n> > -\t\tctx->num_commit_graphs_after--;\n> > +\t\t\tctx->num_commit_graphs_after--;\n> > +\t\t}\n> > \t}\n> >\n\nThanks,\nTaylor\n"},{"id":"391073","messageId":"20200204035940.GD5790@syl.local","threadId":"52725","inReplyTo":"nycvar.QRO.7.76.6.2002012223490.46@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-04T03:59:40Z","receivedAt":"2020-02-04T03:59:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Feb 01, 2020 at 10:25:36PM +0100, Johannes Schindelin wrote:\n> Hi,\n>\n> On Sat, 1 Feb 2020, SZEDER Gábor wrote:\n>\n> > On Thu, Jan 30, 2020 at 04:28:17PM -0800, Taylor Blau wrote:\n> > > diff --git a/commit-graph.c b/commit-graph.c\n> > > index 6d34829f57..02e6ad9d1f 100644\n> > > --- a/commit-graph.c\n> > > +++ b/commit-graph.c\n> > > @@ -1565,15 +1565,18 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n> > > \tnum_commits = ctx->commits.nr;\n> > > \tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n> > >\n> > > -\twhile (g && (g->num_commits <= size_mult * num_commits ||\n> > > -\t\t (max_commits && num_commits > max_commits))) {\n> > > -\t\tif (g->odb != ctx->odb)\n> > > -\t\t\tbreak;\n> > > +\tif (ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n> >\n> > This line segfaults in the tests 'fetch.writeCommitGraph' and\n> > 'fetch.writeCommitGraph with submodules' in 't5510-fetch.sh', because\n> > 'git fetch' doesn't pass a 'split_opts' to the commit-graph functions.\n>\n> I noticed the same. This patch seems to fix it for me:\n>\n> -- snip --\n> diff --git a/commit-graph.c b/commit-graph.c\n> index a5d7624073f..af5c58833cf 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -1566,7 +1566,8 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n> \tnum_commits = ctx->commits.nr;\n> \tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n>\n> -\tif (ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n> +\tif (ctx->split_opts &&\n> +\t ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n> \t\twhile (g && (g->num_commits <= size_mult * num_commits ||\n> \t\t\t (max_commits && num_commits > max_commits) ||\n> \t\t\t (ctx->split_opts->flags == COMMIT_GRAPH_SPLIT_MERGE_REQUIRED))) {\n> -- snap --\n>\n> For your convenience, I also pushed this up as\n> `tb/commit-graph-split-merge` to https://github.com/dscho/git\n\nThanks, Dscho. I also published it under the same name on my fork at\n'https://github.com/ttaylorr/git'.\n\n> Thanks,\n> Dscho\n>\n>\n> >\n> > Thread 1 \"git\" received signal SIGSEGV, Segmentation fault.\n> > 0x00000000005113dd in split_graph_merge_strategy (ctx=0x9ca930)\n> > at commit-graph.c:1568\n> > 1568 if (ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n> > (gdb) p *ctx\n> > $1 = {r = 0x9ae2a0 , odb = 0x9c0df0, graph_name = 0x0, oids = {\n> > list = 0xa02660, nr = 12, alloc = 1024}, commits = {list = 0x9caa00,\n> > nr = 6, alloc = 6}, num_extra_edges = 0, approx_nr_objects = 0,\n> > progress = 0x0, progress_done = 0, progress_cnt = 0, base_graph_name = 0x0,\n> > num_commit_graphs_before = 0, num_commit_graphs_after = 1,\n> > commit_graph_filenames_before = 0x0, commit_graph_filenames_after = 0x0,\n> > commit_graph_hash_after = 0x0, new_num_commits_in_base = 0,\n> > new_base_graph = 0x0, append = 0, report_progress = 1, split = 1,\n> > check_oids = 0, split_opts = 0x0}\n> > ^^^^^^^^^^^^^^^^\n> > (gdb) bt\n> > #0 0x00000000005113dd in split_graph_merge_strategy (ctx=0x9ca930)\n> > at commit-graph.c:1568\n> > #1 0x0000000000512446 in write_commit_graph (odb=0x9c0df0, pack_indexes=0x0,\n> > commit_hex=0x7fffffffd550,\n> > flags=(COMMIT_GRAPH_WRITE_PROGRESS | COMMIT_GRAPH_WRITE_SPLIT),\n> > split_opts=0x0) at commit-graph.c:1891\n> > #2 0x000000000050fd86 in write_commit_graph_reachable (odb=0x9c0df0,\n> > flags=(COMMIT_GRAPH_WRITE_PROGRESS | COMMIT_GRAPH_WRITE_SPLIT),\n> > split_opts=0x0) at commit-graph.c:1174\n> > ^^^^^^^^^^^^^^\n> > #3 0x0000000000444ea4 in cmd_fetch (argc=0, argv=0x7fffffffd9b8, prefix=0x0)\n> > at builtin/fetch.c:1873\n> > #4 0x0000000000406154 in run_builtin (p=0x969a88 , argc=2,\n> > argv=0x7fffffffd9b0) at git.c:444\n> > #5 0x00000000004064a4 in handle_builtin (argc=2, argv=0x7fffffffd9b0)\n> > at git.c:674\n> > #6 0x000000000040674c in run_argv (argcp=0x7fffffffd84c, argv=0x7fffffffd840)\n> > at git.c:741\n> > #7 0x0000000000406bbd in cmd_main (argc=2, argv=0x7fffffffd9b0) at git.c:872\n> > #8 0x00000000004cfd4e in main (argc=5, argv=0x7fffffffd998)\n> > at common-main.c:52\n> >\n> > Note that this function split_graph_merge_strategy() does look at\n> > various fields in 'ctx->split_opts' a bit earlier, but those accesses\n> > are protected by a 'if (ctx->split_opts)' condition.\n> > expire_commit_graphs() does the same.\n> >\n> >\n> > > +\t\twhile (g && (g->num_commits <= size_mult * num_commits ||\n> > > +\t\t\t (max_commits && num_commits > max_commits) ||\n> > > +\t\t\t (ctx->split_opts->flags == COMMIT_GRAPH_SPLIT_MERGE_REQUIRED))) {\n> > > +\t\t\tif (g->odb != ctx->odb)\n> > > +\t\t\t\tbreak;\n> > >\n> > > -\t\tnum_commits += g->num_commits;\n> > > -\t\tg = g->base_graph;\n> > > +\t\t\tnum_commits += g->num_commits;\n> > > +\t\t\tg = g->base_graph;\n> > >\n> > > -\t\tctx->num_commit_graphs_after--;\n> > > +\t\t\tctx->num_commit_graphs_after--;\n> > > +\t\t}\n> > > \t}\n> > >\n> >\n\nThanks,\nTaylor\n"},{"id":"391074","messageId":"20200204040628.GE5790@syl.local","threadId":"52725","inReplyTo":"CAN0heSo29+sf1352EfGv=qFsir-e=Ompja445bh5z3vpOfkaJA@mail.gmail.com","subject":"Re: [PATCH 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-04T04:06:28Z","receivedAt":"2020-02-04T04:06:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jan 31, 2020 at 08:27:27PM +0100, Martin Ågren wrote:\n> On Fri, 31 Jan 2020 at 01:29, Taylor Blau wrote:\n> > With '--split', the commit-graph machinery writes new commits in another\n> > incremental commit-graph which is part of the existing chain, and\n> > optionally decides to condense the chain into a single commit-graph.\n> > This is done to ensure that the aysmptotic behavior of looking up a\n>\n> asymptotic\n\nThanks, fixed.\n\n> > commit in an incremental chain is dominated by the number of\n> > incrementals in that chain. It can be controlled by the '--max-commits'\n> > and '--size-multiple' options.\n> >\n> > On occasion, callers may want to ensure that 'git commit-graph write\n> > --split' always writes an incremental, and never spends effort\n> > condensing the incremental chain [1]. Previously, this was possible by\n> > passing '--size-multiple=0', but this no longer the case following\n> > 63020f175f (commit-graph: prefer default size_mult when given zero,\n> > 2020-01-02).\n> >\n> > Reintroduce a less-magical variant of the above with a new pair of\n> > arguments to '--split': '--split=no-merge' and '--split=merge-all'. When\n> > '--split=no-merge' is given, the commit-graph machinery will never\n> > condense an existing chain and will always write a new incremental.\n> > Conversely, if '--split=merge-all' is given, any invocation including it\n> > will always condense a chain if one exists. If '--split' is given with\n> > no arguments, it behaves as before and defers to '--size-multiple', and\n> > so on.\n>\n> I understand your motivation for doing this -- it all seems quite sound\n> to me. Not being too familiar with this commit-graph splitting and\n> merging, I had a hard time groking this terminology though. From what I\n> understand, before this patch, `--split` means \"write the commit-graph\n> using the 'split' file-format / in a split fashion\". Ok, that makes\n> sense.\n>\n> >From seeing `--split=no-merge`, I have no idea how to even parse that.\n> Of course I don't want to merge, I want to split! Well not split, but\n> write split files.\n>\n> I think it would help me (and others like me) if we could somehow\n> separate \"I want to use 'split' files\" from \"and here's how I want you\n> to decide on the merging\". That is, which \"strategy\" to use. Obviously,\n> talking about a \"merge strategy\" would be stupid and \"split strategy\"\n> also seems a bit odd. \"Coalescing strategy\"? \"Joining strategy\"?\n>\n> Or can you convince me otherwise? From which angle should I look at\n> this?\n\nHeh. This is all very reminiscent of an off-list discussion that I had\nwith Peff and Stolee before sending this upstream. Originally, I had\nimplemented this as:\n\n $ git commit-graph write --split --[no-]merge\n\nbut we decided that this '--merge' and '--no-merge' requiring '--split'\nseemed to indicate that this was better off as an argument to '--split'.\nOf course, there's no getting around that it is... odd to say\n'--split=no-merge' for exactly the reason you suggest.\n\nHere's another way of looking at it: the presence of '--split' means\n\"work with split graph files\" and the '[no-]merge' argument means:\n\"always/never condense multiple layers\".\n\nFor me, this not only makes the new option language jive, but makes it\nclearer to me than the combination of '--split', '--split --no-merge'\nand '--split --merge', where the third one is truly bizarre. At least\ncondensing the second '--' and making 'merge' an argument to 'split'\nmakes it clear that the two work together somehow.\n\nIf you have a different suggestion, I'd certainly love to hear about it\nand discuss. But, at least as far as our internal discussions have gone,\nthis is by far the best option that we have been able to come up with.\n\n> > -With the `--split` option, write the commit-graph as a chain of multiple\n> > -commit-graph files stored in `/info/commit-graphs`. The new commits\n> > -not already in the commit-graph are added in a new \"tip\" file. This file\n> > -is merged with the existing file if the following merge conditions are\n> > -met:\n> > +With the `--split[=]` option, write the commit-graph as a\n> > +chain of multiple commit-graph files stored in\n> > +`/info/commit-graphs`. Commit-graph layers are merged based on the\n> > +strategy and other splitting options. The new commits not already in the\n> > +commit-graph are added in a new \"tip\" file. This file is merged with the\n> > +existing file if the following merge conditions are met:\n> > +* If `--split=merge-always` is specified, then a merge is always\n> > +conducted, and the remaining options are ignored. Conversely, if\n> > +`--split=no-merge` is specified, a merge is never performed, and the\n> > +remaining options are ignored. A bare `--split` defers to the remaining\n> > +options. (Note that merging a chain of commit graphs replaces the\n> > +existing chain with a length-1 chain where the first and only\n> > +incremental holds the entire graph).\n>\n> To better understand the background for this patch, I read the manpage\n> as it stands today. From the section on `--split`, I got this\n> impression: Let's say that `--max-commits` is huge, so all that matters\n> is the `--size-multiple`. Let's say it's two. If the current tip\n> contains three commits and we're about to write one with two, then 2*2 >\n> 3 so we will merge, i.e., write a tip file with five commits. Unless of\n> course *that* is more than half the size of the file before. And so on.\n> We might just merge once, or maybe \"many\" files in an avalanche effect.\n> Every now and then, such avalanches will go all the way back to the\n> first file.\n>\n> Now this says something different, namely that once we decide to merge,\n> we do it all the way back, no matter what.\n>\n> The commit message of 1771be90c8 (\"commit-graph: merge commit-graph\n> chains\", 2019-06-18) seems to support my original understanding, at\n> least for `--size-multiple`, but not `--max-commits`, curiously enough.\n>\n> Can you clarify?\n\n1771be90c8 is right, and this documentation is wrong. Upon re-reading\nit, I found the contents of this documentation between those two\nparenthesis to be confusing rather than helpful. For that reason, I\nsimply removed it.\n\n> > - OPT_BOOL(0, \"split\", &opts.split,\n> > - N_(\"allow writing an incremental commit-graph file\")),\n> > + OPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n> > + N_(\"allow writing an incremental commit-graph file\"),\n>\n> This still sounds very boolean. Cramming in the \"strategy\" might be hard\n> -- is this an argument in favor of having two separate options? ;-)\n\nHeh. Exactly how we started these patches when I originally wrote\nthem...\n\n> > +enum commit_graph_split_flags {\n> > + COMMIT_GRAPH_SPLIT_UNSPECIFIED = 0,\n> > + COMMIT_GRAPH_SPLIT_MERGE_REQUIRED = 1,\n> > + COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED = 2\n> > +};\n>\n> I wonder if this should be \"MERGE_AUTO\" rather than \"UNSPECIFIED\". This\n> is related to Stolee's comment, I think.\n\nI think you're right. I changed it in my local v2.\n>\n> Martin\n\nThanks,\nTaylor\n"},{"id":"391075","messageId":"20200204042129.GF5790@syl.local","threadId":"52725","inReplyTo":"846706e9-efe2-448d-67a3-a96638e9bcbc@gmail.com","subject":"Re: [PATCH 2/3] builtin/commit-graph.c: introduce '--input='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-04T04:21:29Z","receivedAt":"2020-02-04T04:21:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jan 31, 2020 at 09:40:23AM -0500, Derrick Stolee wrote:\n> On 1/30/2020 7:28 PM, Taylor Blau wrote:\n> > The 'write' mode of the 'commit-graph' supports input from a number of\n> > different sources: pack indexes over stdin, commits over stdin, commits\n> > reachable from all references, and so on. Each of these options are\n> > specified with a unique option: '--stdin-packs', '--stdin-commits', etc.\n> >\n> > Similar to our replacement of 'git config [--]' with 'git config\n> > [--type=]' (c.f., fb0dc3bac1 (builtin/config.c: support\n> > `--type=` as preferred alias for `--`, 2018-04-18)), softly\n> > deprecate '[--]' in favor of '[--input=]'.\n> >\n> > This makes it more clear to implement new options that are combinations\n> > of other options (such as, for example, \"none\", a combination of the old\n> > \"--append\" and a new sentinel to specify to _not_ look in other packs,\n> > which we will implement in a future patch).\n> >\n> > Unfortunately, the new enumerated type is a bitfield, even though it\n> > makes much more sense as '0, 1, 2, ...'. Even though *almost* all\n> > options are pairwise exclusive, '--stdin-{packs,commits}' *is*\n> > compatible with '--append'. For this reason, use a bitfield.\n> >\n> > Signed-off-by: Taylor Blau \n> > ---\n> > Documentation/git-commit-graph.txt | 26 +++++-----\n> > builtin/commit-graph.c | 77 ++++++++++++++++++++++--------\n> > t/t5318-commit-graph.sh | 46 +++++++++---------\n> > t/t5324-split-commit-graph.sh | 44 ++++++++---------\n> > 4 files changed, 114 insertions(+), 79 deletions(-)\n> >\n> > diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> > index 8d61ba9f56..cbf80226e9 100644\n> > --- a/Documentation/git-commit-graph.txt\n> > +++ b/Documentation/git-commit-graph.txt\n> > @@ -41,21 +41,21 @@ COMMANDS\n> >\n> > Write a commit-graph file based on the commits found in packfiles.\n> > +\n> > -With the `--stdin-packs` option, generate the new commit graph by\n> > +With the `--input=stdin-packs` option, generate the new commit graph by\n> > walking objects only in the specified pack-indexes. (Cannot be combined\n> > -with `--stdin-commits` or `--reachable`.)\n> > +with `--input=stdin-commits` or `--input=reachable`.)\n> > +\n> > -With the `--stdin-commits` option, generate the new commit graph by\n> > -walking commits starting at the commits specified in stdin as a list\n> > +With the `--input=stdin-commits` option, generate the new commit graph\n> > +by walking commits starting at the commits specified in stdin as a list\n> > of OIDs in hex, one OID per line. (Cannot be combined with\n> > -`--stdin-packs` or `--reachable`.)\n> > +`--input=stdin-packs` or `--input=reachable`.)\n> > +\n> > -With the `--reachable` option, generate the new commit graph by walking\n> > -commits starting at all refs. (Cannot be combined with `--stdin-commits`\n> > -or `--stdin-packs`.)\n> > +With the `--input=reachable` option, generate the new commit graph by\n> > +walking commits starting at all refs. (Cannot be combined with\n> > +`--input=stdin-commits` or `--input=stdin-packs`.)\n> > +\n> > -With the `--append` option, include all commits that are present in the\n> > -existing commit-graph file.\n> > +With the `--input=append` option, include all commits that are present\n> > +in the existing commit-graph file.\n> > +\n> > With the `--split[=]` option, write the commit-graph as a\n> > chain of multiple commit-graph files stored in\n> > @@ -107,20 +107,20 @@ $ git commit-graph write\n> > using commits in ``.\n> > +\n> > ------------------------------------------------\n> > -$ echo | git commit-graph write --stdin-packs\n> > +$ echo | git commit-graph write --input=stdin-packs\n> > ------------------------------------------------\n> >\n> > * Write a commit-graph file containing all reachable commits.\n> > +\n> > ------------------------------------------------\n> > -$ git show-ref -s | git commit-graph write --stdin-commits\n> > +$ git show-ref -s | git commit-graph write --input=stdin-commits\n> > ------------------------------------------------\n> >\n> > * Write a commit-graph file containing all commits in the current\n> > commit-graph file along with those reachable from `HEAD`.\n> > +\n> > ------------------------------------------------\n> > -$ git rev-parse HEAD | git commit-graph write --stdin-commits --append\n> > +$ git rev-parse HEAD | git commit-graph write --input=stdin-commits --input=append\n> > ------------------------------------------------\n> >\n> >\n> > diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> > index f03b46d627..03d815e652 100644\n> > --- a/builtin/commit-graph.c\n> > +++ b/builtin/commit-graph.c\n> > @@ -10,7 +10,7 @@\n> > static char const * const builtin_commit_graph_usage[] = {\n> > \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n> > \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n> > -\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n> > +\t \"[--split[=]] [--input=] \"\n> > \t \"[--[no-]progress] \"),\n> > \tNULL\n> > };\n> > @@ -22,22 +22,48 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n> >\n> > static const char * const builtin_commit_graph_write_usage[] = {\n> > \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n> > -\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n> > +\t \"[--split[=]] [--input=] \"\n> > \t \"[--[no-]progress] \"),\n> > \tNULL\n> > };\n> >\n> > +enum commit_graph_input {\n> > +\tCOMMIT_GRAPH_INPUT_REACHABLE = (1 << 1),\n> > +\tCOMMIT_GRAPH_INPUT_STDIN_PACKS = (1 << 2),\n> > +\tCOMMIT_GRAPH_INPUT_STDIN_COMMITS = (1 << 3),\n> > +\tCOMMIT_GRAPH_INPUT_APPEND = (1 << 4)\n> > +};\n> > +\n> > static struct opts_commit_graph {\n> > \tconst char *obj_dir;\n> > -\tint reachable;\n> > -\tint stdin_packs;\n> > -\tint stdin_commits;\n> > -\tint append;\n> > +\tenum commit_graph_input input;\n> > \tint split;\n> > \tint shallow;\n> > \tint progress;\n> > } opts;\n> >\n> > +static int option_parse_input(const struct option *opt, const char *arg,\n> > +\t\t\t int unset)\n> > +{\n> > +\tenum commit_graph_input *to = opt->value;\n> > +\tif (unset || !strcmp(arg, \"packs\")) {\n> > +\t\t*to = 0;\n> > +\t\treturn 0;\n> > +\t}\n>\n> Here, you _do_ clear the bitfield, allowing \"--input=reachable --input\"\n> to do the correct override. Thanks!\n>\n> > +\n> > +\tif (!strcmp(arg, \"reachable\"))\n> > +\t\t*to |= COMMIT_GRAPH_INPUT_REACHABLE;\n> > +\telse if (!strcmp(arg, \"stdin-packs\"))\n> > +\t\t*to |= COMMIT_GRAPH_INPUT_STDIN_PACKS;\n> > +\telse if (!strcmp(arg, \"stdin-commits\"))\n> > +\t\t*to |= COMMIT_GRAPH_INPUT_STDIN_COMMITS;\n> > +\telse if (!strcmp(arg, \"append\"))\n> > +\t\t*to |= COMMIT_GRAPH_INPUT_APPEND;\n> > +\telse\n> > +\t\tdie(_(\"unrecognized --input source, %s\"), arg);\n> > +\treturn 0;\n> > +}\n> > +\n> > static struct object_directory *find_odb_or_die(struct repository *r,\n> > \t\t\t\t\t\tconst char *obj_dir)\n> > {\n> > @@ -137,14 +163,21 @@ static int graph_write(int argc, const char **argv)\n> > \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n> > \t\t\tN_(\"dir\"),\n> > \t\t\tN_(\"The object directory to store the graph\")),\n> > -\t\tOPT_BOOL(0, \"reachable\", &opts.reachable,\n> > -\t\t\tN_(\"start walk at all refs\")),\n> > -\t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n> > -\t\t\tN_(\"scan pack-indexes listed by stdin for commits\")),\n> > -\t\tOPT_BOOL(0, \"stdin-commits\", &opts.stdin_commits,\n> > -\t\t\tN_(\"start walk at commits listed by stdin\")),\n> > -\t\tOPT_BOOL(0, \"append\", &opts.append,\n> > -\t\t\tN_(\"include all commits already in the commit-graph file\")),\n> > +\t\tOPT_CALLBACK(0, \"input\", &opts.input, NULL,\n> > +\t\t\tN_(\"include commits from this source in the graph\"),\n> > +\t\t\toption_parse_input),\n> > +\t\tOPT_BIT(0, \"reachable\", &opts.input,\n> > +\t\t\tN_(\"start walk at all refs\"),\n> > +\t\t\tCOMMIT_GRAPH_INPUT_REACHABLE),\n> > +\t\tOPT_BIT(0, \"stdin-packs\", &opts.input,\n> > +\t\t\tN_(\"scan pack-indexes listed by stdin for commits\"),\n> > +\t\t\tCOMMIT_GRAPH_INPUT_STDIN_PACKS),\n> > +\t\tOPT_BIT(0, \"stdin-commits\", &opts.input,\n> > +\t\t\tN_(\"start walk at commits listed by stdin\"),\n> > +\t\t\tCOMMIT_GRAPH_INPUT_STDIN_COMMITS),\n> > +\t\tOPT_BIT(0, \"append\", &opts.input,\n> > +\t\t\tN_(\"include all commits already in the commit-graph file\"),\n> > +\t\t\tCOMMIT_GRAPH_INPUT_APPEND),\n>\n> Since you are rewriting how we interpret the deprecated options, perhaps we\n> should keep some tests around that call these versions? It would make the\n> test diff be a bit smaller. These options can be removed from the tests if/when\n> we actually remove the options.\n\nThat sounds good. I thought about doing this in the original round, but\nI talked myself out of it because it wasn't clear to me which tests were\nthe ones worth converting and which should be left alone.\n\nBut, since you think it's good, so do I. I picked the ones to convert\nmostly at random, and left the new ones as-is using the '--input=' form.\n\n> > @@ -351,10 +351,10 @@ test_expect_success '--split=merge-all always merges incrementals' '\n> > \tgit rev-list -3 HEAD~4 >a &&\n> > \tgit rev-list -2 HEAD~2 >b &&\n> > \tgit rev-list -2 HEAD >c &&\n> > -\tgit commit-graph write --split=no-merge --stdin-commits > -\tgit commit-graph write --split=no-merge --stdin-commits > +\tgit commit-graph write --split=no-merge --input=stdin-commits > +\tgit commit-graph write --split=no-merge --input=stdin-commits > \ttest_line_count = 2 $graphdir/commit-graph-chain &&\n> > -\tgit commit-graph write --split=merge-all --stdin-commits > +\tgit commit-graph write --split=merge-all --input=stdin-commits > \ttest_line_count = 1 $graphdir/commit-graph-chain\n> > '\n> >\n> > @@ -364,8 +364,8 @@ test_expect_success '--split=no-merge always writes an incremental' '\n> > \tgit reset --hard commits/2 &&\n> > \tgit rev-list HEAD~1 >a &&\n> > \tgit rev-list HEAD >b &&\n> > -\tgit commit-graph write --split --stdin-commits > -\tgit commit-graph write --split=no-merge --stdin-commits > +\tgit commit-graph write --split --input=stdin-commits > +\tgit commit-graph write --split=no-merge --input=stdin-commits > \ttest_line_count = 2 $graphdir/commit-graph-chain\n> > '\n>\n> Updating these new tests with the given options is good. Perhaps convert only one\n> of the old tests for each of the stdin-packs, reachable, \"\", and \"append\" options?\n\nYup, thanks.\n\n> Thanks,\n> -Stolee\n\nThanks,\nTaylor\n"},{"id":"391076","messageId":"20200204045124.GG5790@syl.local","threadId":"52725","inReplyTo":"CAN0heSqGe0HpAj+qoYkif4pyjaW5EBNsRce9OmxMaZXTSW7C9w@mail.gmail.com","subject":"Re: [PATCH 2/3] builtin/commit-graph.c: introduce '--input='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-04T04:51:24Z","receivedAt":"2020-02-04T04:51:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jan 31, 2020 at 08:34:41PM +0100, Martin Ågren wrote:\n> On Fri, 31 Jan 2020 at 01:30, Taylor Blau wrote:\n> > The 'write' mode of the 'commit-graph' supports input from a number of\n> > different sources: pack indexes over stdin, commits over stdin, commits\n> > reachable from all references, and so on. Each of these options are\n> > specified with a unique option: '--stdin-packs', '--stdin-commits', etc.\n> >\n> > Similar to our replacement of 'git config [--]' with 'git config\n> > [--type=]' (c.f., fb0dc3bac1 (builtin/config.c: support\n> > `--type=` as preferred alias for `--`, 2018-04-18)), softly\n> > deprecate '[--]' in favor of '[--input=]'.\n> >\n> > This makes it more clear to implement new options that are combinations\n> > of other options (such as, for example, \"none\", a combination of the old\n> > \"--append\" and a new sentinel to specify to _not_ look in other packs,\n> > which we will implement in a future patch).\n>\n> Makes sense.\n>\n> > Unfortunately, the new enumerated type is a bitfield, even though it\n> > makes much more sense as '0, 1, 2, ...'. Even though *almost* all\n> > options are pairwise exclusive, '--stdin-{packs,commits}' *is*\n> > compatible with '--append'. For this reason, use a bitfield.\n>\n> > -With the `--append` option, include all commits that are present in the\n> > -existing commit-graph file.\n> > +With the `--input=append` option, include all commits that are present\n> > +in the existing commit-graph file.\n>\n> Would it be too crazy to call this `--input=existing` instead, and have\n> it be the same as `--append`? I find that `--append` makes a lot of\n> sense (it's a mode we can turn on or off), whereas \"input = append\"\n> seems more odd.\n\nHmm. When I wrote this, I was thinking of introducing equivalent options\nthat are identical in name and functionality as '--input=' instead\nof '--'. So, I guess that is to say that I didn't spend an awful\namount of time thinking about whether or not '--input=append' made sense\ngiven anything else.\n\nSo, I don't think that '--input=existing' is a bad idea at all, but I do\nworry about advertising this deprecation as \"'--' becomes\n'--input=', except when your mode is 'append', in which case it\nbecomes '--input=existing'\".\n\nI suppose that, on the other hand, if we *were* to introduce such a\nchange, now would be the time to do it, before '--input=' is on\nmaster and tagged in a release, but I'm not sure that '--input=append'\nis so much worse.\n\nI'm inclined to leave it as is, unless there are others that feel\nstrongly, in which case we can/should come back to it before this moves\ntowards being queued.\n\n> >From the next commit message, we learn that a long `--input=append`\n> triggers `fill_oids_from_all_packs()`, which wouldn't match my expecting\n> from \"--input=existing\". So...\n>\n> Does this hint that we could leave `--append` alone? We'd have lots of\n> different inputs to choose from using `--input`, and an `--append` mode\n> on top of that. That would make your inputs truly mutually exclusive and\n> you don't need the bitfield anymore, as you mention above. Hmm?\n>\n> Would that mean that the falling back to `fill_oids_from_all_packs()`\n> would follow from \"is there an --input?\", as opposed to from \"is there\n> an --input except --input=append?\"?\n>\n> (I don't know whether these inputs really *have* to be exclusive, or if\n> that's more of an implementation detail. That is, even without an\n> \"append\" input, might we one day be able to handle more inputs at once?\n> Maybe this is not the time to worry about that.)\n>\n> Martin\n\nThanks,\nTaylor\n"},{"id":"391077","messageId":"20200204050110.GH5790@syl.local","threadId":"52725","inReplyTo":"CAN0heSpfqMChHO=iNuUExrBsMPcpZ6K2zH68GkHO2ttEiSinSA@mail.gmail.com","subject":"Re: [PATCH 3/3] builtin/commit-graph.c: support '--input=none'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-04T05:01:10Z","receivedAt":"2020-02-04T05:01:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jan 31, 2020 at 08:45:59PM +0100, Martin Ågren wrote:\n> On Fri, 31 Jan 2020 at 01:30, Taylor Blau wrote:\n> > In the previous commit, we introduced '--[no-]merge', and alluded to the\n> > fact that '--merge' would be useful for callers who wish to always\n> > trigger a merge of an incremental chain.\n>\n> Hmmm. So it looks like you've already had similar thoughts as I did\n> about patch 1/3. At some point, you had a separate `--merge=...` option,\n> then later made that `--split=...`. :-) Could you say something about why\n> you changed your mind?\n\nHeh :-). Left overs from an earlier version of this series. I think that\nI already talked about why this was changed further up in the thread.\n\n> > There is a problem with the above approach, which is that there is no\n> > way to specify to the commit-graph builtin that a caller only wants to\n> > include commits already in the graph. One can specify '--input=append'\n> > to include all commits in the existing graphs, but the absence of\n> > '--input=stdin-{commits,packs}' causes the builtin to call\n> > 'fill_oids_from_all_packs()'.\n>\n> (Use one of those options with an empty stdin? Anyway, let's read on.)\n>\n> > Passing '--input=reachable' (as in 'git commit-graph write\n> > --split=merge-all --input=reachable --input=append') works around this\n> > issue by making '--input=reachable' effectively a no-op, but this can be\n> > prohibitively expensive in large repositories, making it an undesirable\n> > choice for some users.\n> >\n> > Teach '--input=none' as an option to behave as if '--input=append' were\n> > given, but to consider no other sources in addition.\n>\n> `--input=none` almost makes me wonder if it should produce an empty\n> commit-graph. But there wouldn't be much point in that... I guess\n> another way of defining this would be that it \"uses no input, and\n> implies `--append`\".\n\nI suppose, although (like you) I can't imagine why anybody would want to\ndo that.\n\n> > This, in conjunction with the option introduced in the previous patch\n> > offers the convenient way to force the commit-graph machinery to\n> > condense a chain of incrementals without requiring any new commits:\n> >\n> > $ git commit-graph write --split=merge-all --input=none\n>\n> Right.\n>\n> > --- a/Documentation/git-commit-graph.txt\n> > +++ b/Documentation/git-commit-graph.txt\n> > @@ -39,24 +39,29 @@ COMMANDS\n> > --------\n> > 'write'::\n> >\n> > -Write a commit-graph file based on the commits found in packfiles.\n> > +Write a commit-graph file based on the commits specified:\n> > +* With the `--input=stdin-packs` option, generate the new commit graph\n> > +by walking objects only in the specified pack-indexes. (Cannot be\n> > +combined with `--input=stdin-commits` or `--input=reachable`.)\n> > +\n> > -With the `--input=stdin-packs` option, generate the new commit graph by\n> > -walking objects only in the specified pack-indexes. (Cannot be combined\n> > -with `--input=stdin-commits` or `--input=reachable`.)\n> > -+\n> > -With the `--input=stdin-commits` option, generate the new commit graph\n> > +* With the `--input=stdin-commits` option, generate the new commit graph\n> > by walking commits starting at the commits specified in stdin as a list\n> > of OIDs in hex, one OID per line. (Cannot be combined with\n> > `--input=stdin-packs` or `--input=reachable`.)\n> > +\n> > -With the `--input=reachable` option, generate the new commit graph by\n> > +* With the `--input=reachable` option, generate the new commit graph by\n> > walking commits starting at all refs. (Cannot be combined with\n> > `--input=stdin-commits` or `--input=stdin-packs`.)\n> > +\n> > -With the `--input=append` option, include all commits that are present\n> > +* With the `--input=append` option, include all commits that are present\n> > in the existing commit-graph file.\n>\n> Do these changes above really belong in this commit?\n\nI think so. My thought here was to leave this documentation as-is until\nthis patch, when adding '--input=none' would... somehow change this, but\ntrying to construct a reply, I can't seem to come up with why I thought\nthat this was a good idea in the first place ;-).\n\n> > +* With the `--input=none` option, behave as if `input=append` were\n> > +given, but do not walk other packs to find additional commits.\n> > +\n> > +If none of the above options are given, then commits found in\n> > +packfiles are specified.\n>\n> \"specified\"? Plus, that also happens for `--input=append` right? (It\n> really seems like \"append\" is an odd one among all the inputs.)\n\nI reworded this slightly to not use \"specified\", which I agree is indeed\nweird.\n\n>\n> > N_(\"git commit-graph write [--object-dir ] [--append] \"\n> > - \"[--split[=]] [--input=] \"\n> > + \"[--split[=]] \"\n> > + \"[--input=] \"\n> > \"[--[no-]progress] \"),\n>\n> Hmm, you've left \"--append\" the old way.\n\nFixed, and thanks for noticing.\n>\n> Martin\n\nThanks,\nTaylor\n"},{"id":"391082","messageId":"20200204141441.GA563710@coredump.intra.peff.net","threadId":"52725","inReplyTo":"20200204035821.GB5790@syl.local","subject":"Re: [PATCH 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-02-04T14:14:41Z","receivedAt":"2020-02-04T14:14:47Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Feb 03, 2020 at 07:58:21PM -0800, Taylor Blau wrote:\n\n> I think that this seems like a good step that we should probably take,\n> but I don't think that it's necessary for the series at hand. The\n> pattern in this function is to define a local variable which has the\n> same value as in split_opts, or a reasonable default if split_opts is\n> NULL (c.f., 'max_commits' and 'size_mult').\n> \n> So, I think that a safe thing to do which prevents the segv and doesn't\n> change the pattern too much is to write:\n> \n> enum commit_graph_split_flags flags = COMMIT_GRAPH_SPLIT_MERGE_AUTO;\n> if (ctx->split_opts) {\n> /* ... */\n> flags = ctx->split_opts->flags;\n> }\n> \n> /* ... */\n> \n> if (flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n> while ( ... ) { ... }\n> }\n> \n> This is adding another local variable, which seems like an odd thing to\n> do *every* time that we add another member to split_opts. So for that\n> reason it seems like in the longer-term we should either force the\n> caller to pass in a blank, or do something else that doesn't require\n> this, but I think that the intermediate cost isn't too bad.\n\nIt would perhaps be simpler to turn NULL into a _single_ default-filled\nsplit_opts variable, and then just use it throughout the function. And\nsince presumably zero-initialization would yield good defaults (or we'd\ndefine an INIT macro for the convenience of callers), it would be a\none-liner that we'd only have to do once.\n\nBut I think that can wait; the \"if\" solution discussed seems like a\nstraightforward way to make this patch correct both on top of master,\nand when merged with next.\n\n-Peff\n"},{"id":"391137","messageId":"xmqq5zgl7waj.fsf@gitster-ct.c.googlers.com","threadId":"52725","inReplyTo":"cover.1580430057.git.me@ttaylorr.com","subject":"Re: [PATCH 0/3] builtin/commit-graph.c: new split/merge options","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-02-04T23:44:04Z","receivedAt":"2020-02-04T23:44:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"As the topoic this depends on has been updated, I tried to rebase\nthis on top, but I am seeing segfaults in tests. We'd probably need\na fresh round of this one to replace it.\n\nThanks.\n\n"},{"id":"391138","messageId":"3e19d50148c8d53b30f8f0036a2d3af9f4bb3499.1580862307.git.me@ttaylorr.com","threadId":"52725","inReplyTo":"cover.1580862307.git.me@ttaylorr.com","subject":"[PATCH v2 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-05T00:28:32Z","receivedAt":"2020-02-05T00:28:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"With '--split', the commit-graph machinery writes new commits in another\nincremental commit-graph which is part of the existing chain, and\noptionally decides to condense the chain into a single commit-graph.\nThis is done to ensure that the asymptotic behavior of looking up a\ncommit in an incremental chain is dominated by the number of\nincrementals in that chain. It can be controlled by the '--max-commits'\nand '--size-multiple' options.\n\nOn occasion, callers may want to ensure that 'git commit-graph write\n--split' always writes an incremental, and never spends effort\ncondensing the incremental chain [1]. Previously, this was possible by\npassing '--size-multiple=0', but this no longer the case following\n63020f175f (commit-graph: prefer default size_mult when given zero,\n2020-01-02).\n\nReintroduce a less-magical variant of the above with a new pair of\narguments to '--split': '--split=no-merge' and '--split=merge-all'. When\n'--split=no-merge' is given, the commit-graph machinery will never\ncondense an existing chain and will always write a new incremental.\nConversely, if '--split=merge-all' is given, any invocation including it\nwill always condense a chain if one exists. If '--split' is given with\nno arguments, it behaves as before and defers to '--size-multiple', and\nso on.\n\n[1]: This might occur when, for example, a server administrator running\nsome program after each push may want to ensure that each job runs\nproportional in time to the size of the push, and does not \"jump\" when\nthe commit-graph machinery decides to trigger a merge.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-commit-graph.txt | 16 +++++++++-----\n builtin/commit-graph.c | 35 ++++++++++++++++++++++++++----\n commit-graph.c | 22 ++++++++++++-------\n commit-graph.h | 7 ++++++\n t/t5324-split-commit-graph.sh | 25 +++++++++++++++++++++\n 5 files changed, 88 insertions(+), 17 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 28d1fee505..b7fe65ef21 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -57,11 +57,17 @@ or `--stdin-packs`.)\n With the `--append` option, include all commits that are present in the\n existing commit-graph file.\n +\n-With the `--split` option, write the commit-graph as a chain of multiple\n-commit-graph files stored in `/info/commit-graphs`. The new commits\n-not already in the commit-graph are added in a new \"tip\" file. This file\n-is merged with the existing file if the following merge conditions are\n-met:\n+With the `--split[=]` option, write the commit-graph as a\n+chain of multiple commit-graph files stored in\n+`/info/commit-graphs`. Commit-graph layers are merged based on the\n+strategy and other splitting options. The new commits not already in the\n+commit-graph are added in a new \"tip\" file. This file is merged with the\n+existing file if the following merge conditions are met:\n+* If `--split=merge-always` is specified, then a merge is always\n+conducted, and the remaining options are ignored. Conversely, if\n+`--split=no-merge` is specified, a merge is never performed, and the\n+remaining options are ignored. A bare `--split` defers to the remaining\n+options.\n +\n * If `--size-multiple=` is not specified, let `X` equal 2. If the new\n tip file would have `N` commits and the previous tip has `M` commits and\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 4a70b33fb5..4d3c1c46c2 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -9,7 +9,9 @@\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] [--[no-]progress] \"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] \"\n+\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n+\t \"[--[no-]progress] \"),\n \tNULL\n };\n \n@@ -19,7 +21,9 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] [--[no-]progress] \"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] \"\n+\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n+\t \"[--[no-]progress] \"),\n \tNULL\n };\n \n@@ -111,6 +115,27 @@ static int graph_verify(int argc, const char **argv)\n extern int read_replace_refs;\n static struct split_commit_graph_opts split_opts;\n \n+static int write_option_parse_split(const struct option *opt, const char *arg,\n+\t\t\t\t int unset)\n+{\n+\tenum commit_graph_split_flags *flags = opt->value;\n+\n+\topts.split = 1;\n+\tif (!arg) {\n+\t\t*flags = COMMIT_GRAPH_SPLIT_MERGE_AUTO;\n+\t\treturn 0;\n+\t}\n+\n+\tif (!strcmp(arg, \"merge-all\"))\n+\t\t*flags = COMMIT_GRAPH_SPLIT_MERGE_REQUIRED;\n+\telse if (!strcmp(arg, \"no-merge\"))\n+\t\t*flags = COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED;\n+\telse\n+\t\tdie(_(\"unrecognized --split argument, %s\"), arg);\n+\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tstruct string_list *pack_indexes = NULL;\n@@ -133,8 +158,10 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_BOOL(0, \"append\", &opts.append,\n \t\t\tN_(\"include all commits already in the commit-graph file\")),\n \t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n-\t\tOPT_BOOL(0, \"split\", &opts.split,\n-\t\t\tN_(\"allow writing an incremental commit-graph file\")),\n+\t\tOPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n+\t\t\tN_(\"allow writing an incremental commit-graph file\"),\n+\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n+\t\t\twrite_option_parse_split),\n \t\tOPT_INTEGER(0, \"max-commits\", &split_opts.max_commits,\n \t\t\tN_(\"maximum number of commits in a non-base split commit-graph\")),\n \t\tOPT_INTEGER(0, \"size-multiple\", &split_opts.size_multiple,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 656dd647d5..3a5cb23cd7 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1533,27 +1533,33 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \n \tint max_commits = 0;\n \tint size_mult = 2;\n+\tenum commit_graph_split_flags flags = COMMIT_GRAPH_SPLIT_MERGE_AUTO;\n \n \tif (ctx->split_opts) {\n \t\tmax_commits = ctx->split_opts->max_commits;\n \n \t\tif (ctx->split_opts->size_multiple)\n \t\t\tsize_mult = ctx->split_opts->size_multiple;\n+\n+\t\tflags = ctx->split_opts->flags;\n \t}\n \n \tg = ctx->r->objects->commit_graph;\n \tnum_commits = ctx->commits.nr;\n \tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n \n-\twhile (g && (g->num_commits <= size_mult * num_commits ||\n-\t\t (max_commits && num_commits > max_commits))) {\n-\t\tif (g->odb != ctx->odb)\n-\t\t\tbreak;\n+\tif (flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n+\t\twhile (g && (g->num_commits <= size_mult * num_commits ||\n+\t\t\t (max_commits && num_commits > max_commits) ||\n+\t\t\t (flags == COMMIT_GRAPH_SPLIT_MERGE_REQUIRED))) {\n+\t\t\tif (g->odb != ctx->odb)\n+\t\t\t\tbreak;\n \n-\t\tnum_commits += g->num_commits;\n-\t\tg = g->base_graph;\n+\t\t\tnum_commits += g->num_commits;\n+\t\t\tg = g->base_graph;\n \n-\t\tctx->num_commit_graphs_after--;\n+\t\t\tctx->num_commit_graphs_after--;\n+\t\t}\n \t}\n \n \tctx->new_base_graph = g;\n@@ -1861,7 +1867,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\tgoto cleanup;\n \t}\n \n-\tif (!ctx->commits.nr)\n+\tif (!ctx->commits.nr && (!ctx->split_opts || ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_REQUIRED))\n \t\tgoto cleanup;\n \n \tif (ctx->split) {\ndiff --git a/commit-graph.h b/commit-graph.h\nindex e87a6f6360..65a7d2edae 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -82,10 +82,17 @@ enum commit_graph_write_flags {\n \tCOMMIT_GRAPH_WRITE_CHECK_OIDS = (1 << 3)\n };\n \n+enum commit_graph_split_flags {\n+\tCOMMIT_GRAPH_SPLIT_MERGE_AUTO = 0,\n+\tCOMMIT_GRAPH_SPLIT_MERGE_REQUIRED = 1,\n+\tCOMMIT_GRAPH_SPLIT_MERGE_PROHIBITED = 2\n+};\n+\n struct split_commit_graph_opts {\n \tint size_multiple;\n \tint max_commits;\n \ttimestamp_t expire_time;\n+\tenum commit_graph_split_flags flags;\n };\n \n /*\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex c24823431f..a165b48afe 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -344,4 +344,29 @@ test_expect_success 'split across alternate where alternate is not split' '\n \ttest_cmp commit-graph .git/objects/info/commit-graph\n '\n \n+test_expect_success '--split=merge-all always merges incrementals' '\n+\ttest_when_finished rm -rf a b c &&\n+\trm -rf $graphdir $infodir/commit-graph &&\n+\tgit reset --hard commits/10 &&\n+\tgit rev-list -3 HEAD~4 >a &&\n+\tgit rev-list -2 HEAD~2 >b &&\n+\tgit rev-list -2 HEAD >c &&\n+\tgit commit-graph write --split=no-merge --stdin-commits a &&\n+\tgit rev-list HEAD >b &&\n+\tgit commit-graph write --split --stdin-commits '","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-05T00:28:34Z","receivedAt":"2020-02-05T00:28:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The 'write' mode of the 'commit-graph' supports input from a number of\ndifferent sources: pack indexes over stdin, commits over stdin, commits\nreachable from all references, and so on. Each of these options are\nspecified with a unique option: '--stdin-packs', '--stdin-commits', etc.\n\nSimilar to our replacement of 'git config [--]' with 'git config\n[--type=]' (c.f., fb0dc3bac1 (builtin/config.c: support\n`--type=` as preferred alias for `--`, 2018-04-18)), softly\ndeprecate '[--]' in favor of '[--input=]'.\n\nThis makes it more clear to implement new options that are combinations\nof other options (such as, for example, \"none\", a combination of the old\n\"--append\" and a new sentinel to specify to _not_ look in other packs,\nwhich we will implement in a future patch).\n\nUnfortunately, the new enumerated type is a bitfield, even though it\nmakes much more sense as '0, 1, 2, ...'. Even though *almost* all\noptions are pairwise exclusive, '--stdin-{packs,commits}' *is*\ncompatible with '--append'. For this reason, use a bitfield.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-commit-graph.txt | 26 +++++-----\n builtin/commit-graph.c | 83 +++++++++++++++++++++---------\n t/t5318-commit-graph.sh | 4 +-\n t/t5324-split-commit-graph.sh | 2 +-\n 4 files changed, 76 insertions(+), 39 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex b7fe65ef21..2ae9de679a 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -41,21 +41,21 @@ COMMANDS\n \n Write a commit-graph file based on the commits found in packfiles.\n +\n-With the `--stdin-packs` option, generate the new commit graph by\n+With the `--input=stdin-packs` option, generate the new commit graph by\n walking objects only in the specified pack-indexes. (Cannot be combined\n-with `--stdin-commits` or `--reachable`.)\n+with `--input=stdin-commits` or `--input=reachable`.)\n +\n-With the `--stdin-commits` option, generate the new commit graph by\n-walking commits starting at the commits specified in stdin as a list\n+With the `--input=stdin-commits` option, generate the new commit graph\n+by walking commits starting at the commits specified in stdin as a list\n of OIDs in hex, one OID per line. (Cannot be combined with\n-`--stdin-packs` or `--reachable`.)\n+`--input=stdin-packs` or `--input=reachable`.)\n +\n-With the `--reachable` option, generate the new commit graph by walking\n-commits starting at all refs. (Cannot be combined with `--stdin-commits`\n-or `--stdin-packs`.)\n+With the `--input=reachable` option, generate the new commit graph by\n+walking commits starting at all refs. (Cannot be combined with\n+`--input=stdin-commits` or `--input=stdin-packs`.)\n +\n-With the `--append` option, include all commits that are present in the\n-existing commit-graph file.\n+With the `--input=append` option, include all commits that are present\n+in the existing commit-graph file.\n +\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n@@ -105,20 +105,20 @@ $ git commit-graph write\n using commits in ``.\n +\n ------------------------------------------------\n-$ echo | git commit-graph write --stdin-packs\n+$ echo | git commit-graph write --input=stdin-packs\n ------------------------------------------------\n \n * Write a commit-graph file containing all reachable commits.\n +\n ------------------------------------------------\n-$ git show-ref -s | git commit-graph write --stdin-commits\n+$ git show-ref -s | git commit-graph write --input=stdin-commits\n ------------------------------------------------\n \n * Write a commit-graph file containing all commits in the current\n commit-graph file along with those reachable from `HEAD`.\n +\n ------------------------------------------------\n-$ git rev-parse HEAD | git commit-graph write --stdin-commits --append\n+$ git rev-parse HEAD | git commit-graph write --input=stdin-commits --input=append\n ------------------------------------------------\n \n \ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 4d3c1c46c2..0ff25896d0 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -9,8 +9,9 @@\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append] \"\n-\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n+\tN_(\"git commit-graph write [--object-dir ] \"\n+\t \"[--split[=]] \"\n+\t \"[--input=] \"\n \t \"[--[no-]progress] \"),\n \tNULL\n };\n@@ -21,18 +22,23 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append] \"\n-\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n+\tN_(\"git commit-graph write [--object-dir ] \"\n+\t \"[--split[=]] \"\n+\t \"[--input=] \"\n \t \"[--[no-]progress] \"),\n \tNULL\n };\n \n+enum commit_graph_input {\n+\tCOMMIT_GRAPH_INPUT_REACHABLE = (1 << 1),\n+\tCOMMIT_GRAPH_INPUT_STDIN_PACKS = (1 << 2),\n+\tCOMMIT_GRAPH_INPUT_STDIN_COMMITS = (1 << 3),\n+\tCOMMIT_GRAPH_INPUT_APPEND = (1 << 4)\n+};\n+\n static struct opts_commit_graph {\n \tconst char *obj_dir;\n-\tint reachable;\n-\tint stdin_packs;\n-\tint stdin_commits;\n-\tint append;\n+\tenum commit_graph_input input;\n \tint split;\n \tint shallow;\n \tint progress;\n@@ -57,6 +63,28 @@ static struct object_directory *find_odb(struct repository *r,\n \treturn odb;\n }\n \n+static int option_parse_input(const struct option *opt, const char *arg,\n+\t\t\t int unset)\n+{\n+\tenum commit_graph_input *to = opt->value;\n+\tif (unset || !strcmp(arg, \"packs\")) {\n+\t\t*to = 0;\n+\t\treturn 0;\n+\t}\n+\n+\tif (!strcmp(arg, \"reachable\"))\n+\t\t*to |= COMMIT_GRAPH_INPUT_REACHABLE;\n+\telse if (!strcmp(arg, \"stdin-packs\"))\n+\t\t*to |= COMMIT_GRAPH_INPUT_STDIN_PACKS;\n+\telse if (!strcmp(arg, \"stdin-commits\"))\n+\t\t*to |= COMMIT_GRAPH_INPUT_STDIN_COMMITS;\n+\telse if (!strcmp(arg, \"append\"))\n+\t\t*to |= COMMIT_GRAPH_INPUT_APPEND;\n+\telse\n+\t\tdie(_(\"unrecognized --input source, %s\"), arg);\n+\treturn 0;\n+}\n+\n static int graph_verify(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n@@ -149,14 +177,21 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n \t\t\tN_(\"dir\"),\n \t\t\tN_(\"The object directory to store the graph\")),\n-\t\tOPT_BOOL(0, \"reachable\", &opts.reachable,\n-\t\t\tN_(\"start walk at all refs\")),\n-\t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n-\t\t\tN_(\"scan pack-indexes listed by stdin for commits\")),\n-\t\tOPT_BOOL(0, \"stdin-commits\", &opts.stdin_commits,\n-\t\t\tN_(\"start walk at commits listed by stdin\")),\n-\t\tOPT_BOOL(0, \"append\", &opts.append,\n-\t\t\tN_(\"include all commits already in the commit-graph file\")),\n+\t\tOPT_CALLBACK(0, \"input\", &opts.input, NULL,\n+\t\t\tN_(\"include commits from this source in the graph\"),\n+\t\t\toption_parse_input),\n+\t\tOPT_BIT(0, \"reachable\", &opts.input,\n+\t\t\tN_(\"start walk at all refs\"),\n+\t\t\tCOMMIT_GRAPH_INPUT_REACHABLE),\n+\t\tOPT_BIT(0, \"stdin-packs\", &opts.input,\n+\t\t\tN_(\"scan pack-indexes listed by stdin for commits\"),\n+\t\t\tCOMMIT_GRAPH_INPUT_STDIN_PACKS),\n+\t\tOPT_BIT(0, \"stdin-commits\", &opts.input,\n+\t\t\tN_(\"start walk at commits listed by stdin\"),\n+\t\t\tCOMMIT_GRAPH_INPUT_STDIN_COMMITS),\n+\t\tOPT_BIT(0, \"append\", &opts.input,\n+\t\t\tN_(\"include all commits already in the commit-graph file\"),\n+\t\t\tCOMMIT_GRAPH_INPUT_APPEND),\n \t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n \t\tOPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n \t\t\tN_(\"allow writing an incremental commit-graph file\"),\n@@ -182,11 +217,13 @@ static int graph_write(int argc, const char **argv)\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\n \n-\tif (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)\n-\t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\n+\tif ((!!(opts.input & COMMIT_GRAPH_INPUT_REACHABLE) +\n+\t !!(opts.input & COMMIT_GRAPH_INPUT_STDIN_PACKS) +\n+\t !!(opts.input & COMMIT_GRAPH_INPUT_STDIN_COMMITS)) > 1)\n+\t\tdie(_(\"use at most one of --input=reachable, --input=stdin-commits, or --input=stdin-packs\"));\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n-\tif (opts.append)\n+\tif (opts.input & COMMIT_GRAPH_INPUT_APPEND)\n \t\tflags |= COMMIT_GRAPH_WRITE_APPEND;\n \tif (opts.split)\n \t\tflags |= COMMIT_GRAPH_WRITE_SPLIT;\n@@ -196,22 +233,22 @@ static int graph_write(int argc, const char **argv)\n \tread_replace_refs = 0;\n \todb = find_odb(the_repository, opts.obj_dir);\n \n-\tif (opts.reachable) {\n+\tif (opts.input & COMMIT_GRAPH_INPUT_REACHABLE) {\n \t\tif (write_commit_graph_reachable(odb, flags, &split_opts))\n \t\t\treturn 1;\n \t\treturn 0;\n \t}\n \n \tstring_list_init(&lines, 0);\n-\tif (opts.stdin_packs || opts.stdin_commits) {\n+\tif (opts.input & (COMMIT_GRAPH_INPUT_STDIN_PACKS | COMMIT_GRAPH_INPUT_STDIN_COMMITS)) {\n \t\tstruct strbuf buf = STRBUF_INIT;\n \n \t\twhile (strbuf_getline(&buf, stdin) != EOF)\n \t\t\tstring_list_append(&lines, strbuf_detach(&buf, NULL));\n \n-\t\tif (opts.stdin_packs)\n+\t\tif (opts.input & COMMIT_GRAPH_INPUT_STDIN_PACKS)\n \t\t\tpack_indexes = &lines;\n-\t\tif (opts.stdin_commits) {\n+\t\tif (opts.input & COMMIT_GRAPH_INPUT_STDIN_COMMITS) {\n \t\t\tcommit_hex = &lines;\n \t\t\tflags |= COMMIT_GRAPH_WRITE_CHECK_OIDS;\n \t\t}\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 0bf98b56ec..786b5f73ef 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -227,7 +227,7 @@ graph_git_behavior 'cleared graph, commit 8 vs merge 2' full commits/8 merge/2\n \n test_expect_success 'build graph from latest pack with closure' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n-\tcat new-idx | git commit-graph write --stdin-packs &&\n+\tcat new-idx | git commit-graph write --input=stdin-packs &&\n \ttest_path_is_file $objdir/info/commit-graph &&\n \tgraph_read_expect \"9\" \"extra_edges\"\n '\n@@ -240,7 +240,7 @@ test_expect_success 'build graph from commits with closure' '\n \tgit tag -a -m \"merge\" tag/merge merge/2 &&\n \tgit rev-parse tag/merge >commits-in &&\n \tgit rev-parse merge/1 >>commits-in &&\n-\tcat commits-in | git commit-graph write --stdin-commits &&\n+\tcat commits-in | git commit-graph write --input=stdin-commits &&\n \ttest_path_is_file $objdir/info/commit-graph &&\n \tgraph_read_expect \"6\"\n '\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex a165b48afe..353523eca4 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -35,7 +35,7 @@ test_expect_success 'create commits and write commit-graph' '\n \t\ttest_commit $i &&\n \t\tgit branch commits/$i || return 1\n \tdone &&\n-\tgit commit-graph write --reachable &&\n+\tgit commit-graph write --input=reachable &&\n \ttest_path_is_file $infodir/commit-graph &&\n \tgraph_read_expect 3\n '\n-- \n2.25.0.119.gaa12b7378b\n\n"},{"id":"391140","messageId":"cover.1580862307.git.me@ttaylorr.com","threadId":"52725","inReplyTo":"cover.1580430057.git.me@ttaylorr.com","subject":"[PATCH v2 0/3] builtin/commit-graph.c: new split/merge options","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-05T00:28:29Z","receivedAt":"2020-02-05T00:28:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi,\n\nHere is an updated 'v2' of my series to introduce new splitting and\nmerging options to the 'commit-graph write' builtin.\n\nThese patches are updated to be based on the latest changes in the\nseries upon which this is based (tb/commit-graph-use-odb), and contain\nsome other fixes that I picked up during the last round of review. For\nconvenience, I included a range-diff against 'v1' below.\n\nThanks as always for your review.\n\nTaylor\n\nTaylor Blau (3):\n builtin/commit-graph.c: support '--split[=]'\n builtin/commit-graph.c: introduce '--input='\n builtin/commit-graph.c: support '--input=none'\n\n Documentation/git-commit-graph.txt | 50 ++++++++-----\n builtin/commit-graph.c | 115 +++++++++++++++++++++++------\n commit-graph.c | 28 ++++---\n commit-graph.h | 10 ++-\n t/t5318-commit-graph.sh | 4 +-\n t/t5324-split-commit-graph.sh | 53 ++++++++++++-\n 6 files changed, 204 insertions(+), 56 deletions(-)\n\nRange-diff against v1:\n1: 470bbe3cef ! 1: 3e19d50148 builtin/commit-graph.c: support '--split[=]'\n @@ Commit message\n With '--split', the commit-graph machinery writes new commits in another\n incremental commit-graph which is part of the existing chain, and\n optionally decides to condense the chain into a single commit-graph.\n - This is done to ensure that the aysmptotic behavior of looking up a\n + This is done to ensure that the asymptotic behavior of looking up a\n commit in an incremental chain is dominated by the number of\n incrementals in that chain. It can be controlled by the '--max-commits'\n and '--size-multiple' options.\n @@ Documentation/git-commit-graph.txt: or `--stdin-packs`.)\n +conducted, and the remaining options are ignored. Conversely, if\n +`--split=no-merge` is specified, a merge is never performed, and the\n +remaining options are ignored. A bare `--split` defers to the remaining\n -+options. (Note that merging a chain of commit graphs replaces the\n -+existing chain with a length-1 chain where the first and only\n -+incremental holds the entire graph).\n ++options.\n +\n * If `--size-multiple=` is not specified, let `X` equal 2. If the new\n tip file would have `N` commits and the previous tip has `M` commits and\n @@ builtin/commit-graph.c: static int graph_verify(int argc, const char **argv)\n +\tenum commit_graph_split_flags *flags = opt->value;\n +\n +\topts.split = 1;\n -+\tif (!arg)\n ++\tif (!arg) {\n ++\t\t*flags = COMMIT_GRAPH_SPLIT_MERGE_AUTO;\n +\t\treturn 0;\n ++\t}\n +\n +\tif (!strcmp(arg, \"merge-all\"))\n +\t\t*flags = COMMIT_GRAPH_SPLIT_MERGE_REQUIRED;\n @@ builtin/commit-graph.c: static int graph_write(int argc, const char **argv)\n\n ## commit-graph.c ##\n @@ commit-graph.c: static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n +\n + \tint max_commits = 0;\n + \tint size_mult = 2;\n ++\tenum commit_graph_split_flags flags = COMMIT_GRAPH_SPLIT_MERGE_AUTO;\n +\n + \tif (ctx->split_opts) {\n + \t\tmax_commits = ctx->split_opts->max_commits;\n +\n + \t\tif (ctx->split_opts->size_multiple)\n + \t\t\tsize_mult = ctx->split_opts->size_multiple;\n ++\n ++\t\tflags = ctx->split_opts->flags;\n + \t}\n +\n + \tg = ctx->r->objects->commit_graph;\n \tnum_commits = ctx->commits.nr;\n \tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n\n @@ commit-graph.c: static void split_graph_merge_strategy(struct write_commit_graph\n -\t\t (max_commits && num_commits > max_commits))) {\n -\t\tif (g->odb != ctx->odb)\n -\t\t\tbreak;\n -+\tif (ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n ++\tif (flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n +\t\twhile (g && (g->num_commits <= size_mult * num_commits ||\n +\t\t\t (max_commits && num_commits > max_commits) ||\n -+\t\t\t (ctx->split_opts->flags == COMMIT_GRAPH_SPLIT_MERGE_REQUIRED))) {\n ++\t\t\t (flags == COMMIT_GRAPH_SPLIT_MERGE_REQUIRED))) {\n +\t\t\tif (g->odb != ctx->odb)\n +\t\t\t\tbreak;\n\n @@ commit-graph.c: int write_commit_graph(struct object_directory *odb,\n \t}\n\n -\tif (!ctx->commits.nr)\n -+\tif (!ctx->commits.nr && (!ctx->split || ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_REQUIRED))\n ++\tif (!ctx->commits.nr && (!ctx->split_opts || ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_REQUIRED))\n \t\tgoto cleanup;\n\n \tif (ctx->split) {\n @@ commit-graph.h: enum commit_graph_write_flags {\n };\n\n +enum commit_graph_split_flags {\n -+\tCOMMIT_GRAPH_SPLIT_UNSPECIFIED = 0,\n ++\tCOMMIT_GRAPH_SPLIT_MERGE_AUTO = 0,\n +\tCOMMIT_GRAPH_SPLIT_MERGE_REQUIRED = 1,\n +\tCOMMIT_GRAPH_SPLIT_MERGE_PROHIBITED = 2\n +};\n2: 1b585fe4e7 ! 2: 1589bc1d69 builtin/commit-graph.c: introduce '--input='\n @@ Documentation/git-commit-graph.txt: $ git commit-graph write\n\n ## builtin/commit-graph.c ##\n @@\n +\n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n - \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n +-\tN_(\"git commit-graph write [--object-dir ] [--append] \"\n -\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n -+\t \"[--split[=]] [--input=] \"\n ++\tN_(\"git commit-graph write [--object-dir ] \"\n ++\t \"[--split[=]] \"\n ++\t \"[--input=] \"\n \t \"[--[no-]progress] \"),\n \tNULL\n };\n @@ builtin/commit-graph.c: static const char * const builtin_commit_graph_verify_usage[] = {\n + };\n\n static const char * const builtin_commit_graph_write_usage[] = {\n - \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n +-\tN_(\"git commit-graph write [--object-dir ] [--append] \"\n -\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n -+\t \"[--split[=]] [--input=] \"\n ++\tN_(\"git commit-graph write [--object-dir ] \"\n ++\t \"[--split[=]] \"\n ++\t \"[--input=] \"\n \t \"[--[no-]progress] \"),\n \tNULL\n };\n @@ builtin/commit-graph.c: static int graph_write(int argc, const char **argv)\n \t\t}\n\n ## t/t5318-commit-graph.sh ##\n -@@ t/t5318-commit-graph.sh: test_expect_success 'write graph with no packs' '\n - \ttest_path_is_missing $objdir/info/commit-graph\n - '\n -\n --test_expect_success 'exit with correct error on bad input to --stdin-packs' '\n -+test_expect_success 'exit with correct error on bad input to --input=stdin-packs' '\n - \tcd \"$TRASH_DIRECTORY/full\" &&\n - \techo doesnotexist >in &&\n --\ttest_expect_code 1 git commit-graph write --stdin-packs stderr &&\n -+\ttest_expect_code 1 git commit-graph write --input=stdin-packs stderr &&\n - \ttest_i18ngrep \"error adding pack\" stderr\n - '\n -\n -@@ t/t5318-commit-graph.sh: test_expect_success 'create commits and repack' '\n - \tgit repack\n - '\n -\n --test_expect_success 'exit with correct error on bad input to --stdin-commits' '\n -+test_expect_success 'exit with correct error on bad input to --input=stdin-commits' '\n - \tcd \"$TRASH_DIRECTORY/full\" &&\n --\techo HEAD | test_expect_code 1 git commit-graph write --stdin-commits 2>stderr &&\n -+\techo HEAD | test_expect_code 1 git commit-graph write --input=stdin-commits 2>stderr &&\n - \ttest_i18ngrep \"invalid commit object id\" stderr &&\n - \t# valid tree OID, but not a commit OID\n --\tgit rev-parse HEAD^{tree} | test_expect_code 1 git commit-graph write --stdin-commits 2>stderr &&\n -+\tgit rev-parse HEAD^{tree} | test_expect_code 1 git commit-graph write --input=stdin-commits 2>stderr &&\n - \ttest_i18ngrep \"invalid commit object id\" stderr\n - '\n -\n @@ t/t5318-commit-graph.sh: graph_git_behavior 'cleared graph, commit 8 vs merge 2' full commits/8 merge/2\n\n test_expect_success 'build graph from latest pack with closure' '\n @@ t/t5318-commit-graph.sh: test_expect_success 'build graph from commits with clos\n \ttest_path_is_file $objdir/info/commit-graph &&\n \tgraph_read_expect \"6\"\n '\n -@@ t/t5318-commit-graph.sh: graph_git_behavior 'graph from commits, commit 8 vs merge 2' full commits/8 merg\n -\n - test_expect_success 'build graph from commits with append' '\n - \tcd \"$TRASH_DIRECTORY/full\" &&\n --\tgit rev-parse merge/3 | git commit-graph write --stdin-commits --append &&\n -+\tgit rev-parse merge/3 | git commit-graph write --input=stdin-commits --input=append &&\n - \ttest_path_is_file $objdir/info/commit-graph &&\n - \tgraph_read_expect \"10\" \"extra_edges\"\n - '\n -@@ t/t5318-commit-graph.sh: graph_git_behavior 'append graph, commit 8 vs merge 2' full commits/8 merge/2\n -\n - test_expect_success 'build graph using --reachable' '\n - \tcd \"$TRASH_DIRECTORY/full\" &&\n --\tgit commit-graph write --reachable &&\n -+\tgit commit-graph write --input=reachable &&\n - \ttest_path_is_file $objdir/info/commit-graph &&\n - \tgraph_read_expect \"11\" \"extra_edges\"\n - '\n -@@ t/t5318-commit-graph.sh: test_expect_success 'perform fast-forward merge in full repo' '\n - test_expect_success 'check that gc computes commit-graph' '\n - \tcd \"$TRASH_DIRECTORY/full\" &&\n - \tgit commit --allow-empty -m \"blank\" &&\n --\tgit commit-graph write --reachable &&\n -+\tgit commit-graph write --input=reachable &&\n - \tcp $objdir/info/commit-graph commit-graph-before-gc &&\n - \tgit reset --hard HEAD~1 &&\n - \tgit config gc.writeCommitGraph true &&\n - \tgit gc &&\n - \tcp $objdir/info/commit-graph commit-graph-after-gc &&\n - \t! test_cmp_bin commit-graph-before-gc commit-graph-after-gc &&\n --\tgit commit-graph write --reachable &&\n -+\tgit commit-graph write --input=reachable &&\n - \ttest_cmp_bin commit-graph-after-gc $objdir/info/commit-graph\n - '\n -\n -@@ t/t5318-commit-graph.sh: test_expect_success 'replace-objects invalidates commit-graph' '\n - \tgit clone full replace &&\n - \t(\n - \t\tcd replace &&\n --\t\tgit commit-graph write --reachable &&\n -+\t\tgit commit-graph write --input=reachable &&\n - \t\ttest_path_is_file .git/objects/info/commit-graph &&\n - \t\tgit replace HEAD~1 HEAD~2 &&\n - \t\tgit -c core.commitGraph=false log >expect &&\n - \t\tgit -c core.commitGraph=true log >actual &&\n - \t\ttest_cmp expect actual &&\n --\t\tgit commit-graph write --reachable &&\n -+\t\tgit commit-graph write --input=reachable &&\n - \t\tgit -c core.commitGraph=false --no-replace-objects log >expect &&\n - \t\tgit -c core.commitGraph=true --no-replace-objects log >actual &&\n - \t\ttest_cmp expect actual &&\n - \t\trm -rf .git/objects/info/commit-graph &&\n --\t\tgit commit-graph write --reachable &&\n -+\t\tgit commit-graph write --input=reachable &&\n - \t\ttest_path_is_file .git/objects/info/commit-graph\n - \t)\n - '\n -@@ t/t5318-commit-graph.sh: test_expect_success 'commit grafts invalidate commit-graph' '\n - \tgit clone full graft &&\n - \t(\n - \t\tcd graft &&\n --\t\tgit commit-graph write --reachable &&\n -+\t\tgit commit-graph write --input=reachable &&\n - \t\ttest_path_is_file .git/objects/info/commit-graph &&\n - \t\tH1=$(git rev-parse --verify HEAD~1) &&\n - \t\tH3=$(git rev-parse --verify HEAD~3) &&\n -@@ t/t5318-commit-graph.sh: test_expect_success 'commit grafts invalidate commit-graph' '\n - \t\tgit -c core.commitGraph=false log >expect &&\n - \t\tgit -c core.commitGraph=true log >actual &&\n - \t\ttest_cmp expect actual &&\n --\t\tgit commit-graph write --reachable &&\n -+\t\tgit commit-graph write --input=reachable &&\n - \t\tgit -c core.commitGraph=false --no-replace-objects log >expect &&\n - \t\tgit -c core.commitGraph=true --no-replace-objects log >actual &&\n - \t\ttest_cmp expect actual &&\n - \t\trm -rf .git/objects/info/commit-graph &&\n --\t\tgit commit-graph write --reachable &&\n -+\t\tgit commit-graph write --input=reachable &&\n - \t\ttest_path_is_missing .git/objects/info/commit-graph\n - \t)\n - '\n -@@ t/t5318-commit-graph.sh: test_expect_success 'replace-objects invalidates commit-graph' '\n - \tgit clone --depth 2 \"file://$TRASH_DIRECTORY/full\" shallow &&\n - \t(\n - \t\tcd shallow &&\n --\t\tgit commit-graph write --reachable &&\n -+\t\tgit commit-graph write --input=reachable &&\n - \t\ttest_path_is_missing .git/objects/info/commit-graph &&\n - \t\tgit fetch origin --unshallow &&\n --\t\tgit commit-graph write --reachable &&\n -+\t\tgit commit-graph write --input=reachable &&\n - \t\ttest_path_is_file .git/objects/info/commit-graph\n - \t)\n - '\n -@@ t/t5318-commit-graph.sh: test_expect_success 'replace-objects invalidates commit-graph' '\n -\n - test_expect_success 'git commit-graph verify' '\n - \tcd \"$TRASH_DIRECTORY/full\" &&\n --\tgit rev-parse commits/8 | git commit-graph write --stdin-commits &&\n -+\tgit rev-parse commits/8 | git commit-graph write --input=stdin-commits &&\n - \tgit commit-graph verify >output\n - '\n -\n -@@ t/t5318-commit-graph.sh: test_expect_success 'setup non-the_repository tests' '\n - \ttest_commit -C repo two &&\n - \tgit -C repo config core.commitGraph true &&\n - \tgit -C repo rev-parse two | \\\n --\t\tgit -C repo commit-graph write --stdin-commits\n -+\t\tgit -C repo commit-graph write --input=stdin-commits\n - '\n -\n - test_expect_success 'parse_commit_in_graph works for non-the_repository' '\n -@@ t/t5318-commit-graph.sh: test_expect_success 'corrupt commit-graph write (broken parent)' '\n - \t\tEOF\n - \t\tbroken=\"$(git hash-object -w -t commit --literally broken)\" &&\n - \t\tgit commit-tree -p \"$broken\" -m \"good commit\" \"$empty\" >good &&\n --\t\ttest_must_fail git commit-graph write --stdin-commits \\\n -+\t\ttest_must_fail git commit-graph write --input=stdin-commits \\\n - \t\t\ttest_err &&\n - \t\ttest_i18ngrep \"unable to parse commit\" test_err\n - \t)\n -@@ t/t5318-commit-graph.sh: test_expect_success 'corrupt commit-graph write (missing tree)' '\n - \t\tEOF\n - \t\tbroken=\"$(git hash-object -w -t commit --literally broken)\" &&\n - \t\tgit commit-tree -p \"$broken\" -m \"good\" \"$tree\" >good &&\n --\t\ttest_must_fail git commit-graph write --stdin-commits \\\n -+\t\ttest_must_fail git commit-graph write --input=stdin-commits \\\n - \t\t\ttest_err &&\n - \t\ttest_i18ngrep \"unable to parse commit\" test_err\n - \t)\n\n ## t/t5324-split-commit-graph.sh ##\n @@ t/t5324-split-commit-graph.sh: test_expect_success 'create commits and write commit-graph' '\n @@ t/t5324-split-commit-graph.sh: test_expect_success 'create commits and write com\n \ttest_path_is_file $infodir/commit-graph &&\n \tgraph_read_expect 3\n '\n -@@ t/t5324-split-commit-graph.sh: test_expect_success 'add more commits, and write a new base graph' '\n - \tgit reset --hard commits/4 &&\n - \tgit merge commits/6 &&\n - \tgit branch merge/2 &&\n --\tgit commit-graph write --reachable &&\n -+\tgit commit-graph write --input=reachable &&\n - \tgraph_read_expect 12\n - '\n -\n -@@ t/t5324-split-commit-graph.sh: test_expect_success 'fork and fail to base a chain on a commit-graph file' '\n - \t\trm .git/objects/info/commit-graph &&\n - \t\techo \"$(pwd)/../.git/objects\" >.git/objects/info/alternates &&\n - \t\ttest_commit new-commit &&\n --\t\tgit commit-graph write --reachable --split &&\n -+\t\tgit commit-graph write --input=reachable --split &&\n - \t\ttest_path_is_file $graphdir/commit-graph-chain &&\n - \t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n - \t\tverify_chain_files_exist $graphdir\n -@@ t/t5324-split-commit-graph.sh: test_expect_success 'add three more commits, write a tip graph' '\n - \tgit merge commits/5 &&\n - \tgit merge merge/2 &&\n - \tgit branch merge/3 &&\n --\tgit commit-graph write --reachable --split &&\n -+\tgit commit-graph write --input=reachable --split &&\n - \ttest_path_is_missing $infodir/commit-graph &&\n - \ttest_path_is_file $graphdir/commit-graph-chain &&\n - \tls $graphdir/graph-*.graph >graph-files &&\n -@@ t/t5324-split-commit-graph.sh: graph_git_behavior 'split commit-graph: merge 3 vs 2' merge/3 merge/2\n - test_expect_success 'add one commit, write a tip graph' '\n - \ttest_commit 11 &&\n - \tgit branch commits/11 &&\n --\tgit commit-graph write --reachable --split &&\n -+\tgit commit-graph write --input=reachable --split &&\n - \ttest_path_is_missing $infodir/commit-graph &&\n - \ttest_path_is_file $graphdir/commit-graph-chain &&\n - \tls $graphdir/graph-*.graph >graph-files &&\n -@@ t/t5324-split-commit-graph.sh: graph_git_behavior 'three-layer commit-graph: commit 11 vs 6' commits/11 commits\n - test_expect_success 'add one commit, write a merged graph' '\n - \ttest_commit 12 &&\n - \tgit branch commits/12 &&\n --\tgit commit-graph write --reachable --split &&\n -+\tgit commit-graph write --input=reachable --split &&\n - \ttest_path_is_file $graphdir/commit-graph-chain &&\n - \ttest_line_count = 2 $graphdir/commit-graph-chain &&\n - \tls $graphdir/graph-*.graph >graph-files &&\n -@@ t/t5324-split-commit-graph.sh: test_expect_success 'create fork and chain across alternate' '\n - \t\techo \"$(pwd)/../.git/objects\" >.git/objects/info/alternates &&\n - \t\ttest_commit 13 &&\n - \t\tgit branch commits/13 &&\n --\t\tgit commit-graph write --reachable --split &&\n -+\t\tgit commit-graph write --input=reachable --split &&\n - \t\ttest_path_is_file $graphdir/commit-graph-chain &&\n - \t\ttest_line_count = 3 $graphdir/commit-graph-chain &&\n - \t\tls $graphdir/graph-*.graph >graph-files &&\n -@@ t/t5324-split-commit-graph.sh: test_expect_success 'create fork and chain across alternate' '\n - \t\tgit -c core.commitGraph=false rev-list HEAD >actual &&\n - \t\ttest_cmp expect actual &&\n - \t\ttest_commit 14 &&\n --\t\tgit commit-graph write --reachable --split --object-dir=.git/objects/ &&\n -+\t\tgit commit-graph write --input=reachable --split --object-dir=.git/objects/ &&\n - \t\ttest_line_count = 3 $graphdir/commit-graph-chain &&\n - \t\tls $graphdir/graph-*.graph >graph-files &&\n - \t\ttest_line_count = 1 graph-files\n -@@ t/t5324-split-commit-graph.sh: test_expect_success 'test merge stragety constants' '\n - \t\tgit config core.commitGraph true &&\n - \t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n - \t\ttest_commit 14 &&\n --\t\tgit commit-graph write --reachable --split --size-multiple=2 &&\n -+\t\tgit commit-graph write --input=reachable --split --size-multiple=2 &&\n - \t\ttest_line_count = 3 $graphdir/commit-graph-chain\n -\n - \t) &&\n -@@ t/t5324-split-commit-graph.sh: test_expect_success 'test merge stragety constants' '\n - \t\tgit config core.commitGraph true &&\n - \t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n - \t\ttest_commit 14 &&\n --\t\tgit commit-graph write --reachable --split --size-multiple=10 &&\n -+\t\tgit commit-graph write --input=reachable --split --size-multiple=10 &&\n - \t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n - \t\tls $graphdir/graph-*.graph >graph-files &&\n - \t\ttest_line_count = 1 graph-files\n -@@ t/t5324-split-commit-graph.sh: test_expect_success 'test merge stragety constants' '\n - \t\tgit config core.commitGraph true &&\n - \t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n - \t\ttest_commit 15 &&\n --\t\tgit commit-graph write --reachable --split --size-multiple=10 --expire-time=1980-01-01 &&\n -+\t\tgit commit-graph write --input=reachable --split --size-multiple=10 --expire-time=1980-01-01 &&\n - \t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n - \t\tls $graphdir/graph-*.graph >graph-files &&\n - \t\ttest_line_count = 3 graph-files\n -@@ t/t5324-split-commit-graph.sh: test_expect_success 'test merge stragety constants' '\n - \t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n - \t\ttest_commit 16 &&\n - \t\ttest_commit 17 &&\n --\t\tgit commit-graph write --reachable --split --max-commits=1 &&\n -+\t\tgit commit-graph write --input=reachable --split --max-commits=1 &&\n - \t\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n - \t\tls $graphdir/graph-*.graph >graph-files &&\n - \t\ttest_line_count = 1 graph-files\n -@@ t/t5324-split-commit-graph.sh: test_expect_success 'remove commit-graph-chain file after flattening' '\n - \t(\n - \t\tcd flatten &&\n - \t\ttest_line_count = 2 $graphdir/commit-graph-chain &&\n --\t\tgit commit-graph write --reachable &&\n -+\t\tgit commit-graph write --input=reachable &&\n - \t\ttest_path_is_missing $graphdir/commit-graph-chain &&\n - \t\tls $graphdir >graph-files &&\n - \t\ttest_line_count = 0 graph-files\n -@@ t/t5324-split-commit-graph.sh: test_expect_success 'verify across alternates' '\n - \t\techo \"$altdir\" >.git/objects/info/alternates &&\n - \t\tgit commit-graph verify --object-dir=\"$altdir/\" &&\n - \t\ttest_commit extra &&\n --\t\tgit commit-graph write --reachable --split &&\n -+\t\tgit commit-graph write --input=reachable --split &&\n - \t\ttip_file=$graphdir/graph-$(tail -n 1 $graphdir/commit-graph-chain).graph &&\n - \t\tcorrupt_file \"$tip_file\" 100 \"\\01\" &&\n - \t\ttest_must_fail git commit-graph verify --shallow 2>test_err &&\n -@@ t/t5324-split-commit-graph.sh: test_expect_success 'add octopus merge' '\n - \tgit reset --hard commits/10 &&\n - \tgit merge commits/3 commits/4 &&\n - \tgit branch merge/octopus &&\n --\tgit commit-graph write --reachable --split &&\n -+\tgit commit-graph write --input=reachable --split &&\n - \tgit commit-graph verify --progress 2>err &&\n - \ttest_line_count = 3 err &&\n - \ttest_i18ngrep ! warning err &&\n -@@ t/t5324-split-commit-graph.sh: test_expect_success 'add octopus merge' '\n - graph_git_behavior 'graph exists' merge/octopus commits/12\n -\n - test_expect_success 'split across alternate where alternate is not split' '\n --\tgit commit-graph write --reachable &&\n -+\tgit commit-graph write --input=reachable &&\n - \ttest_path_is_file .git/objects/info/commit-graph &&\n - \tcp .git/objects/info/commit-graph . &&\n - \tgit clone --no-hardlinks . alt-split &&\n -@@ t/t5324-split-commit-graph.sh: test_expect_success 'split across alternate where alternate is not split' '\n - \t\trm -f .git/objects/info/commit-graph &&\n - \t\techo \"$(pwd)\"/../.git/objects >.git/objects/info/alternates &&\n - \t\ttest_commit 18 &&\n --\t\tgit commit-graph write --reachable --split &&\n -+\t\tgit commit-graph write --input=reachable --split &&\n - \t\ttest_line_count = 1 $graphdir/commit-graph-chain\n - \t) &&\n - \ttest_cmp commit-graph .git/objects/info/commit-graph\n -@@ t/t5324-split-commit-graph.sh: test_expect_success '--split=merge-all always merges incrementals' '\n - \tgit rev-list -3 HEAD~4 >a &&\n - \tgit rev-list -2 HEAD~2 >b &&\n - \tgit rev-list -2 HEAD >c &&\n --\tgit commit-graph write --split=no-merge --stdin-commits a &&\n - \tgit rev-list HEAD >b &&\n --\tgit commit-graph write --split --stdin-commits ',\n + and alluded to the fact that '--split=merge-all' would be useful for\n + callers who wish to always trigger a merge of an incremental chain.\n\n There is a problem with the above approach, which is that there is no\n way to specify to the commit-graph builtin that a caller only wants to\n @@ Documentation/git-commit-graph.txt: COMMANDS\n 'write'::\n\n -Write a commit-graph file based on the commits found in packfiles.\n -+Write a commit-graph file based on the commits specified:\n -+* With the `--input=stdin-packs` option, generate the new commit graph\n -+by walking objects only in the specified pack-indexes. (Cannot be\n -+combined with `--input=stdin-commits` or `--input=reachable`.)\n ++Write a commit-graph file based on the specified sources of input:\n +\n --With the `--input=stdin-packs` option, generate the new commit graph by\n --walking objects only in the specified pack-indexes. (Cannot be combined\n --with `--input=stdin-commits` or `--input=reachable`.)\n --+\n --With the `--input=stdin-commits` option, generate the new commit graph\n -+* With the `--input=stdin-commits` option, generate the new commit graph\n - by walking commits starting at the commits specified in stdin as a list\n - of OIDs in hex, one OID per line. (Cannot be combined with\n - `--input=stdin-packs` or `--input=reachable`.)\n - +\n --With the `--input=reachable` option, generate the new commit graph by\n -+* With the `--input=reachable` option, generate the new commit graph by\n - walking commits starting at all refs. (Cannot be combined with\n - `--input=stdin-commits` or `--input=stdin-packs`.)\n - +\n --With the `--input=append` option, include all commits that are present\n -+* With the `--input=append` option, include all commits that are present\n + With the `--input=stdin-packs` option, generate the new commit graph by\n + walking objects only in the specified pack-indexes. (Cannot be combined\n +@@ Documentation/git-commit-graph.txt: walking commits starting at all refs. (Cannot be combined with\n + With the `--input=append` option, include all commits that are present\n in the existing commit-graph file.\n +\n -+* With the `--input=none` option, behave as if `input=append` were\n ++With the `--input=none` option, behave as if `--input=append` were\n +given, but do not walk other packs to find additional commits.\n +\n -+If none of the above options are given, then commits found in\n -+packfiles are specified.\n ++If none of the above options are given, then generate the new\n ++commit-graph by walking over all pack-indexes.\n ++\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n `/info/commit-graphs`. Commit-graph layers are merged based on the\n\n ## builtin/commit-graph.c ##\n -@@\n - static char const * const builtin_commit_graph_usage[] = {\n +@@ builtin/commit-graph.c: static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n - \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n --\t \"[--split[=]] [--input=] \"\n -+\t \"[--split[=]] \"\n -+\t \"[--input=] \"\n + \tN_(\"git commit-graph write [--object-dir ] \"\n + \t \"[--split[=]] \"\n +-\t \"[--input=] \"\n ++\t \"[--input=] \"\n \t \"[--[no-]progress] \"),\n \tNULL\n };\n @@ builtin/commit-graph.c: static const char * const builtin_commit_graph_verify_usage[] = {\n -\n static const char * const builtin_commit_graph_write_usage[] = {\n - \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n --\t \"[--split[=]] [--input=] \"\n -+\t \"[--split[=]] \"\n -+\t \"[--input=] \"\n + \tN_(\"git commit-graph write [--object-dir ] \"\n + \t \"[--split[=]] \"\n +-\t \"[--input=] \"\n ++\t \"[--input=] \"\n \t \"[--[no-]progress] \"),\n \tNULL\n };\n--\n2.25.0.119.gaa12b7378b\n"},{"id":"391141","messageId":"4c6425f0da9a6e5ae86530a12f18959ada07404b.1580862307.git.me@ttaylorr.com","threadId":"52725","inReplyTo":"cover.1580862307.git.me@ttaylorr.com","subject":"[PATCH v2 3/3] builtin/commit-graph.c: support '--input=none'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-05T00:28:36Z","receivedAt":"2020-02-05T00:28:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In the previous commit, we introduced '--split=',\nand alluded to the fact that '--split=merge-all' would be useful for\ncallers who wish to always trigger a merge of an incremental chain.\n\nThere is a problem with the above approach, which is that there is no\nway to specify to the commit-graph builtin that a caller only wants to\ninclude commits already in the graph. One can specify '--input=append'\nto include all commits in the existing graphs, but the absence of\n'--input=stdin-{commits,packs}' causes the builtin to call\n'fill_oids_from_all_packs()'.\n\nPassing '--input=reachable' (as in 'git commit-graph write\n--split=merge-all --input=reachable --input=append') works around this\nissue by making '--input=reachable' effectively a no-op, but this can be\nprohibitively expensive in large repositories, making it an undesirable\nchoice for some users.\n\nTeach '--input=none' as an option to behave as if '--input=append' were\ngiven, but to consider no other sources in addition.\n\nThis, in conjunction with the option introduced in the previous patch\noffers the convenient way to force the commit-graph machinery to\ncondense a chain of incrementals without requiring any new commits:\n\n $ git commit-graph write --split=merge-all --input=none\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-commit-graph.txt | 8 +++++++-\n builtin/commit-graph.c | 11 ++++++++---\n commit-graph.c | 6 ++++--\n commit-graph.h | 3 ++-\n t/t5324-split-commit-graph.sh | 26 ++++++++++++++++++++++++++\n 5 files changed, 47 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 2ae9de679a..633cfbe023 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -39,7 +39,7 @@ COMMANDS\n --------\n 'write'::\n \n-Write a commit-graph file based on the commits found in packfiles.\n+Write a commit-graph file based on the specified sources of input:\n +\n With the `--input=stdin-packs` option, generate the new commit graph by\n walking objects only in the specified pack-indexes. (Cannot be combined\n@@ -57,6 +57,12 @@ walking commits starting at all refs. (Cannot be combined with\n With the `--input=append` option, include all commits that are present\n in the existing commit-graph file.\n +\n+With the `--input=none` option, behave as if `--input=append` were\n+given, but do not walk other packs to find additional commits.\n+\n+If none of the above options are given, then generate the new\n+commit-graph by walking over all pack-indexes.\n++\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n `/info/commit-graphs`. Commit-graph layers are merged based on the\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 0ff25896d0..a71af88815 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -11,7 +11,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n \tN_(\"git commit-graph write [--object-dir ] \"\n \t \"[--split[=]] \"\n-\t \"[--input=] \"\n+\t \"[--input=] \"\n \t \"[--[no-]progress] \"),\n \tNULL\n };\n@@ -24,7 +24,7 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n static const char * const builtin_commit_graph_write_usage[] = {\n \tN_(\"git commit-graph write [--object-dir ] \"\n \t \"[--split[=]] \"\n-\t \"[--input=] \"\n+\t \"[--input=] \"\n \t \"[--[no-]progress] \"),\n \tNULL\n };\n@@ -33,7 +33,8 @@ enum commit_graph_input {\n \tCOMMIT_GRAPH_INPUT_REACHABLE = (1 << 1),\n \tCOMMIT_GRAPH_INPUT_STDIN_PACKS = (1 << 2),\n \tCOMMIT_GRAPH_INPUT_STDIN_COMMITS = (1 << 3),\n-\tCOMMIT_GRAPH_INPUT_APPEND = (1 << 4)\n+\tCOMMIT_GRAPH_INPUT_APPEND = (1 << 4),\n+\tCOMMIT_GRAPH_INPUT_NONE = (1 << 5)\n };\n \n static struct opts_commit_graph {\n@@ -80,6 +81,8 @@ static int option_parse_input(const struct option *opt, const char *arg,\n \t\t*to |= COMMIT_GRAPH_INPUT_STDIN_COMMITS;\n \telse if (!strcmp(arg, \"append\"))\n \t\t*to |= COMMIT_GRAPH_INPUT_APPEND;\n+\telse if (!strcmp(arg, \"none\"))\n+\t\t*to |= (COMMIT_GRAPH_INPUT_APPEND | COMMIT_GRAPH_INPUT_NONE);\n \telse\n \t\tdie(_(\"unrecognized --input source, %s\"), arg);\n \treturn 0;\n@@ -225,6 +228,8 @@ static int graph_write(int argc, const char **argv)\n \t\topts.obj_dir = get_object_directory();\n \tif (opts.input & COMMIT_GRAPH_INPUT_APPEND)\n \t\tflags |= COMMIT_GRAPH_WRITE_APPEND;\n+\tif (opts.input & COMMIT_GRAPH_INPUT_NONE)\n+\t\tflags |= COMMIT_GRAPH_WRITE_NO_INPUT;\n \tif (opts.split)\n \t\tflags |= COMMIT_GRAPH_WRITE_SPLIT;\n \tif (opts.progress)\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 3a5cb23cd7..417b7eac9c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -788,7 +788,8 @@ struct write_commit_graph_context {\n \tunsigned append:1,\n \t\t report_progress:1,\n \t\t split:1,\n-\t\t check_oids:1;\n+\t\t check_oids:1,\n+\t\t no_input:1;\n \n \tconst struct split_commit_graph_opts *split_opts;\n };\n@@ -1785,6 +1786,7 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->split = flags & COMMIT_GRAPH_WRITE_SPLIT ? 1 : 0;\n \tctx->check_oids = flags & COMMIT_GRAPH_WRITE_CHECK_OIDS ? 1 : 0;\n \tctx->split_opts = split_opts;\n+\tctx->no_input = flags & COMMIT_GRAPH_WRITE_NO_INPUT ? 1 : 0;\n \n \tif (ctx->split) {\n \t\tstruct commit_graph *g;\n@@ -1843,7 +1845,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\t\tgoto cleanup;\n \t}\n \n-\tif (!pack_indexes && !commit_hex)\n+\tif (!ctx->no_input && !pack_indexes && !commit_hex)\n \t\tfill_oids_from_all_packs(ctx);\n \n \tclose_reachable(ctx);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 65a7d2edae..df7f3f5961 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -79,7 +79,8 @@ enum commit_graph_write_flags {\n \tCOMMIT_GRAPH_WRITE_PROGRESS = (1 << 1),\n \tCOMMIT_GRAPH_WRITE_SPLIT = (1 << 2),\n \t/* Make sure that each OID in the input is a valid commit OID. */\n-\tCOMMIT_GRAPH_WRITE_CHECK_OIDS = (1 << 3)\n+\tCOMMIT_GRAPH_WRITE_CHECK_OIDS = (1 << 3),\n+\tCOMMIT_GRAPH_WRITE_NO_INPUT = (1 << 4)\n };\n \n enum commit_graph_split_flags {\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 353523eca4..e3f317a1f4 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -369,4 +369,30 @@ test_expect_success '--split=no-merge always writes an incremental' '\n \ttest_line_count = 2 $graphdir/commit-graph-chain\n '\n \n+test_expect_success '--split=no-merge, --input=none writes nothing' '\n+\ttest_when_finished rm -rf a graphs.before graphs.after &&\n+\trm -rf $graphdir &&\n+\tgit reset --hard commits/2 &&\n+\tgit rev-list -1 HEAD~1 >a &&\n+\tgit commit-graph write --split=no-merge --input=stdin-commits graphs.before &&\n+\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n+\tgit commit-graph write --split --input=none &&\n+\tls $graphdir/graph-*.graph >graphs.after &&\n+\ttest_cmp graphs.before graphs.after\n+'\n+\n+test_expect_success '--split=merge-all, --input=none merges the chain' '\n+\ttest_when_finished rm -rf a b &&\n+\trm -rf $graphdir &&\n+\tgit reset --hard commits/2 &&\n+\tgit rev-list -1 HEAD~1 >a &&\n+\tgit rev-list -1 HEAD >b &&\n+\tgit commit-graph write --split=no-merge --input=stdin-commits As the topoic this depends on has been updated, I tried to rebase\n> this on top, but I am seeing segfaults in tests. We'd probably need\n> a fresh round of this one to replace it.\n\nSure, although in fairness this test failure is unique to this series.\nBut, I sent a rebased version of these three patches to be based on the\nlatest from the 'tb/commit-graph-use-odb.v2' topic.\n\n> Thanks.\n\nThanks for taking a look at it.\n\nThanks,\nTaylor\n"},{"id":"391189","messageId":"xmqqimkk6bnk.fsf@gitster-ct.c.googlers.com","threadId":"52725","inReplyTo":"cover.1580862307.git.me@ttaylorr.com","subject":"Re: [PATCH v2 0/3] builtin/commit-graph.c: new split/merge options","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-02-05T20:07:27Z","receivedAt":"2020-02-05T20:07:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Here is an updated 'v2' of my series to introduce new splitting and\n> merging options to the 'commit-graph write' builtin.\n>\n> These patches are updated to be based on the latest changes in the\n> series upon which this is based (tb/commit-graph-use-odb), and contain\n> some other fixes that I picked up during the last round of review. For\n> convenience, I included a range-diff against 'v1' below.\n>\n> Thanks as always for your review.\n\nThanks, replaced.\n"},{"id":"391262","messageId":"CAN0heSpg6i7m9tT6LM-8C+cOMGOtDYMdpsuGn-weUo23jgJOTQ@mail.gmail.com","threadId":"52725","inReplyTo":"20200204040628.GE5790@syl.local","subject":"Re: [PATCH 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-02-06T19:15:03Z","receivedAt":"2020-02-06T19:15:16Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Tue, 4 Feb 2020 at 05:06, Taylor Blau wrote:\n>\n> > Or can you convince me otherwise? From which angle should I look at\n> > this?\n>\n> Heh. This is all very reminiscent of an off-list discussion that I had\n> with Peff and Stolee before sending this upstream. Originally, I had\n> implemented this as:\n>\n> $ git commit-graph write --split --[no-]merge\n>\n> but we decided that this '--merge' and '--no-merge' requiring '--split'\n> seemed to indicate that this was better off as an argument to '--split'.\n> Of course, there's no getting around that it is... odd to say\n> '--split=no-merge' for exactly the reason you suggest.\n>\n> Here's another way of looking at it: the presence of '--split' means\n> \"work with split graph files\" and the '[no-]merge' argument means:\n> \"always/never condense multiple layers\".\n>\n> For me, this not only makes the new option language jive, but makes it\n> clearer to me than the combination of '--split', '--split --no-merge'\n> and '--split --merge', where the third one is truly bizarre. At least\n> condensing the second '--' and making 'merge' an argument to 'split'\n> makes it clear that the two work together somehow.\n\nYes, \"--split --merge\" sounds no better. :-)\n\n> If you have a different suggestion, I'd certainly love to hear about it\n> and discuss. But, at least as far as our internal discussions have gone,\n> this is by far the best option that we have been able to come up with.\n\nI can't come up with anything better, so please feel free to carry on\n(as you already have).\n\n\n> > > - OPT_BOOL(0, \"split\", &opts.split,\n> > > - N_(\"allow writing an incremental commit-graph file\")),\n> > > + OPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n> > > + N_(\"allow writing an incremental commit-graph file\"),\n> >\n> > This still sounds very boolean. Cramming in the \"strategy\" might be hard\n> > -- is this an argument in favor of having two separate options? ;-)\n>\n> Heh. Exactly how we started these patches when I originally wrote\n> them...\n\nYou left this as-is in v2. I don't have any immediate improvements to\noffer. I could see shortening the original to \"use the 'split' file\nformat\", in which case maybe one could allude to a strategy here. (I\ndon't think \"allow\" is really needed, right? Maybe it tries to cover for\nthe situation where there's no commit graph yet, so you might say we\nwouldn't write an \"incremental\" one, but the format would still be the\nsame, AFAIU. Anyway, that's outside the scope of this patch.)\n\nMartin\n"},{"id":"391265","messageId":"CAN0heSrXZd7ktBTHaYFWjhW=NcGx5gL52-unSDaC4ZoNf96HFA@mail.gmail.com","threadId":"52725","inReplyTo":"3e19d50148c8d53b30f8f0036a2d3af9f4bb3499.1580862307.git.me@ttaylorr.com","subject":"Re: [PATCH v2 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-02-06T19:41:28Z","receivedAt":"2020-02-06T19:41:41Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 5 Feb 2020 at 01:28, Taylor Blau wrote:\n> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> index 28d1fee505..b7fe65ef21 100644\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -57,11 +57,17 @@ or `--stdin-packs`.)\n> With the `--append` option, include all commits that are present in the\n> existing commit-graph file.\n> +\n> -With the `--split` option, write the commit-graph as a chain of multiple\n> -commit-graph files stored in `/info/commit-graphs`. The new commits\n> -not already in the commit-graph are added in a new \"tip\" file. This file\n> -is merged with the existing file if the following merge conditions are\n> -met:\n> +With the `--split[=]` option, write the commit-graph as a\n> +chain of multiple commit-graph files stored in\n> +`/info/commit-graphs`. Commit-graph layers are merged based on the\n> +strategy and other splitting options. The new commits not already in the\n> +commit-graph are added in a new \"tip\" file. This file is merged with the\n> +existing file if the following merge conditions are met:\n\nPlease add a lone \"+\" here.\n\n> +* If `--split=merge-always` is specified, then a merge is always\n> +conducted, and the remaining options are ignored. Conversely, if\n> +`--split=no-merge` is specified, a merge is never performed, and the\n> +remaining options are ignored. A bare `--split` defers to the remaining\n> +options.\n> +\n\nSimilar to this existing one here. There's some minor misrendering here\notherwise.\n\n> * If `--size-multiple=` is not specified, let `X` equal 2. If the new\n> tip file would have `N` commits and the previous tip has `M` commits and\n\n> - OPT_BOOL(0, \"split\", &opts.split,\n> - N_(\"allow writing an incremental commit-graph file\")),\n> + OPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n> + N_(\"allow writing an incremental commit-graph file\"),\n> + PARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n> + write_option_parse_split),\n\n\nI keep getting back to this -- sorry! So this actually forbids\n\"--no-split\", which used to work before. Unfortunate?\n\nI have to ask, what is the long-term plan for the two formats (split and\nnon-split)? As I understand it, and I might well be wrong, the non-split\nformat came first and the split format was a user-experience\nimprovement. Should we expect that `--split` becomes the default? In\nwhich case `--no-split` would be needed. Or might the non-split format\ngo away entirely, leaving `--split` a no-op and `--split=` a\npretty funky way of choosing a strategy for the one-and-only file\nformat?\n\nTo try to be concrete, here's a suggestion: `--format=split` and\n`--split-strategy=`.\n\nMartin\n"},{"id":"391269","messageId":"CAN0heSrRJYc8DoWQ+NHR5LJ1z7d1GGBni2waP0_Lv7gMYjPPEg@mail.gmail.com","threadId":"52725","inReplyTo":"4c6425f0da9a6e5ae86530a12f18959ada07404b.1580862307.git.me@ttaylorr.com","subject":"Re: [PATCH v2 3/3] builtin/commit-graph.c: support '--input=none'","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-02-06T19:50:57Z","receivedAt":"2020-02-06T19:51:10Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 5 Feb 2020 at 01:28, Taylor Blau wrote:\n> @@ -57,6 +57,12 @@ walking commits starting at all refs. (Cannot be combined with\n> With the `--input=append` option, include all commits that are present\n> in the existing commit-graph file.\n> +\n> +With the `--input=none` option, behave as if `--input=append` were\n> +given, but do not walk other packs to find additional commits.\n> +\n\nSimilar to my comment in patch 1/3. Please add a \"+\" instead of an empty\nline. This one actually trips up the rendering quite a bit of a lot of\nwhat follows.\n\n> +If none of the above options are given, then generate the new\n> +commit-graph by walking over all pack-indexes.\n> ++\n\nThis one's good.\n\nMartin\n"},{"id":"391334","messageId":"3acac399-9476-e4ad-556e-e0138380eeb0@gmail.com","threadId":"52725","inReplyTo":"CAN0heSrXZd7ktBTHaYFWjhW=NcGx5gL52-unSDaC4ZoNf96HFA@mail.gmail.com","subject":"Re: [PATCH v2 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-02-07T15:48:39Z","receivedAt":"2020-02-07T15:48:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/6/2020 2:41 PM, Martin Ågren wrote:\n> On Wed, 5 Feb 2020 at 01:28, Taylor Blau wrote:\n>> * If `--size-multiple=` is not specified, let `X` equal 2. If the new\n>> tip file would have `N` commits and the previous tip has `M` commits and\n> \n>> - OPT_BOOL(0, \"split\", &opts.split,\n>> - N_(\"allow writing an incremental commit-graph file\")),\n>> + OPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n>> + N_(\"allow writing an incremental commit-graph file\"),\n>> + PARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n>> + write_option_parse_split),\n> \n> \n> I keep getting back to this -- sorry! So this actually forbids\n> \"--no-split\", which used to work before. Unfortunate?\n\nThat certainly is unfortunate. Hopefully no one is taking a dependence on\nthis, which only means something if they had a `--split` previously in\nthe command-line arguments.\n\n> I have to ask, what is the long-term plan for the two formats (split and\n> non-split)? As I understand it, and I might well be wrong, the non-split\n> format came first and the split format was a user-experience\n> improvement. Should we expect that `--split` becomes the default?\n\nIn some ways, the split is now the default because that is how it is\nwritten during 'git fetch' using fetch.writeCommitGraph. However, I\ndon't think that it will ever become the default for the commit-graph\nbuiltin.\n\n> In\n> which case `--no-split` would be needed. Or might the non-split format\n> go away entirely, leaving `--split` a no-op and `--split=` a\n> pretty funky way of choosing a strategy for the one-and-only file\n> format?\n\nIn some ways, the --split=merge-all is similar, except it writes a one-line\ncommit-graph-chain file and puts a .graph file in\n.git/objects/info/commit-graphs instead of writing to .git/objects/commit-graph.\n\n> To try to be concrete, here's a suggestion: `--format=split` and\n> `--split-strategy=`.\n\nWhy --format=split instead of leaving it as --[no-]split? Is there a reason to\nintroduce this string-based option when there are only two options right now?\n\nPerhaps using --split-strategy= is the most backwards-compatible\noption, especially because we won't need --split=\"\" to substitute for\n\"auto-merge\". However, I wonder if this is a case where we should make the\nhard choice to sacrifice a narrow backwards-compatibility in favor of a\nsimplified set of options?\n\nThanks,\n-Stolee\n"},{"id":"391424","messageId":"20200209232702.GD4530@syl.local","threadId":"52725","inReplyTo":"CAN0heSpg6i7m9tT6LM-8C+cOMGOtDYMdpsuGn-weUo23jgJOTQ@mail.gmail.com","subject":"Re: [PATCH 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-09T23:27:02Z","receivedAt":"2020-02-09T23:27:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Feb 06, 2020 at 08:15:03PM +0100, Martin Ågren wrote:\n> On Tue, 4 Feb 2020 at 05:06, Taylor Blau wrote:\n> >\n> > > Or can you convince me otherwise? From which angle should I look at\n> > > this?\n> >\n> > Heh. This is all very reminiscent of an off-list discussion that I had\n> > with Peff and Stolee before sending this upstream. Originally, I had\n> > implemented this as:\n> >\n> > $ git commit-graph write --split --[no-]merge\n> >\n> > but we decided that this '--merge' and '--no-merge' requiring '--split'\n> > seemed to indicate that this was better off as an argument to '--split'.\n> > Of course, there's no getting around that it is... odd to say\n> > '--split=no-merge' for exactly the reason you suggest.\n> >\n> > Here's another way of looking at it: the presence of '--split' means\n> > \"work with split graph files\" and the '[no-]merge' argument means:\n> > \"always/never condense multiple layers\".\n> >\n> > For me, this not only makes the new option language jive, but makes it\n> > clearer to me than the combination of '--split', '--split --no-merge'\n> > and '--split --merge', where the third one is truly bizarre. At least\n> > condensing the second '--' and making 'merge' an argument to 'split'\n> > makes it clear that the two work together somehow.\n>\n> Yes, \"--split --merge\" sounds no better. :-)\n\n:-).\n\n> > If you have a different suggestion, I'd certainly love to hear about it\n> > and discuss. But, at least as far as our internal discussions have gone,\n> > this is by far the best option that we have been able to come up with.\n>\n> I can't come up with anything better, so please feel free to carry on\n> (as you already have).\n\nSounds good. It looks like you might have had some further thoughts a\nlittle bit lower down in the thread, so I'll respond to those shortly\njust to make sure that I didn't miss anything before readying a 'v3' for\nsubmission.\n\n>\n> > > > - OPT_BOOL(0, \"split\", &opts.split,\n> > > > - N_(\"allow writing an incremental commit-graph file\")),\n> > > > + OPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n> > > > + N_(\"allow writing an incremental commit-graph file\"),\n> > >\n> > > This still sounds very boolean. Cramming in the \"strategy\" might be hard\n> > > -- is this an argument in favor of having two separate options? ;-)\n> >\n> > Heh. Exactly how we started these patches when I originally wrote\n> > them...\n>\n> You left this as-is in v2. I don't have any immediate improvements to\n> offer. I could see shortening the original to \"use the 'split' file\n> format\", in which case maybe one could allude to a strategy here. (I\n> don't think \"allow\" is really needed, right? Maybe it tries to cover for\n> the situation where there's no commit graph yet, so you might say we\n> wouldn't write an \"incremental\" one, but the format would still be the\n> same, AFAIU. Anyway, that's outside the scope of this patch.)\n\nYeah, I agree that the use of \"allow\" is a little funny for these\nreasons. That said, I don't think that it's in dire need of changing,\nand so since we agree that it's outside the scope of this series, I'm\nhappy to ignore it for now.\n\n> Martin\n\nThanks,\nTaylor\n"},{"id":"391425","messageId":"20200209233042.GE4530@syl.local","threadId":"52725","inReplyTo":"CAN0heSrXZd7ktBTHaYFWjhW=NcGx5gL52-unSDaC4ZoNf96HFA@mail.gmail.com","subject":"Re: [PATCH v2 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-09T23:30:42Z","receivedAt":"2020-02-09T23:30:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Feb 06, 2020 at 08:41:28PM +0100, Martin Ågren wrote:\n> On Wed, 5 Feb 2020 at 01:28, Taylor Blau wrote:\n> > diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> > index 28d1fee505..b7fe65ef21 100644\n> > --- a/Documentation/git-commit-graph.txt\n> > +++ b/Documentation/git-commit-graph.txt\n> > @@ -57,11 +57,17 @@ or `--stdin-packs`.)\n> > With the `--append` option, include all commits that are present in the\n> > existing commit-graph file.\n> > +\n> > -With the `--split` option, write the commit-graph as a chain of multiple\n> > -commit-graph files stored in `/info/commit-graphs`. The new commits\n> > -not already in the commit-graph are added in a new \"tip\" file. This file\n> > -is merged with the existing file if the following merge conditions are\n> > -met:\n> > +With the `--split[=]` option, write the commit-graph as a\n> > +chain of multiple commit-graph files stored in\n> > +`/info/commit-graphs`. Commit-graph layers are merged based on the\n> > +strategy and other splitting options. The new commits not already in the\n> > +commit-graph are added in a new \"tip\" file. This file is merged with the\n> > +existing file if the following merge conditions are met:\n>\n> Please add a lone \"+\" here.\n\nSure, thanks for noticing.\n\n> > +* If `--split=merge-always` is specified, then a merge is always\n> > +conducted, and the remaining options are ignored. Conversely, if\n> > +`--split=no-merge` is specified, a merge is never performed, and the\n> > +remaining options are ignored. A bare `--split` defers to the remaining\n> > +options.\n> > +\n>\n> Similar to this existing one here. There's some minor misrendering here\n> otherwise.\n>\n> > * If `--size-multiple=` is not specified, let `X` equal 2. If the new\n> > tip file would have `N` commits and the previous tip has `M` commits and\n>\n> > - OPT_BOOL(0, \"split\", &opts.split,\n> > - N_(\"allow writing an incremental commit-graph file\")),\n> > + OPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n> > + N_(\"allow writing an incremental commit-graph file\"),\n> > + PARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n> > + write_option_parse_split),\n>\n>\n> I keep getting back to this -- sorry! So this actually forbids\n> \"--no-split\", which used to work before. Unfortunate?\n\nAh, I see. Yes, this definitely *does* forbid that. My thinking when I\ndecided to give this 'PARSE_OPT_NONEG' was that '--no-split' is\nconfusing for users: does it mean \"don't split\" or \"unset any split\noptions\"?\n\nThis probably would be ameliorated by your suggestion below, maybe of\n'--split-strategy', specifically (I could probably go either way on\n'--format=split', but it really depends on what Stolee has planned\nlong-term). Then, '--[no-]split' remains clear, as does\n'--no-split-strategy' (although I suppose that you could make the\nargument that '--no-split-strategy' sounds a little bit like letting the\nmachinery use its defaults, which may or may not be true depending on\nhow it's implemented.)\n\nBut, I'm not sure that it's all worth it to add another option here.\nThis sub-builtin has a plethora of options already, and I'm skeptical\nthat there are a lot of real-world uses of '--no-split' in the wild that\nwe'd be breaking.\n\n> I have to ask, what is the long-term plan for the two formats (split and\n> non-split)? As I understand it, and I might well be wrong, the non-split\n> format came first and the split format was a user-experience\n> improvement. Should we expect that `--split` becomes the default? In\n> which case `--no-split` would be needed. Or might the non-split format\n> go away entirely, leaving `--split` a no-op and `--split=` a\n> pretty funky way of choosing a strategy for the one-and-only file\n> format?\n>\n> To try to be concrete, here's a suggestion: `--format=split` and\n> `--split-strategy=`.\n>\n> Martin\n\nThanks,\nTaylor\n"},{"id":"391426","messageId":"20200209233227.GF4530@syl.local","threadId":"52725","inReplyTo":"3acac399-9476-e4ad-556e-e0138380eeb0@gmail.com","subject":"Re: [PATCH v2 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-09T23:32:27Z","receivedAt":"2020-02-09T23:32:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Feb 07, 2020 at 10:48:39AM -0500, Derrick Stolee wrote:\n> On 2/6/2020 2:41 PM, Martin Ågren wrote:\n> > On Wed, 5 Feb 2020 at 01:28, Taylor Blau wrote:\n> >> * If `--size-multiple=` is not specified, let `X` equal 2. If the new\n> >> tip file would have `N` commits and the previous tip has `M` commits and\n> >\n> >> - OPT_BOOL(0, \"split\", &opts.split,\n> >> - N_(\"allow writing an incremental commit-graph file\")),\n> >> + OPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n> >> + N_(\"allow writing an incremental commit-graph file\"),\n> >> + PARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n> >> + write_option_parse_split),\n> >\n> >\n> > I keep getting back to this -- sorry! So this actually forbids\n> > \"--no-split\", which used to work before. Unfortunate?\n>\n> That certainly is unfortunate. Hopefully no one is taking a dependence on\n> this, which only means something if they had a `--split` previously in\n> the command-line arguments.\n>\n> > I have to ask, what is the long-term plan for the two formats (split and\n> > non-split)? As I understand it, and I might well be wrong, the non-split\n> > format came first and the split format was a user-experience\n> > improvement. Should we expect that `--split` becomes the default?\n>\n> In some ways, the split is now the default because that is how it is\n> written during 'git fetch' using fetch.writeCommitGraph. However, I\n> don't think that it will ever become the default for the commit-graph\n> builtin.\n>\n> > In\n> > which case `--no-split` would be needed. Or might the non-split format\n> > go away entirely, leaving `--split` a no-op and `--split=` a\n> > pretty funky way of choosing a strategy for the one-and-only file\n> > format?\n>\n> In some ways, the --split=merge-all is similar, except it writes a one-line\n> commit-graph-chain file and puts a .graph file in\n> .git/objects/info/commit-graphs instead of writing to .git/objects/commit-graph.\n>\n> > To try to be concrete, here's a suggestion: `--format=split` and\n> > `--split-strategy=`.\n>\n> Why --format=split instead of leaving it as --[no-]split? Is there a reason to\n> introduce this string-based option when there are only two options right now?\n>\n> Perhaps using --split-strategy= is the most backwards-compatible\n> option, especially because we won't need --split=\"\" to substitute for\n> \"auto-merge\". However, I wonder if this is a case where we should make the\n> hard choice to sacrifice a narrow backwards-compatibility in favor of a\n> simplified set of options?\n\nMy preference would be the latter, which I vaguely indicated in my last\nemail to Martin. Like I said, I think that the number of hypothetical\ncases that we're breaking is pretty small, if not zero, and so I don't\nfeel too worried about changing the behavior like this.\n\nIf others feel strongly that keeping '--no-split' functional in the\nclassical sense is worthwhile, then I'm certainly happy to introduce\n'--split-strategy' as another option, but I think that we agree that the\nsimplicity is worth the tradeoff here.\n\n> Thanks,\n> -Stolee\n\nThanks,\nTaylor\n"},{"id":"391427","messageId":"20200209233242.GG4530@syl.local","threadId":"52725","inReplyTo":"CAN0heSrRJYc8DoWQ+NHR5LJ1z7d1GGBni2waP0_Lv7gMYjPPEg@mail.gmail.com","subject":"Re: [PATCH v2 3/3] builtin/commit-graph.c: support '--input=none'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-09T23:32:42Z","receivedAt":"2020-02-09T23:32:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Feb 06, 2020 at 08:50:57PM +0100, Martin Ågren wrote:\n> On Wed, 5 Feb 2020 at 01:28, Taylor Blau wrote:\n> > @@ -57,6 +57,12 @@ walking commits starting at all refs. (Cannot be combined with\n> > With the `--input=append` option, include all commits that are present\n> > in the existing commit-graph file.\n> > +\n> > +With the `--input=none` option, behave as if `--input=append` were\n> > +given, but do not walk other packs to find additional commits.\n> > +\n>\n> Similar to my comment in patch 1/3. Please add a \"+\" instead of an empty\n> line. This one actually trips up the rendering quite a bit of a lot of\n> what follows.\n\nThanks again :-).\n\n> > +If none of the above options are given, then generate the new\n> > +commit-graph by walking over all pack-indexes.\n> > ++\n>\n> This one's good.\n>\n> Martin\n\nThanks,\nTaylor\n"},{"id":"391592","messageId":"cover.1581486293.git.me@ttaylorr.com","threadId":"52725","inReplyTo":"cover.1580430057.git.me@ttaylorr.com","subject":"[PATCH v3 0/3] builtin/commit-graph.c: new split/merge options","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-12T05:47:50Z","receivedAt":"2020-02-12T05:47:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi everybody,\n\nAttached is what I anticipate/hope to be the final reroll of my series\nto add new arguments to the 'git commit-graph write --split' flag to\nallow callers to force or prohibit the commit-graph machinery to merge\nmultiple commit-graph layers.\n\nI was keeping an eye out for more discussion about whether or not these\nflags were acceptable by reviewers. Martin Ågren and Derrick Stolee have\nboth chimed in that they seem OK.\n\nSince there hasn't been much more discussion in this thread, I replayed\nthis series on top of 'tb/commit-graph-use-odb' (which was itself\nrebased on 'master'). I picked up a couple of ASCIIDoc changes along the\nway, and a range-diff is included below.\n\nThanks again.\n\nTaylor Blau (3):\n builtin/commit-graph.c: support '--split[=]'\n builtin/commit-graph.c: introduce '--input='\n builtin/commit-graph.c: support '--input=none'\n\n Documentation/git-commit-graph.txt | 51 ++++++++-----\n builtin/commit-graph.c | 115 +++++++++++++++++++++++------\n commit-graph.c | 28 ++++---\n commit-graph.h | 10 ++-\n t/t5318-commit-graph.sh | 4 +-\n t/t5324-split-commit-graph.sh | 53 ++++++++++++-\n 6 files changed, 205 insertions(+), 56 deletions(-)\n\nRange-diff against v2:\n1: 6428dac6e5 ! 1: e1635a0e34 builtin/commit-graph.c: support '--split[=]'\n @@ Documentation/git-commit-graph.txt: or `--stdin-packs`.)\n +strategy and other splitting options. The new commits not already in the\n +commit-graph are added in a new \"tip\" file. This file is merged with the\n +existing file if the following merge conditions are met:\n +++\n +* If `--split=merge-always` is specified, then a merge is always\n +conducted, and the remaining options are ignored. Conversely, if\n +`--split=no-merge` is specified, a merge is never performed, and the\n2: c7ba70e19d = 2: 655fe63076 builtin/commit-graph.c: introduce '--input='\n3: 7d6a608acd ! 3: 4e85c6f7e4 builtin/commit-graph.c: support '--input=none'\n @@ Documentation/git-commit-graph.txt: walking commits starting at all refs. (Canno\n +\n +With the `--input=none` option, behave as if `--input=append` were\n +given, but do not walk other packs to find additional commits.\n -+\n +++\n +If none of the above options are given, then generate the new\n +commit-graph by walking over all pack-indexes.\n ++\n--\n2.25.0.119.gaa12b7378b\n"},{"id":"391593","messageId":"e1635a0e34de62095e85ee3e42222c8fa2384688.1581486293.git.me@ttaylorr.com","threadId":"52725","inReplyTo":"cover.1581486293.git.me@ttaylorr.com","subject":"[PATCH v3 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-12T05:47:53Z","receivedAt":"2020-02-12T05:47:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"With '--split', the commit-graph machinery writes new commits in another\nincremental commit-graph which is part of the existing chain, and\noptionally decides to condense the chain into a single commit-graph.\nThis is done to ensure that the asymptotic behavior of looking up a\ncommit in an incremental chain is dominated by the number of\nincrementals in that chain. It can be controlled by the '--max-commits'\nand '--size-multiple' options.\n\nOn occasion, callers may want to ensure that 'git commit-graph write\n--split' always writes an incremental, and never spends effort\ncondensing the incremental chain [1]. Previously, this was possible by\npassing '--size-multiple=0', but this no longer the case following\n63020f175f (commit-graph: prefer default size_mult when given zero,\n2020-01-02).\n\nReintroduce a less-magical variant of the above with a new pair of\narguments to '--split': '--split=no-merge' and '--split=merge-all'. When\n'--split=no-merge' is given, the commit-graph machinery will never\ncondense an existing chain and will always write a new incremental.\nConversely, if '--split=merge-all' is given, any invocation including it\nwill always condense a chain if one exists. If '--split' is given with\nno arguments, it behaves as before and defers to '--size-multiple', and\nso on.\n\n[1]: This might occur when, for example, a server administrator running\nsome program after each push may want to ensure that each job runs\nproportional in time to the size of the push, and does not \"jump\" when\nthe commit-graph machinery decides to trigger a merge.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-commit-graph.txt | 17 ++++++++++-----\n builtin/commit-graph.c | 35 ++++++++++++++++++++++++++----\n commit-graph.c | 22 ++++++++++++-------\n commit-graph.h | 7 ++++++\n t/t5324-split-commit-graph.sh | 25 +++++++++++++++++++++\n 5 files changed, 89 insertions(+), 17 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 28d1fee505..269c355b0a 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -57,11 +57,18 @@ or `--stdin-packs`.)\n With the `--append` option, include all commits that are present in the\n existing commit-graph file.\n +\n-With the `--split` option, write the commit-graph as a chain of multiple\n-commit-graph files stored in `/info/commit-graphs`. The new commits\n-not already in the commit-graph are added in a new \"tip\" file. This file\n-is merged with the existing file if the following merge conditions are\n-met:\n+With the `--split[=]` option, write the commit-graph as a\n+chain of multiple commit-graph files stored in\n+`/info/commit-graphs`. Commit-graph layers are merged based on the\n+strategy and other splitting options. The new commits not already in the\n+commit-graph are added in a new \"tip\" file. This file is merged with the\n+existing file if the following merge conditions are met:\n++\n+* If `--split=merge-always` is specified, then a merge is always\n+conducted, and the remaining options are ignored. Conversely, if\n+`--split=no-merge` is specified, a merge is never performed, and the\n+remaining options are ignored. A bare `--split` defers to the remaining\n+options.\n +\n * If `--size-multiple=` is not specified, let `X` equal 2. If the new\n tip file would have `N` commits and the previous tip has `M` commits and\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 4a70b33fb5..4d3c1c46c2 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -9,7 +9,9 @@\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] [--[no-]progress] \"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] \"\n+\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n+\t \"[--[no-]progress] \"),\n \tNULL\n };\n \n@@ -19,7 +21,9 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append|--split] [--reachable|--stdin-packs|--stdin-commits] [--[no-]progress] \"),\n+\tN_(\"git commit-graph write [--object-dir ] [--append] \"\n+\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n+\t \"[--[no-]progress] \"),\n \tNULL\n };\n \n@@ -111,6 +115,27 @@ static int graph_verify(int argc, const char **argv)\n extern int read_replace_refs;\n static struct split_commit_graph_opts split_opts;\n \n+static int write_option_parse_split(const struct option *opt, const char *arg,\n+\t\t\t\t int unset)\n+{\n+\tenum commit_graph_split_flags *flags = opt->value;\n+\n+\topts.split = 1;\n+\tif (!arg) {\n+\t\t*flags = COMMIT_GRAPH_SPLIT_MERGE_AUTO;\n+\t\treturn 0;\n+\t}\n+\n+\tif (!strcmp(arg, \"merge-all\"))\n+\t\t*flags = COMMIT_GRAPH_SPLIT_MERGE_REQUIRED;\n+\telse if (!strcmp(arg, \"no-merge\"))\n+\t\t*flags = COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED;\n+\telse\n+\t\tdie(_(\"unrecognized --split argument, %s\"), arg);\n+\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tstruct string_list *pack_indexes = NULL;\n@@ -133,8 +158,10 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_BOOL(0, \"append\", &opts.append,\n \t\t\tN_(\"include all commits already in the commit-graph file\")),\n \t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n-\t\tOPT_BOOL(0, \"split\", &opts.split,\n-\t\t\tN_(\"allow writing an incremental commit-graph file\")),\n+\t\tOPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n+\t\t\tN_(\"allow writing an incremental commit-graph file\"),\n+\t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n+\t\t\twrite_option_parse_split),\n \t\tOPT_INTEGER(0, \"max-commits\", &split_opts.max_commits,\n \t\t\tN_(\"maximum number of commits in a non-base split commit-graph\")),\n \t\tOPT_INTEGER(0, \"size-multiple\", &split_opts.size_multiple,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 656dd647d5..3a5cb23cd7 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1533,27 +1533,33 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \n \tint max_commits = 0;\n \tint size_mult = 2;\n+\tenum commit_graph_split_flags flags = COMMIT_GRAPH_SPLIT_MERGE_AUTO;\n \n \tif (ctx->split_opts) {\n \t\tmax_commits = ctx->split_opts->max_commits;\n \n \t\tif (ctx->split_opts->size_multiple)\n \t\t\tsize_mult = ctx->split_opts->size_multiple;\n+\n+\t\tflags = ctx->split_opts->flags;\n \t}\n \n \tg = ctx->r->objects->commit_graph;\n \tnum_commits = ctx->commits.nr;\n \tctx->num_commit_graphs_after = ctx->num_commit_graphs_before + 1;\n \n-\twhile (g && (g->num_commits <= size_mult * num_commits ||\n-\t\t (max_commits && num_commits > max_commits))) {\n-\t\tif (g->odb != ctx->odb)\n-\t\t\tbreak;\n+\tif (flags != COMMIT_GRAPH_SPLIT_MERGE_PROHIBITED) {\n+\t\twhile (g && (g->num_commits <= size_mult * num_commits ||\n+\t\t\t (max_commits && num_commits > max_commits) ||\n+\t\t\t (flags == COMMIT_GRAPH_SPLIT_MERGE_REQUIRED))) {\n+\t\t\tif (g->odb != ctx->odb)\n+\t\t\t\tbreak;\n \n-\t\tnum_commits += g->num_commits;\n-\t\tg = g->base_graph;\n+\t\t\tnum_commits += g->num_commits;\n+\t\t\tg = g->base_graph;\n \n-\t\tctx->num_commit_graphs_after--;\n+\t\t\tctx->num_commit_graphs_after--;\n+\t\t}\n \t}\n \n \tctx->new_base_graph = g;\n@@ -1861,7 +1867,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\tgoto cleanup;\n \t}\n \n-\tif (!ctx->commits.nr)\n+\tif (!ctx->commits.nr && (!ctx->split_opts || ctx->split_opts->flags != COMMIT_GRAPH_SPLIT_MERGE_REQUIRED))\n \t\tgoto cleanup;\n \n \tif (ctx->split) {\ndiff --git a/commit-graph.h b/commit-graph.h\nindex e87a6f6360..65a7d2edae 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -82,10 +82,17 @@ enum commit_graph_write_flags {\n \tCOMMIT_GRAPH_WRITE_CHECK_OIDS = (1 << 3)\n };\n \n+enum commit_graph_split_flags {\n+\tCOMMIT_GRAPH_SPLIT_MERGE_AUTO = 0,\n+\tCOMMIT_GRAPH_SPLIT_MERGE_REQUIRED = 1,\n+\tCOMMIT_GRAPH_SPLIT_MERGE_PROHIBITED = 2\n+};\n+\n struct split_commit_graph_opts {\n \tint size_multiple;\n \tint max_commits;\n \ttimestamp_t expire_time;\n+\tenum commit_graph_split_flags flags;\n };\n \n /*\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 53b2e6b455..bb2b724178 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -351,4 +351,29 @@ test_expect_success 'split across alternate where alternate is not split' '\n \ttest_cmp commit-graph .git/objects/info/commit-graph\n '\n \n+test_expect_success '--split=merge-all always merges incrementals' '\n+\ttest_when_finished rm -rf a b c &&\n+\trm -rf $graphdir $infodir/commit-graph &&\n+\tgit reset --hard commits/10 &&\n+\tgit rev-list -3 HEAD~4 >a &&\n+\tgit rev-list -2 HEAD~2 >b &&\n+\tgit rev-list -2 HEAD >c &&\n+\tgit commit-graph write --split=no-merge --stdin-commits a &&\n+\tgit rev-list HEAD >b &&\n+\tgit commit-graph write --split --stdin-commits '","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-12T05:47:55Z","receivedAt":"2020-02-12T05:48:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The 'write' mode of the 'commit-graph' supports input from a number of\ndifferent sources: pack indexes over stdin, commits over stdin, commits\nreachable from all references, and so on. Each of these options are\nspecified with a unique option: '--stdin-packs', '--stdin-commits', etc.\n\nSimilar to our replacement of 'git config [--]' with 'git config\n[--type=]' (c.f., fb0dc3bac1 (builtin/config.c: support\n`--type=` as preferred alias for `--`, 2018-04-18)), softly\ndeprecate '[--]' in favor of '[--input=]'.\n\nThis makes it more clear to implement new options that are combinations\nof other options (such as, for example, \"none\", a combination of the old\n\"--append\" and a new sentinel to specify to _not_ look in other packs,\nwhich we will implement in a future patch).\n\nUnfortunately, the new enumerated type is a bitfield, even though it\nmakes much more sense as '0, 1, 2, ...'. Even though *almost* all\noptions are pairwise exclusive, '--stdin-{packs,commits}' *is*\ncompatible with '--append'. For this reason, use a bitfield.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-commit-graph.txt | 26 +++++-----\n builtin/commit-graph.c | 83 +++++++++++++++++++++---------\n t/t5318-commit-graph.sh | 4 +-\n t/t5324-split-commit-graph.sh | 2 +-\n 4 files changed, 76 insertions(+), 39 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 269c355b0a..0a320cccdd 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -41,21 +41,21 @@ COMMANDS\n \n Write a commit-graph file based on the commits found in packfiles.\n +\n-With the `--stdin-packs` option, generate the new commit graph by\n+With the `--input=stdin-packs` option, generate the new commit graph by\n walking objects only in the specified pack-indexes. (Cannot be combined\n-with `--stdin-commits` or `--reachable`.)\n+with `--input=stdin-commits` or `--input=reachable`.)\n +\n-With the `--stdin-commits` option, generate the new commit graph by\n-walking commits starting at the commits specified in stdin as a list\n+With the `--input=stdin-commits` option, generate the new commit graph\n+by walking commits starting at the commits specified in stdin as a list\n of OIDs in hex, one OID per line. (Cannot be combined with\n-`--stdin-packs` or `--reachable`.)\n+`--input=stdin-packs` or `--input=reachable`.)\n +\n-With the `--reachable` option, generate the new commit graph by walking\n-commits starting at all refs. (Cannot be combined with `--stdin-commits`\n-or `--stdin-packs`.)\n+With the `--input=reachable` option, generate the new commit graph by\n+walking commits starting at all refs. (Cannot be combined with\n+`--input=stdin-commits` or `--input=stdin-packs`.)\n +\n-With the `--append` option, include all commits that are present in the\n-existing commit-graph file.\n+With the `--input=append` option, include all commits that are present\n+in the existing commit-graph file.\n +\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n@@ -106,20 +106,20 @@ $ git commit-graph write\n using commits in ``.\n +\n ------------------------------------------------\n-$ echo | git commit-graph write --stdin-packs\n+$ echo | git commit-graph write --input=stdin-packs\n ------------------------------------------------\n \n * Write a commit-graph file containing all reachable commits.\n +\n ------------------------------------------------\n-$ git show-ref -s | git commit-graph write --stdin-commits\n+$ git show-ref -s | git commit-graph write --input=stdin-commits\n ------------------------------------------------\n \n * Write a commit-graph file containing all commits in the current\n commit-graph file along with those reachable from `HEAD`.\n +\n ------------------------------------------------\n-$ git rev-parse HEAD | git commit-graph write --stdin-commits --append\n+$ git rev-parse HEAD | git commit-graph write --input=stdin-commits --input=append\n ------------------------------------------------\n \n \ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 4d3c1c46c2..0ff25896d0 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -9,8 +9,9 @@\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--append] \"\n-\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n+\tN_(\"git commit-graph write [--object-dir ] \"\n+\t \"[--split[=]] \"\n+\t \"[--input=] \"\n \t \"[--[no-]progress] \"),\n \tNULL\n };\n@@ -21,18 +22,23 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--append] \"\n-\t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n+\tN_(\"git commit-graph write [--object-dir ] \"\n+\t \"[--split[=]] \"\n+\t \"[--input=] \"\n \t \"[--[no-]progress] \"),\n \tNULL\n };\n \n+enum commit_graph_input {\n+\tCOMMIT_GRAPH_INPUT_REACHABLE = (1 << 1),\n+\tCOMMIT_GRAPH_INPUT_STDIN_PACKS = (1 << 2),\n+\tCOMMIT_GRAPH_INPUT_STDIN_COMMITS = (1 << 3),\n+\tCOMMIT_GRAPH_INPUT_APPEND = (1 << 4)\n+};\n+\n static struct opts_commit_graph {\n \tconst char *obj_dir;\n-\tint reachable;\n-\tint stdin_packs;\n-\tint stdin_commits;\n-\tint append;\n+\tenum commit_graph_input input;\n \tint split;\n \tint shallow;\n \tint progress;\n@@ -57,6 +63,28 @@ static struct object_directory *find_odb(struct repository *r,\n \treturn odb;\n }\n \n+static int option_parse_input(const struct option *opt, const char *arg,\n+\t\t\t int unset)\n+{\n+\tenum commit_graph_input *to = opt->value;\n+\tif (unset || !strcmp(arg, \"packs\")) {\n+\t\t*to = 0;\n+\t\treturn 0;\n+\t}\n+\n+\tif (!strcmp(arg, \"reachable\"))\n+\t\t*to |= COMMIT_GRAPH_INPUT_REACHABLE;\n+\telse if (!strcmp(arg, \"stdin-packs\"))\n+\t\t*to |= COMMIT_GRAPH_INPUT_STDIN_PACKS;\n+\telse if (!strcmp(arg, \"stdin-commits\"))\n+\t\t*to |= COMMIT_GRAPH_INPUT_STDIN_COMMITS;\n+\telse if (!strcmp(arg, \"append\"))\n+\t\t*to |= COMMIT_GRAPH_INPUT_APPEND;\n+\telse\n+\t\tdie(_(\"unrecognized --input source, %s\"), arg);\n+\treturn 0;\n+}\n+\n static int graph_verify(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n@@ -149,14 +177,21 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_STRING(0, \"object-dir\", &opts.obj_dir,\n \t\t\tN_(\"dir\"),\n \t\t\tN_(\"The object directory to store the graph\")),\n-\t\tOPT_BOOL(0, \"reachable\", &opts.reachable,\n-\t\t\tN_(\"start walk at all refs\")),\n-\t\tOPT_BOOL(0, \"stdin-packs\", &opts.stdin_packs,\n-\t\t\tN_(\"scan pack-indexes listed by stdin for commits\")),\n-\t\tOPT_BOOL(0, \"stdin-commits\", &opts.stdin_commits,\n-\t\t\tN_(\"start walk at commits listed by stdin\")),\n-\t\tOPT_BOOL(0, \"append\", &opts.append,\n-\t\t\tN_(\"include all commits already in the commit-graph file\")),\n+\t\tOPT_CALLBACK(0, \"input\", &opts.input, NULL,\n+\t\t\tN_(\"include commits from this source in the graph\"),\n+\t\t\toption_parse_input),\n+\t\tOPT_BIT(0, \"reachable\", &opts.input,\n+\t\t\tN_(\"start walk at all refs\"),\n+\t\t\tCOMMIT_GRAPH_INPUT_REACHABLE),\n+\t\tOPT_BIT(0, \"stdin-packs\", &opts.input,\n+\t\t\tN_(\"scan pack-indexes listed by stdin for commits\"),\n+\t\t\tCOMMIT_GRAPH_INPUT_STDIN_PACKS),\n+\t\tOPT_BIT(0, \"stdin-commits\", &opts.input,\n+\t\t\tN_(\"start walk at commits listed by stdin\"),\n+\t\t\tCOMMIT_GRAPH_INPUT_STDIN_COMMITS),\n+\t\tOPT_BIT(0, \"append\", &opts.input,\n+\t\t\tN_(\"include all commits already in the commit-graph file\"),\n+\t\t\tCOMMIT_GRAPH_INPUT_APPEND),\n \t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n \t\tOPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n \t\t\tN_(\"allow writing an incremental commit-graph file\"),\n@@ -182,11 +217,13 @@ static int graph_write(int argc, const char **argv)\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\n \n-\tif (opts.reachable + opts.stdin_packs + opts.stdin_commits > 1)\n-\t\tdie(_(\"use at most one of --reachable, --stdin-commits, or --stdin-packs\"));\n+\tif ((!!(opts.input & COMMIT_GRAPH_INPUT_REACHABLE) +\n+\t !!(opts.input & COMMIT_GRAPH_INPUT_STDIN_PACKS) +\n+\t !!(opts.input & COMMIT_GRAPH_INPUT_STDIN_COMMITS)) > 1)\n+\t\tdie(_(\"use at most one of --input=reachable, --input=stdin-commits, or --input=stdin-packs\"));\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n-\tif (opts.append)\n+\tif (opts.input & COMMIT_GRAPH_INPUT_APPEND)\n \t\tflags |= COMMIT_GRAPH_WRITE_APPEND;\n \tif (opts.split)\n \t\tflags |= COMMIT_GRAPH_WRITE_SPLIT;\n@@ -196,22 +233,22 @@ static int graph_write(int argc, const char **argv)\n \tread_replace_refs = 0;\n \todb = find_odb(the_repository, opts.obj_dir);\n \n-\tif (opts.reachable) {\n+\tif (opts.input & COMMIT_GRAPH_INPUT_REACHABLE) {\n \t\tif (write_commit_graph_reachable(odb, flags, &split_opts))\n \t\t\treturn 1;\n \t\treturn 0;\n \t}\n \n \tstring_list_init(&lines, 0);\n-\tif (opts.stdin_packs || opts.stdin_commits) {\n+\tif (opts.input & (COMMIT_GRAPH_INPUT_STDIN_PACKS | COMMIT_GRAPH_INPUT_STDIN_COMMITS)) {\n \t\tstruct strbuf buf = STRBUF_INIT;\n \n \t\twhile (strbuf_getline(&buf, stdin) != EOF)\n \t\t\tstring_list_append(&lines, strbuf_detach(&buf, NULL));\n \n-\t\tif (opts.stdin_packs)\n+\t\tif (opts.input & COMMIT_GRAPH_INPUT_STDIN_PACKS)\n \t\t\tpack_indexes = &lines;\n-\t\tif (opts.stdin_commits) {\n+\t\tif (opts.input & COMMIT_GRAPH_INPUT_STDIN_COMMITS) {\n \t\t\tcommit_hex = &lines;\n \t\t\tflags |= COMMIT_GRAPH_WRITE_CHECK_OIDS;\n \t\t}\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 07b3207595..1aa81d2383 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -227,7 +227,7 @@ graph_git_behavior 'cleared graph, commit 8 vs merge 2' full commits/8 merge/2\n \n test_expect_success 'build graph from latest pack with closure' '\n \tcd \"$TRASH_DIRECTORY/full\" &&\n-\tcat new-idx | git commit-graph write --stdin-packs &&\n+\tcat new-idx | git commit-graph write --input=stdin-packs &&\n \ttest_path_is_file $objdir/info/commit-graph &&\n \tgraph_read_expect \"9\" \"extra_edges\"\n '\n@@ -240,7 +240,7 @@ test_expect_success 'build graph from commits with closure' '\n \tgit tag -a -m \"merge\" tag/merge merge/2 &&\n \tgit rev-parse tag/merge >commits-in &&\n \tgit rev-parse merge/1 >>commits-in &&\n-\tcat commits-in | git commit-graph write --stdin-commits &&\n+\tcat commits-in | git commit-graph write --input=stdin-commits &&\n \ttest_path_is_file $objdir/info/commit-graph &&\n \tgraph_read_expect \"6\"\n '\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex bb2b724178..6894106727 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -42,7 +42,7 @@ test_expect_success 'create commits and write commit-graph' '\n \t\ttest_commit $i &&\n \t\tgit branch commits/$i || return 1\n \tdone &&\n-\tgit commit-graph write --reachable &&\n+\tgit commit-graph write --input=reachable &&\n \ttest_path_is_file $infodir/commit-graph &&\n \tgraph_read_expect 3\n '\n-- \n2.25.0.119.gaa12b7378b\n\n"},{"id":"391595","messageId":"4e85c6f7e40e7d6a8d93574645d65971b7cfa4f8.1581486293.git.me@ttaylorr.com","threadId":"52725","inReplyTo":"cover.1581486293.git.me@ttaylorr.com","subject":"[PATCH v3 3/3] builtin/commit-graph.c: support '--input=none'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-12T05:47:57Z","receivedAt":"2020-02-12T05:48:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In the previous commit, we introduced '--split=',\nand alluded to the fact that '--split=merge-all' would be useful for\ncallers who wish to always trigger a merge of an incremental chain.\n\nThere is a problem with the above approach, which is that there is no\nway to specify to the commit-graph builtin that a caller only wants to\ninclude commits already in the graph. One can specify '--input=append'\nto include all commits in the existing graphs, but the absence of\n'--input=stdin-{commits,packs}' causes the builtin to call\n'fill_oids_from_all_packs()'.\n\nPassing '--input=reachable' (as in 'git commit-graph write\n--split=merge-all --input=reachable --input=append') works around this\nissue by making '--input=reachable' effectively a no-op, but this can be\nprohibitively expensive in large repositories, making it an undesirable\nchoice for some users.\n\nTeach '--input=none' as an option to behave as if '--input=append' were\ngiven, but to consider no other sources in addition.\n\nThis, in conjunction with the option introduced in the previous patch\noffers the convenient way to force the commit-graph machinery to\ncondense a chain of incrementals without requiring any new commits:\n\n $ git commit-graph write --split=merge-all --input=none\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-commit-graph.txt | 8 +++++++-\n builtin/commit-graph.c | 11 ++++++++---\n commit-graph.c | 6 ++++--\n commit-graph.h | 3 ++-\n t/t5324-split-commit-graph.sh | 26 ++++++++++++++++++++++++++\n 5 files changed, 47 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 0a320cccdd..b210cef52f 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -39,7 +39,7 @@ COMMANDS\n --------\n 'write'::\n \n-Write a commit-graph file based on the commits found in packfiles.\n+Write a commit-graph file based on the specified sources of input:\n +\n With the `--input=stdin-packs` option, generate the new commit graph by\n walking objects only in the specified pack-indexes. (Cannot be combined\n@@ -57,6 +57,12 @@ walking commits starting at all refs. (Cannot be combined with\n With the `--input=append` option, include all commits that are present\n in the existing commit-graph file.\n +\n+With the `--input=none` option, behave as if `--input=append` were\n+given, but do not walk other packs to find additional commits.\n++\n+If none of the above options are given, then generate the new\n+commit-graph by walking over all pack-indexes.\n++\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n `/info/commit-graphs`. Commit-graph layers are merged based on the\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 0ff25896d0..a71af88815 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -11,7 +11,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n \tN_(\"git commit-graph write [--object-dir ] \"\n \t \"[--split[=]] \"\n-\t \"[--input=] \"\n+\t \"[--input=] \"\n \t \"[--[no-]progress] \"),\n \tNULL\n };\n@@ -24,7 +24,7 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n static const char * const builtin_commit_graph_write_usage[] = {\n \tN_(\"git commit-graph write [--object-dir ] \"\n \t \"[--split[=]] \"\n-\t \"[--input=] \"\n+\t \"[--input=] \"\n \t \"[--[no-]progress] \"),\n \tNULL\n };\n@@ -33,7 +33,8 @@ enum commit_graph_input {\n \tCOMMIT_GRAPH_INPUT_REACHABLE = (1 << 1),\n \tCOMMIT_GRAPH_INPUT_STDIN_PACKS = (1 << 2),\n \tCOMMIT_GRAPH_INPUT_STDIN_COMMITS = (1 << 3),\n-\tCOMMIT_GRAPH_INPUT_APPEND = (1 << 4)\n+\tCOMMIT_GRAPH_INPUT_APPEND = (1 << 4),\n+\tCOMMIT_GRAPH_INPUT_NONE = (1 << 5)\n };\n \n static struct opts_commit_graph {\n@@ -80,6 +81,8 @@ static int option_parse_input(const struct option *opt, const char *arg,\n \t\t*to |= COMMIT_GRAPH_INPUT_STDIN_COMMITS;\n \telse if (!strcmp(arg, \"append\"))\n \t\t*to |= COMMIT_GRAPH_INPUT_APPEND;\n+\telse if (!strcmp(arg, \"none\"))\n+\t\t*to |= (COMMIT_GRAPH_INPUT_APPEND | COMMIT_GRAPH_INPUT_NONE);\n \telse\n \t\tdie(_(\"unrecognized --input source, %s\"), arg);\n \treturn 0;\n@@ -225,6 +228,8 @@ static int graph_write(int argc, const char **argv)\n \t\topts.obj_dir = get_object_directory();\n \tif (opts.input & COMMIT_GRAPH_INPUT_APPEND)\n \t\tflags |= COMMIT_GRAPH_WRITE_APPEND;\n+\tif (opts.input & COMMIT_GRAPH_INPUT_NONE)\n+\t\tflags |= COMMIT_GRAPH_WRITE_NO_INPUT;\n \tif (opts.split)\n \t\tflags |= COMMIT_GRAPH_WRITE_SPLIT;\n \tif (opts.progress)\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 3a5cb23cd7..417b7eac9c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -788,7 +788,8 @@ struct write_commit_graph_context {\n \tunsigned append:1,\n \t\t report_progress:1,\n \t\t split:1,\n-\t\t check_oids:1;\n+\t\t check_oids:1,\n+\t\t no_input:1;\n \n \tconst struct split_commit_graph_opts *split_opts;\n };\n@@ -1785,6 +1786,7 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->split = flags & COMMIT_GRAPH_WRITE_SPLIT ? 1 : 0;\n \tctx->check_oids = flags & COMMIT_GRAPH_WRITE_CHECK_OIDS ? 1 : 0;\n \tctx->split_opts = split_opts;\n+\tctx->no_input = flags & COMMIT_GRAPH_WRITE_NO_INPUT ? 1 : 0;\n \n \tif (ctx->split) {\n \t\tstruct commit_graph *g;\n@@ -1843,7 +1845,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\t\tgoto cleanup;\n \t}\n \n-\tif (!pack_indexes && !commit_hex)\n+\tif (!ctx->no_input && !pack_indexes && !commit_hex)\n \t\tfill_oids_from_all_packs(ctx);\n \n \tclose_reachable(ctx);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 65a7d2edae..df7f3f5961 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -79,7 +79,8 @@ enum commit_graph_write_flags {\n \tCOMMIT_GRAPH_WRITE_PROGRESS = (1 << 1),\n \tCOMMIT_GRAPH_WRITE_SPLIT = (1 << 2),\n \t/* Make sure that each OID in the input is a valid commit OID. */\n-\tCOMMIT_GRAPH_WRITE_CHECK_OIDS = (1 << 3)\n+\tCOMMIT_GRAPH_WRITE_CHECK_OIDS = (1 << 3),\n+\tCOMMIT_GRAPH_WRITE_NO_INPUT = (1 << 4)\n };\n \n enum commit_graph_split_flags {\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 6894106727..7614f3915b 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -376,4 +376,30 @@ test_expect_success '--split=no-merge always writes an incremental' '\n \ttest_line_count = 2 $graphdir/commit-graph-chain\n '\n \n+test_expect_success '--split=no-merge, --input=none writes nothing' '\n+\ttest_when_finished rm -rf a graphs.before graphs.after &&\n+\trm -rf $graphdir &&\n+\tgit reset --hard commits/2 &&\n+\tgit rev-list -1 HEAD~1 >a &&\n+\tgit commit-graph write --split=no-merge --input=stdin-commits graphs.before &&\n+\ttest_line_count = 1 $graphdir/commit-graph-chain &&\n+\tgit commit-graph write --split --input=none &&\n+\tls $graphdir/graph-*.graph >graphs.after &&\n+\ttest_cmp graphs.before graphs.after\n+'\n+\n+test_expect_success '--split=merge-all, --input=none merges the chain' '\n+\ttest_when_finished rm -rf a b &&\n+\trm -rf $graphdir &&\n+\tgit reset --hard commits/2 &&\n+\tgit rev-list -1 HEAD~1 >a &&\n+\tgit rev-list -1 HEAD >b &&\n+\tgit commit-graph write --split=no-merge --input=stdin-commits ]'","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-02-12T06:03:46Z","receivedAt":"2020-02-12T06:04:04Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Fri, 7 Feb 2020 at 16:48, Derrick Stolee wrote:\n>\n> On 2/6/2020 2:41 PM, Martin Ågren wrote:\n> > On Wed, 5 Feb 2020 at 01:28, Taylor Blau wrote:\n> >> - OPT_BOOL(0, \"split\", &opts.split,\n> >> - N_(\"allow writing an incremental commit-graph file\")),\n> >> + OPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n> >> + N_(\"allow writing an incremental commit-graph file\"),\n> >> + PARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n> >> + write_option_parse_split),\n> >\n> >\n> > I keep getting back to this -- sorry! So this actually forbids\n> > \"--no-split\", which used to work before. Unfortunate?\n>\n> That certainly is unfortunate. Hopefully no one is taking a dependence on\n> this, which only means something if they had a `--split` previously in\n> the command-line arguments.\n>\n> > I have to ask, what is the long-term plan for the two formats (split and\n> > non-split)? As I understand it, and I might well be wrong, the non-split\n> > format came first and the split format was a user-experience\n> > improvement. Should we expect that `--split` becomes the default?\n>\n> In some ways, the split is now the default because that is how it is\n> written during 'git fetch' using fetch.writeCommitGraph. However, I\n> don't think that it will ever become the default for the commit-graph\n> builtin.\n\nThanks for giving this piece of background.\n\n> > To try to be concrete, here's a suggestion: `--format=split` and\n> > `--split-strategy=`.\n>\n> Why --format=split instead of leaving it as --[no-]split? Is there a reason to\n> introduce this string-based option when there are only two options right now?\n\nMy thinking was, if my concern is \"--split\" being overloaded, what would\nit look like to \"unload\" it entirely? From \"--split\" it isn't obvious\nwhether it's a verb or an adjective (shall we split, or shall we do\nthings the split way?). Having \"--format=split\" would help avoid *that*,\npossibly leaving a cleaner field for the issue of \"do we\nallow/force/forbid the 'merging' to happen?\". But I'm happy to accept\n\"--split=\" and move on. :-)\n\nI see that Taylor juuust posted a v3. I'll try to find time to look it\nover, but I won't be raising this point again.\n\nMartin\n"},{"id":"391608","messageId":"xmqqy2t7u0rh.fsf@gitster-ct.c.googlers.com","threadId":"52725","inReplyTo":"cover.1581486293.git.me@ttaylorr.com","subject":"Re: [PATCH v3 0/3] builtin/commit-graph.c: new split/merge options","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-02-12T18:19:14Z","receivedAt":"2020-02-12T18:19:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Attached is what I anticipate/hope to be the final reroll of my series\n> to add new arguments to the 'git commit-graph write --split' flag to\n> allow callers to force or prohibit the commit-graph machinery to merge\n> multiple commit-graph layers.\n>\n> I was keeping an eye out for more discussion about whether or not these\n> flags were acceptable by reviewers. Martin Ågren and Derrick Stolee have\n> both chimed in that they seem OK.\n>\n> Since there hasn't been much more discussion in this thread, I replayed\n> this series on top of 'tb/commit-graph-use-odb' (which was itself\n> rebased on 'master'). I picked up a couple of ASCIIDoc changes along the\n> way, and a range-diff is included below.\n\nI haven't had a chance to form an opinion on this topic, and will\nlet others comment on it first.\n\nThis topic and the \"changed paths bloom filter\" topic obviously and\ninevitably have trivial textual conflicts. When today's integration\nresult is pushed out in 6 hours or so, please see if the resolution\nis reasonable in 'pu'.\n\nThanks.\n"},{"id":"391619","messageId":"20200212205028.GE4364@syl.local","threadId":"52725","inReplyTo":"CAN0heSrNJ5XdJ=w=xgNQORAyprOD6xudivveroXV-OJwO6nBvg@mail.gmail.com","subject":"Re: [PATCH v2 1/3] builtin/commit-graph.c: support '--split[=]'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-12T20:50:28Z","receivedAt":"2020-02-12T20:50:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Feb 12, 2020 at 07:03:46AM +0100, Martin Ågren wrote:\n> On Fri, 7 Feb 2020 at 16:48, Derrick Stolee wrote:\n> >\n> > On 2/6/2020 2:41 PM, Martin Ågren wrote:\n> > > On Wed, 5 Feb 2020 at 01:28, Taylor Blau wrote:\n> > >> - OPT_BOOL(0, \"split\", &opts.split,\n> > >> - N_(\"allow writing an incremental commit-graph file\")),\n> > >> + OPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n> > >> + N_(\"allow writing an incremental commit-graph file\"),\n> > >> + PARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n> > >> + write_option_parse_split),\n> > >\n> > >\n> > > I keep getting back to this -- sorry! So this actually forbids\n> > > \"--no-split\", which used to work before. Unfortunate?\n> >\n> > That certainly is unfortunate. Hopefully no one is taking a dependence on\n> > this, which only means something if they had a `--split` previously in\n> > the command-line arguments.\n> >\n> > > I have to ask, what is the long-term plan for the two formats (split and\n> > > non-split)? As I understand it, and I might well be wrong, the non-split\n> > > format came first and the split format was a user-experience\n> > > improvement. Should we expect that `--split` becomes the default?\n> >\n> > In some ways, the split is now the default because that is how it is\n> > written during 'git fetch' using fetch.writeCommitGraph. However, I\n> > don't think that it will ever become the default for the commit-graph\n> > builtin.\n>\n> Thanks for giving this piece of background.\n>\n> > > To try to be concrete, here's a suggestion: `--format=split` and\n> > > `--split-strategy=`.\n> >\n> > Why --format=split instead of leaving it as --[no-]split? Is there a reason to\n> > introduce this string-based option when there are only two options right now?\n>\n> My thinking was, if my concern is \"--split\" being overloaded, what would\n> it look like to \"unload\" it entirely? From \"--split\" it isn't obvious\n> whether it's a verb or an adjective (shall we split, or shall we do\n> things the split way?). Having \"--format=split\" would help avoid *that*,\n> possibly leaving a cleaner field for the issue of \"do we\n> allow/force/forbid the 'merging' to happen?\". But I'm happy to accept\n> \"--split=\" and move on. :-)\n>\n> I see that Taylor juuust posted a v3. I'll try to find time to look it\n> over, but I won't be raising this point again.\n\nIt looks like we raced :-). Sorry about that. I didn't see your email\nuntil after I sent, and I certainly would have waited if I knew that you\nwere writing an email to the same thread as I was working in at the same\ntime.\n\nI'm still fairly happy with the '--split[=]' approach that is\nimplemented in all versions of this patch series, although I do\nunderstand your suggestions.\n\nMy preference would be to see if anybody else feels like the trade-off\n*is* worth it (I explained earlier in the thread some reasons why I feel\nthat the trade-off is *not* worth it), but I'd be happy to move this\nseries forward as-is unless others echo this idea.\n\n> Martin\n\nThanks,\nTaylor\n"},{"id":"391653","messageId":"20200213113313.GO10482@szeder.dev","threadId":"52725","inReplyTo":"20200204045124.GG5790@syl.local","subject":"Re: [PATCH 2/3] builtin/commit-graph.c: introduce '--input='","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-02-13T11:33:13Z","receivedAt":"2020-02-13T11:33:21Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Feb 03, 2020 at 08:51:24PM -0800, Taylor Blau wrote:\n> On Fri, Jan 31, 2020 at 08:34:41PM +0100, Martin Ågren wrote:\n> > On Fri, 31 Jan 2020 at 01:30, Taylor Blau wrote:\n> > > The 'write' mode of the 'commit-graph' supports input from a number of\n\ns/mode/subcommand/\n\n> > > different sources:\n\nI note that you use the word \"sources\" here, in the subject line as\nwell (as '--input='), and in the code as well (e.g. the in\nthe error message \"unrecognized --input source, %s\"). I like this\nword, I think the words \"input\" and \"source\" go really well together.\n\n> > > pack indexes over stdin, commits over stdin, commits\n> > > reachable from all references, and so on.\n\nIt's interesting to see that you stopped listing and went for \"and so\non\" right when it got interesting/controversial with '--append'... :)\n\n> > > Each of these options are\n> > > specified with a unique option: '--stdin-packs', '--stdin-commits', etc.\n\nIt also supports the very inefficient scanning through all objects in\nall pack files to find commit objects, which, sadly, ended up being\nthe default, and thus doesn't have its own --option. Should there be\na corresponding '--input=' as well? (Note that I don't mean\nthis as a suggestion to add one; on the contrary, the less exposure it\ngets the better.)\n\n> > > Similar to our replacement of 'git config [--]' with 'git config\n> > > [--type=]' (c.f., fb0dc3bac1 (builtin/config.c: support\n> > > `--type=` as preferred alias for `--`, 2018-04-18)), softly\n> > > deprecate '[--]' in favor of '[--input=]'.\n> > >\n> > > This makes it more clear to implement new options that are combinations\n> > > of other options (such as, for example, \"none\", a combination of the old\n> > > \"--append\" and a new sentinel to specify to _not_ look in other packs,\n> > > which we will implement in a future patch).\n> >\n> > Makes sense.\n> >\n> > > Unfortunately, the new enumerated type is a bitfield, even though it\n> > > makes much more sense as '0, 1, 2, ...'. Even though *almost* all\n> > > options are pairwise exclusive, '--stdin-{packs,commits}' *is*\n> > > compatible with '--append'. For this reason, use a bitfield.\n> >\n> > > -With the `--append` option, include all commits that are present in the\n> > > -existing commit-graph file.\n> > > +With the `--input=append` option, include all commits that are present\n> > > +in the existing commit-graph file.\n> >\n> > Would it be too crazy to call this `--input=existing` instead, and have\n> > it be the same as `--append`? I find that `--append` makes a lot of\n> > sense (it's a mode we can turn on or off), whereas \"input = append\"\n> > seems more odd.\n> \n> Hmm. When I wrote this, I was thinking of introducing equivalent options\n> that are identical in name and functionality as '--input=' instead\n> of '--'. So, I guess that is to say that I didn't spend an awful\n> amount of time thinking about whether or not '--input=append' made sense\n> given anything else.\n> \n> So, I don't think that '--input=existing' is a bad idea at all, but I do\n> worry about advertising this deprecation as \"'--' becomes\n> '--input=', except when your mode is 'append', in which case it\n> becomes '--input=existing'\".\n\nBut here you suddenly start using the word \"mode\" both in\n'--input=' and in '--'.\n\nOn one hand, I don't think that the word \"mode\" goes as well with\n\"input\" as \"source\" does.\n\nOn the other, is '--append' really a source/mode, like '--reachable'\nand '--stdin-commits' are? Source, no: from wordsmithing perspective\nit doesn't fit with \"source\", and being orthogonal to the \"real\"\nsource options while they are mutually exclusive seems to be a clear\nindication that it isn't. Mode, yes: it's a mode of operation where\nno longer reachable/present commits are not discarded from the\ncommit-graph.\n\nSo I don't think that adding '--input=append' is a good idea, even if\nwe were call it differently, e.g. '--input=existing' as suggested\nabove.\n\nHowever, I do think that '--input=existing' would better express what\n'--input=none' in the next patch wants to achieve.\n\n"},{"id":"391654","messageId":"20200213113925.GJ2868@szeder.dev","threadId":"52725","inReplyTo":"4e85c6f7e40e7d6a8d93574645d65971b7cfa4f8.1581486293.git.me@ttaylorr.com","subject":"Re: [PATCH v3 3/3] builtin/commit-graph.c: support '--input=none'","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-02-13T11:39:25Z","receivedAt":"2020-02-13T11:39:31Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Feb 11, 2020 at 09:47:57PM -0800, Taylor Blau wrote:\n> In the previous commit, we introduced '--split=',\n\nNit: the previous commit introduced '--input='.\n\n> and alluded to the fact that '--split=merge-all' would be useful for\n> callers who wish to always trigger a merge of an incremental chain.\n"},{"id":"391655","messageId":"20200213114800.GP10482@szeder.dev","threadId":"52725","inReplyTo":"20200213113313.GO10482@szeder.dev","subject":"Re: [PATCH 2/3] builtin/commit-graph.c: introduce '--input='","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-02-13T11:48:00Z","receivedAt":"2020-02-13T11:48:07Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Feb 13, 2020 at 12:33:13PM +0100, SZEDER Gábor wrote:\n> On Mon, Feb 03, 2020 at 08:51:24PM -0800, Taylor Blau wrote:\n> > On Fri, Jan 31, 2020 at 08:34:41PM +0100, Martin Ågren wrote:\n> > > On Fri, 31 Jan 2020 at 01:30, Taylor Blau wrote:\n> > > > The 'write' mode of the 'commit-graph' supports input from a number of\n> \n> s/mode/subcommand/\n> \n> > > > different sources:\n> \n> I note that you use the word \"sources\" here, in the subject line as\n> well (as '--input='), and in the code as well (e.g. the in\n> the error message \"unrecognized --input source, %s\"). I like this\n> word, I think the words \"input\" and \"source\" go really well together.\n> \n> > > > pack indexes over stdin, commits over stdin, commits\n> > > > reachable from all references, and so on.\n> \n> It's interesting to see that you stopped listing and went for \"and so\n> on\" right when it got interesting/controversial with '--append'... :)\n> \n> > > > Each of these options are\n> > > > specified with a unique option: '--stdin-packs', '--stdin-commits', etc.\n> \n> It also supports the very inefficient scanning through all objects in\n> all pack files to find commit objects, which, sadly, ended up being\n> the default, and thus doesn't have its own --option. Should there be\n> a corresponding '--input=' as well? (Note that I don't mean\n> this as a suggestion to add one; on the contrary, the less exposure it\n> gets the better.)\n> \n> > > > Similar to our replacement of 'git config [--]' with 'git config\n> > > > [--type=]' (c.f., fb0dc3bac1 (builtin/config.c: support\n> > > > `--type=` as preferred alias for `--`, 2018-04-18)), softly\n> > > > deprecate '[--]' in favor of '[--input=]'.\n> > > >\n> > > > This makes it more clear to implement new options that are combinations\n> > > > of other options (such as, for example, \"none\", a combination of the old\n> > > > \"--append\" and a new sentinel to specify to _not_ look in other packs,\n> > > > which we will implement in a future patch).\n> > >\n> > > Makes sense.\n> > >\n> > > > Unfortunately, the new enumerated type is a bitfield, even though it\n> > > > makes much more sense as '0, 1, 2, ...'. Even though *almost* all\n> > > > options are pairwise exclusive, '--stdin-{packs,commits}' *is*\n> > > > compatible with '--append'. For this reason, use a bitfield.\n> > >\n> > > > -With the `--append` option, include all commits that are present in the\n> > > > -existing commit-graph file.\n> > > > +With the `--input=append` option, include all commits that are present\n> > > > +in the existing commit-graph file.\n> > >\n> > > Would it be too crazy to call this `--input=existing` instead, and have\n> > > it be the same as `--append`? I find that `--append` makes a lot of\n> > > sense (it's a mode we can turn on or off), whereas \"input = append\"\n> > > seems more odd.\n> > \n> > Hmm. When I wrote this, I was thinking of introducing equivalent options\n> > that are identical in name and functionality as '--input=' instead\n> > of '--'. So, I guess that is to say that I didn't spend an awful\n> > amount of time thinking about whether or not '--input=append' made sense\n> > given anything else.\n> > \n> > So, I don't think that '--input=existing' is a bad idea at all, but I do\n> > worry about advertising this deprecation as \"'--' becomes\n> > '--input=', except when your mode is 'append', in which case it\n> > becomes '--input=existing'\".\n> \n> But here you suddenly start using the word \"mode\" both in\n> '--input=' and in '--'.\n> \n> On one hand, I don't think that the word \"mode\" goes as well with\n> \"input\" as \"source\" does.\n> \n> On the other, is '--append' really a source/mode, like '--reachable'\n> and '--stdin-commits' are?\n\nWell, re-reading this question got me confused right after sending it,\nso let me try to rephrase.\n\nIs '--append' really a \"source\", like '--reachable' and\n'--stdin-commits' are? No:\n\n> Source, no: from wordsmithing perspective\n> it doesn't fit with \"source\", and being orthogonal to the \"real\"\n> source options while they are mutually exclusive seems to be a clear\n> indication that it isn't.\n\nOr is it a \"mode\" modifying how other options are handled? Yes:\n\n> Mode, yes: it's a mode of operation where\n> no longer reachable/present commits are not discarded from the\n> commit-graph.\n> \n> So I don't think that adding '--input=append' is a good idea, even if\n> we were call it differently, e.g. '--input=existing' as suggested\n> above.\n> \n> However, I do think that '--input=existing' would better express what\n> '--input=none' in the next patch wants to achieve.\n> \n"},{"id":"391657","messageId":"20200213123129.GQ10482@szeder.dev","threadId":"52725","inReplyTo":"4e85c6f7e40e7d6a8d93574645d65971b7cfa4f8.1581486293.git.me@ttaylorr.com","subject":"Re: [PATCH v3 3/3] builtin/commit-graph.c: support '--input=none'","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-02-13T12:31:29Z","receivedAt":"2020-02-13T12:31:36Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Feb 11, 2020 at 09:47:57PM -0800, Taylor Blau wrote:\n> In the previous commit, we introduced '--split=',\n> and alluded to the fact that '--split=merge-all' would be useful for\n> callers who wish to always trigger a merge of an incremental chain.\n> \n> There is a problem with the above approach, which is that there is no\n> way to specify to the commit-graph builtin that a caller only wants to\n> include commits already in the graph.\n\nI'd like clarification on a detail here. Is it only about not adding\nany new commits, or about keeping all existing commits as well? IOW,\ndo you want to:\n\n - include only commits already existing in the commit-graph, without\n adding any new commits, but remove any commits that do not exist\n in the object database anymore.\n\nor:\n\n - include _all_ commits already existing in the commit-graph, even\n those that don't exist anymore in the object database, without\n adding any new commits.\n\n"},{"id":"391668","messageId":"xmqqzhdmsc5s.fsf@gitster-ct.c.googlers.com","threadId":"52725","inReplyTo":"20200213123129.GQ10482@szeder.dev","subject":"Re: [PATCH v3 3/3] builtin/commit-graph.c: support '--input=none'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-02-13T16:08:15Z","receivedAt":"2020-02-13T16:08:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"SZEDER Gábor writes:\n\n> On Tue, Feb 11, 2020 at 09:47:57PM -0800, Taylor Blau wrote:\n>> In the previous commit, we introduced '--split=',\n>> and alluded to the fact that '--split=merge-all' would be useful for\n>> callers who wish to always trigger a merge of an incremental chain.\n>> \n>> There is a problem with the above approach, which is that there is no\n>> way to specify to the commit-graph builtin that a caller only wants to\n>> include commits already in the graph.\n>\n> I'd like clarification on a detail here. Is it only about not adding\n> any new commits, or about keeping all existing commits as well? IOW,\n> do you want to:\n>\n> - include only commits already existing in the commit-graph, without\n> adding any new commits, but remove any commits that do not exist\n> in the object database anymore.\n>\n> or:\n>\n> - include _all_ commits already existing in the commit-graph, even\n> those that don't exist anymore in the object database, without\n> adding any new commits.\n\nFWIW, I read it as the former, but now you brought it up, it can be\nread either way.\n\nThanks for good review comments, as always.\n"},{"id":"391673","messageId":"20200213174110.GB45518@syl.local","threadId":"52725","inReplyTo":"xmqqy2t7u0rh.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3 0/3] builtin/commit-graph.c: new split/merge options","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-13T17:41:10Z","receivedAt":"2020-02-13T17:41:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi Junio,\n\nOn Wed, Feb 12, 2020 at 10:19:14AM -0800, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > Attached is what I anticipate/hope to be the final reroll of my series\n> > to add new arguments to the 'git commit-graph write --split' flag to\n> > allow callers to force or prohibit the commit-graph machinery to merge\n> > multiple commit-graph layers.\n> >\n> > I was keeping an eye out for more discussion about whether or not these\n> > flags were acceptable by reviewers. Martin Ågren and Derrick Stolee have\n> > both chimed in that they seem OK.\n> >\n> > Since there hasn't been much more discussion in this thread, I replayed\n> > this series on top of 'tb/commit-graph-use-odb' (which was itself\n> > rebased on 'master'). I picked up a couple of ASCIIDoc changes along the\n> > way, and a range-diff is included below.\n>\n> I haven't had a chance to form an opinion on this topic, and will\n> let others comment on it first.\n>\n> This topic and the \"changed paths bloom filter\" topic obviously and\n> inevitably have trivial textual conflicts. When today's integration\n> result is pushed out in 6 hours or so, please see if the resolution\n> is reasonable in 'pu'.\n\nThe resolution looks good to me, thanks.\n\n> Thanks.\n\nThanks,\nTaylor\n"},{"id":"391675","messageId":"20200213175605.GC45518@syl.local","threadId":"52725","inReplyTo":"20200213114800.GP10482@szeder.dev","subject":"Re: [PATCH 2/3] builtin/commit-graph.c: introduce '--input='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-13T17:56:05Z","receivedAt":"2020-02-13T17:56:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Feb 13, 2020 at 12:48:00PM +0100, SZEDER Gábor wrote:\n> On Thu, Feb 13, 2020 at 12:33:13PM +0100, SZEDER Gábor wrote:\n> > On Mon, Feb 03, 2020 at 08:51:24PM -0800, Taylor Blau wrote:\n> > > On Fri, Jan 31, 2020 at 08:34:41PM +0100, Martin Ågren wrote:\n> > > > On Fri, 31 Jan 2020 at 01:30, Taylor Blau wrote:\n> > > > > The 'write' mode of the 'commit-graph' supports input from a number of\n> >\n> > s/mode/subcommand/\n\nSure.\n\n> > > > > different sources:\n> >\n> > I note that you use the word \"sources\" here, in the subject line as\n> > well (as '--input='), and in the code as well (e.g. the in\n> > the error message \"unrecognized --input source, %s\"). I like this\n> > word, I think the words \"input\" and \"source\" go really well together.\n> >\n> > > > > pack indexes over stdin, commits over stdin, commits\n> > > > > reachable from all references, and so on.\n> >\n> > It's interesting to see that you stopped listing and went for \"and so\n> > on\" right when it got interesting/controversial with '--append'... :)\n\nOnly because I figured that I had illustrated the point as \"here are the\ninput sources that we currently understand\", like \"--stdin-packs\",\n\"--stdin-commits\", and so on, not because I find this option to be\ncontroversial.\n\n> > > > > Each of these options are\n> > > > > specified with a unique option: '--stdin-packs', '--stdin-commits', etc.\n> >\n> > It also supports the very inefficient scanning through all objects in\n> > all pack files to find commit objects, which, sadly, ended up being\n> > the default, and thus doesn't have its own --option. Should there be\n> > a corresponding '--input=' as well? (Note that I don't mean\n> > this as a suggestion to add one; on the contrary, the less exposure it\n> > gets the better.)\n\nMaybe... although I (like you, I think) have a hard time imagining that\nthis would ever get used, since it *is* the default source of input, so\nyou could just as easily *not* write '--input' anything and get the same\neffect.\n\n> > > > > Similar to our replacement of 'git config [--]' with 'git config\n> > > > > [--type=]' (c.f., fb0dc3bac1 (builtin/config.c: support\n> > > > > `--type=` as preferred alias for `--`, 2018-04-18)), softly\n> > > > > deprecate '[--]' in favor of '[--input=]'.\n> > > > >\n> > > > > This makes it more clear to implement new options that are combinations\n> > > > > of other options (such as, for example, \"none\", a combination of the old\n> > > > > \"--append\" and a new sentinel to specify to _not_ look in other packs,\n> > > > > which we will implement in a future patch).\n> > > >\n> > > > Makes sense.\n> > > >\n> > > > > Unfortunately, the new enumerated type is a bitfield, even though it\n> > > > > makes much more sense as '0, 1, 2, ...'. Even though *almost* all\n> > > > > options are pairwise exclusive, '--stdin-{packs,commits}' *is*\n> > > > > compatible with '--append'. For this reason, use a bitfield.\n> > > >\n> > > > > -With the `--append` option, include all commits that are present in the\n> > > > > -existing commit-graph file.\n> > > > > +With the `--input=append` option, include all commits that are present\n> > > > > +in the existing commit-graph file.\n> > > >\n> > > > Would it be too crazy to call this `--input=existing` instead, and have\n> > > > it be the same as `--append`? I find that `--append` makes a lot of\n> > > > sense (it's a mode we can turn on or off), whereas \"input = append\"\n> > > > seems more odd.\n\nI don't think that I have a strong preference here, since I don't find\n'--input=append' to be out of the ordinary, so I'd be happy with either.\nIf you'd strongly prefer that we call this '--input=existing', then\nthat's fine with me.\n\nI called it '--input=append' to translate 1-to-1 from '--' to\n'--input='. I would worry a little about saying, \"yeah, if you\nwant to use the new '--input'-style options, just write what you used to\nwrite *unless* that thing is '--append' in which case write 'existing'\ninstead\".\n\nMaybe '--append' was poorly-named to begin with, so I think the\nquestion is really: if we believe that it is poorly named, is now the\nright time to correct that naming?\n\n> > > Hmm. When I wrote this, I was thinking of introducing equivalent options\n> > > that are identical in name and functionality as '--input=' instead\n> > > of '--'. So, I guess that is to say that I didn't spend an awful\n> > > amount of time thinking about whether or not '--input=append' made sense\n> > > given anything else.\n> > >\n> > > So, I don't think that '--input=existing' is a bad idea at all, but I do\n> > > worry about advertising this deprecation as \"'--' becomes\n> > > '--input=', except when your mode is 'append', in which case it\n> > > becomes '--input=existing'\".\n> >\n> > But here you suddenly start using the word \"mode\" both in\n> > '--input=' and in '--'.\n> >\n> > On one hand, I don't think that the word \"mode\" goes as well with\n> > \"input\" as \"source\" does.\n> >\n> > On the other, is '--append' really a source/mode, like '--reachable'\n> > and '--stdin-commits' are?\n>\n> Well, re-reading this question got me confused right after sending it,\n> so let me try to rephrase.\n>\n> Is '--append' really a \"source\", like '--reachable' and\n> '--stdin-commits' are? No:\n>\n> > Source, no: from wordsmithing perspective\n> > it doesn't fit with \"source\", and being orthogonal to the \"real\"\n> > source options while they are mutually exclusive seems to be a clear\n> > indication that it isn't.\n>\n> Or is it a \"mode\" modifying how other options are handled? Yes:\n>\n> > Mode, yes: it's a mode of operation where\n> > no longer reachable/present commits are not discarded from the\n> > commit-graph.\n> >\n> > So I don't think that adding '--input=append' is a good idea, even if\n> > we were call it differently, e.g. '--input=existing' as suggested\n> > above.\n> >\n> > However, I do think that '--input=existing' would better express what\n> > '--input=none' in the next patch wants to achieve.\n\nSo, I guess I'm left wondering what we can do to move forward here. For\nmy part, I see a couple of options:\n\n (a) we could replace every instance of '--input=' with\n '--input=', which seems it would make \"append\" a valid\n pattern-match for \"mode\", and move on.\n\n (b) we could stick with '--input=' and 's/append/existing',\n and move on\n\n (c) or do something else that I didn't think of here and go forward\n with that instead.\n\nLet me know which you'd prefer that I do, and I'd be happy to send an\nupdated version of the series for you to look at.\n\nThanks,\nTaylor\n"},{"id":"391676","messageId":"20200213175654.GD45518@syl.local","threadId":"52725","inReplyTo":"20200213123129.GQ10482@szeder.dev","subject":"Re: [PATCH v3 3/3] builtin/commit-graph.c: support '--input=none'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-13T17:56:54Z","receivedAt":"2020-02-13T17:56:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Feb 13, 2020 at 01:31:29PM +0100, SZEDER Gábor wrote:\n> On Tue, Feb 11, 2020 at 09:47:57PM -0800, Taylor Blau wrote:\n> > In the previous commit, we introduced '--split=',\n> > and alluded to the fact that '--split=merge-all' would be useful for\n> > callers who wish to always trigger a merge of an incremental chain.\n> >\n> > There is a problem with the above approach, which is that there is no\n> > way to specify to the commit-graph builtin that a caller only wants to\n> > include commits already in the graph.\n>\n> I'd like clarification on a detail here. Is it only about not adding\n> any new commits, or about keeping all existing commits as well? IOW,\n> do you want to:\n>\n> - include only commits already existing in the commit-graph, without\n> adding any new commits, but remove any commits that do not exist\n> in the object database anymore.\n\nThis one, since the commit-graph machinery will drop any commits (no\nmatter what input/source/mode you specify) that no longer exist in the\nobject store.\n\n> or:\n>\n> - include _all_ commits already existing in the commit-graph, even\n> those that don't exist anymore in the object database, without\n> adding any new commits.\n\nThanks,\nTaylor\n"},{"id":"391677","messageId":"20200213175842.GE45518@syl.local","threadId":"52725","inReplyTo":"xmqqzhdmsc5s.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3 3/3] builtin/commit-graph.c: support '--input=none'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-02-13T17:58:42Z","receivedAt":"2020-02-13T17:58:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Feb 13, 2020 at 08:08:15AM -0800, Junio C Hamano wrote:\n> SZEDER Gábor writes:\n>\n> > On Tue, Feb 11, 2020 at 09:47:57PM -0800, Taylor Blau wrote:\n> >> In the previous commit, we introduced '--split=',\n> >> and alluded to the fact that '--split=merge-all' would be useful for\n> >> callers who wish to always trigger a merge of an incremental chain.\n> >>\n> >> There is a problem with the above approach, which is that there is no\n> >> way to specify to the commit-graph builtin that a caller only wants to\n> >> include commits already in the graph.\n> >\n> > I'd like clarification on a detail here. Is it only about not adding\n> > any new commits, or about keeping all existing commits as well? IOW,\n> > do you want to:\n> >\n> > - include only commits already existing in the commit-graph, without\n> > adding any new commits, but remove any commits that do not exist\n> > in the object database anymore.\n> >\n> > or:\n> >\n> > - include _all_ commits already existing in the commit-graph, even\n> > those that don't exist anymore in the object database, without\n> > adding any new commits.\n>\n> FWIW, I read it as the former, but now you brought it up, it can be\n> read either way.\n\nIt was intended as the former, but I share both of your feelings that it\ncould be read either way. I amended the commit message to clarify by\nadding:\n\n (and haven't since been deleted from the object store)\n\nas a parenthetical after \"already in the graph...\".\n\n> Thanks for good review comments, as always.\n\nYes, indeed: thank very much for your thoughtful feedback.\n\nThanks,\nTaylor\n"},{"id":"391947","messageId":"CAN0heSrAr0i=JQ5ARqCJzsRo2+L2NFAjBwTEEq-fE=ObT4=Ykw@mail.gmail.com","threadId":"52725","inReplyTo":"cover.1581486293.git.me@ttaylorr.com","subject":"Re: [PATCH v3 0/3] builtin/commit-graph.c: new split/merge options","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-02-17T18:24:26Z","receivedAt":"2020-02-17T18:24:43Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Wed, 12 Feb 2020 at 06:47, Taylor Blau wrote:\n> I picked up a couple of ASCIIDoc changes along the\n> way, and a range-diff is included below.\n\nYup, this fixes the documentation misrendering from the previous round.\n\nMartin\n"}]}