{"thread":{"id":"56018","subject":"[PATCH v2 1/3] p5400: add perf tests for git-receive-pack(1)","startedAt":"2021-06-28T05:33:16Z","lastAt":"2021-08-09T08:12:08Z","messageCount":64,"participants":["Patrick Steinhardt","Ævar Arnfjörð Bjarmason","Taylor Blau","Jeff King","Junio C Hamano"],"isPatch":true,"patchVersion":2,"patchTotal":3},"messages":[{"id":"428588","messageId":"2f6c4e3d102e71104d7d00c78631b149b880609a.1624858240.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1624858240.git.ps@pks.im","subject":"[PATCH v2 1/3] p5400: add perf tests for git-receive-pack(1)","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-06-28T05:33:07Z","receivedAt":"2021-06-28T05:33:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We'll the connectivity check logic for git-receive-pack(1) in the\nfollowing commits to make it perform better. As a preparatory step, add\nsome benchmarks such that we can measure these changes.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/perf/p5400-receive-pack.sh | 97 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 97 insertions(+)\n create mode 100755 t/perf/p5400-receive-pack.sh\n\ndiff --git a/t/perf/p5400-receive-pack.sh b/t/perf/p5400-receive-pack.sh\nnew file mode 100755\nindex 0000000000..a945e014a3\n--- /dev/null\n+++ b/t/perf/p5400-receive-pack.sh\n@@ -0,0 +1,97 @@\n+#!/bin/sh\n+\n+test_description=\"Tests performance of receive-pack\"\n+\n+. ./perf-lib.sh\n+\n+test_perf_large_repo\n+\n+test_expect_success 'setup' '\n+\t# Create a main branch such that we do not have to rely on any specific\n+\t# branch to exist in the perf repository.\n+\tgit switch --force-create main &&\n+\n+\t# Set up a pre-receive hook such that no refs will ever be changed.\n+\t# This easily allows multiple perf runs, but still exercises\n+\t# server-side reference negotiation and checking for consistency.\n+\tmkdir hooks &&\n+\twrite_script hooks/pre-receive <<-EOF &&\n+\t\t#!/bin/sh\n+\t\techo \"failed in pre-receive hook\"\n+\t\texit 1\n+\tEOF\n+\tcat >config <<-EOF &&\n+\t\t[core]\n+\t\t\thooksPath=$(pwd)/hooks\n+\tEOF\n+\tGIT_CONFIG_GLOBAL=\"$(pwd)/config\" &&\n+\texport GIT_CONFIG_GLOBAL &&\n+\n+\tgit switch --create updated &&\n+\ttest_commit --no-tag updated\n+'\n+\n+setup_empty() {\n+\tgit init --bare \"$2\"\n+}\n+\n+setup_clone() {\n+\tgit clone --bare --no-local --branch main \"$1\" \"$2\"\n+}\n+\n+setup_clone_bitmap() {\n+\tgit clone --bare --no-local --branch main \"$1\" \"$2\" &&\n+\tgit -C \"$2\" repack -Adb\n+}\n+\n+# Create a reference for each commit in the target repository with extra-refs.\n+# While this may be an atypical setup, biggish repositories easily end up with\n+# hundreds of thousands of refs, and this is a good enough approximation.\n+setup_extrarefs() {\n+\tgit clone --bare --no-local --branch main \"$1\" \"$2\" &&\n+\tgit -C \"$2\" log --all --format=\"tformat:create refs/commit/%h %H\" |\n+\t\tgit -C \"$2\" update-ref --stdin\n+}\n+\n+# Create a reference for each commit in the target repository with extra-refs.\n+# While this may be an atypical setup, biggish repositories easily end up with\n+# hundreds of thousands of refs, and this is a good enough approximation.\n+setup_extrarefs_bitmap() {\n+\tgit clone --bare --no-local --branch main \"$1\" \"$2\" &&\n+\tgit -C \"$2\" log --all --format=\"tformat:create refs/commit/%h %H\" |\n+\t\tgit -C \"$2\" update-ref --stdin &&\n+\tgit -C \"$2\" repack -Adb\n+}\n+\n+for repo in empty clone clone_bitmap extrarefs extrarefs_bitmap\n+do\n+\ttest_expect_success \"$repo setup\" '\n+\t\trm -rf target.git &&\n+\t\tsetup_$repo \"$(pwd)\" target.git\n+\t'\n+\n+\t# If the target repository is the empty one, then the only thing we can\n+\t# do is to create a new branch.\n+\tcase \"$repo\" in\n+\tempty)\n+\t\trefspecs=\"updated:new\";;\n+\t*)\n+\t\trefspecs=\"updated:new updated:main main~10:main :main\";;\n+\tesac\n+\n+\tfor refspec in $refspecs\n+\tdo\n+\t\ttest_expect_success \"$repo seed $refspec\" \"\n+\t\t\ttest_must_fail git push --force target.git '$refspec' \\\n+\t\t\t\t--receive-pack='tee pack | git receive-pack' 2>err &&\n+\t\t\tgrep 'failed in pre-receive hook' err\n+\t\t\"\n+\n+\t\ttest_perf \"$repo receive-pack $refspec\" \"\n+\t\t\tgit receive-pack target.git negotiation &&\n+\t\t\tgrep 'pre-receive hook declined' negotiation\n+\t\t\"\n+\tdone\n+done\n+\n+test_done\n-- \n2.32.0\n\n"},{"id":"428589","messageId":"cover.1624858240.git.ps@pks.im","threadId":"56018","inReplyTo":null,"subject":"[PATCH v2 0/3] Speed up connectivity checks via bitmaps","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-06-28T05:33:02Z","receivedAt":"2021-06-28T05:33:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is version 2 of my patch series which tries to speed up\nconnectivity checks in git-receive-pack(1).\n\nThis version is a complete rewrite after my initial approach of using\nthe quarantine directory has been shot down due to changes in semantics.\nThis second version is thus an alternative approach using bitmaps. The\nimplementation is quite simple: if we have a bitmap, then we walk the\nnew tips until we hit a bitmapped object. Given that bitmapped objects\nare by definition fully connected, we can then be sure that all pushed\ntips are, too.\n\nI'm not sure I'm happy with results in this series. While it does show a\nspeedup in the area I'm trying to optimize (repos with many refs), there\nare two things which make me hesitant:\n\n - First, there seems to be significant overhead in loading the\n packfile. This is something Peff has already pointed out in \n [1].\n\n - In repos which have out-of-date bitmaps, we're potentially going\n to walk a lot more objects than we used to.\n\nTaking these two issues together, this version is probably more of a\nrequest for comments: do we want to make above tradeoffs? Are there\nbetter alternatives? Any input is welcome.\n\nPatrick\n\nPatrick Steinhardt (3):\n p5400: add perf tests for git-receive-pack(1)\n receive-pack: skip connectivity checks on delete-only commands\n connected: implement connectivity check using bitmaps\n\n builtin/receive-pack.c | 49 ++++++++-----\n connected.c | 136 +++++++++++++++++++++++++++++++++++\n pack-bitmap.c | 4 +-\n pack-bitmap.h | 2 +\n t/perf/p5400-receive-pack.sh | 97 +++++++++++++++++++++++++\n 5 files changed, 267 insertions(+), 21 deletions(-)\n create mode 100755 t/perf/p5400-receive-pack.sh\n\n-- \n2.32.0\n\n"},{"id":"428590","messageId":"b3272f499e51cfc53345f9f09f8762db1a4cf0a6.1624858240.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1624858240.git.ps@pks.im","subject":"[PATCH v2 2/3] receive-pack: skip connectivity checks on delete-only commands","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-06-28T05:33:11Z","receivedAt":"2021-06-28T05:33:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In the case where git-receive-pack(1) receives only commands which\ndelete references, then per technical specification the client MUST NOT\nsend a packfile. As a result, we know that no new objects have been\nreceived, which makes it a moot point to check whether all received\nobjects are fully connected.\n\nFix this by not doing a connectivity check in case there were no pushed\nobjects. Given that git-rev-walk(1) with only negative references will\nnot do any graph walk, no performance improvements are to be expected.\nConceptionally, it is still the right thing to do though.\n\nThe following tests were executed on linux.git and back up above\nexpectation:\n\nTest origin/master HEAD\n---------------------------------------------------------------------------------------------------------\n5400.4: empty receive-pack updated:new 178.36(428.22+164.36) 177.62(421.33+164.48) -0.4%\n5400.7: clone receive-pack updated:new 0.10(0.08+0.02) 0.10(0.08+0.02) +0.0%\n5400.9: clone receive-pack updated:main 0.10(0.08+0.02) 0.11(0.08+0.02) +10.0%\n5400.11: clone receive-pack main~10:main 0.15(0.11+0.04) 0.15(0.10+0.05) +0.0%\n5400.13: clone receive-pack :main 0.01(0.00+0.01) 0.01(0.01+0.00) +0.0%\n5400.16: clone_bitmap receive-pack updated:new 0.10(0.07+0.02) 0.09(0.06+0.02) -10.0%\n5400.18: clone_bitmap receive-pack updated:main 0.10(0.07+0.02) 0.10(0.08+0.02) +0.0%\n5400.20: clone_bitmap receive-pack main~10:main 0.15(0.11+0.03) 0.15(0.12+0.03) +0.0%\n5400.22: clone_bitmap receive-pack :main 0.02(0.01+0.01) 0.01(0.00+0.00) -50.0%\n5400.25: extrarefs receive-pack updated:new 32.34(20.72+11.86) 32.56(20.82+11.95) +0.7%\n5400.27: extrarefs receive-pack updated:main 32.42(21.02+11.61) 32.52(20.64+12.10) +0.3%\n5400.29: extrarefs receive-pack main~10:main 32.53(20.74+12.01) 32.39(20.63+11.97) -0.4%\n5400.31: extrarefs receive-pack :main 7.13(3.53+3.59) 7.15(3.80+3.34) +0.3%\n5400.34: extrarefs_bitmap receive-pack updated:new 32.55(20.72+12.04) 32.65(20.68+12.18) +0.3%\n5400.36: extrarefs_bitmap receive-pack updated:main 32.50(20.90+11.86) 32.67(20.93+11.94) +0.5%\n5400.38: extrarefs_bitmap receive-pack main~10:main 32.43(20.88+11.75) 32.35(20.68+11.89) -0.2%\n5400.40: extrarefs_bitmap receive-pack :main 7.21(3.58+3.63) 7.18(3.61+3.57) -0.4%\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/receive-pack.c | 49 ++++++++++++++++++++++++++----------------\n 1 file changed, 30 insertions(+), 19 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex a34742513a..b9263cec15 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -1918,11 +1918,8 @@ static void execute_commands(struct command *commands,\n \t\t\t struct shallow_info *si,\n \t\t\t const struct string_list *push_options)\n {\n-\tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\n \tstruct command *cmd;\n \tstruct iterate_data data;\n-\tstruct async muxer;\n-\tint err_fd = 0;\n \tint run_proc_receive = 0;\n \n \tif (unpacker_error) {\n@@ -1931,25 +1928,39 @@ static void execute_commands(struct command *commands,\n \t\treturn;\n \t}\n \n-\tif (use_sideband) {\n-\t\tmemset(&muxer, 0, sizeof(muxer));\n-\t\tmuxer.proc = copy_to_sideband;\n-\t\tmuxer.in = -1;\n-\t\tif (!start_async(&muxer))\n-\t\t\terr_fd = muxer.in;\n-\t\t/* ...else, continue without relaying sideband */\n-\t}\n-\n \tdata.cmds = commands;\n \tdata.si = si;\n-\topt.err_fd = err_fd;\n-\topt.progress = err_fd && !quiet;\n-\topt.env = tmp_objdir_env(tmp_objdir);\n-\tif (check_connected(iterate_receive_command_list, &data, &opt))\n-\t\tset_connectivity_errors(commands, si);\n \n-\tif (use_sideband)\n-\t\tfinish_async(&muxer);\n+\t/*\n+\t * If received commands only consist of deletions, then the client MUST\n+\t * NOT send a packfile because there cannot be any new objects in the\n+\t * first place. As a result, we do not set up a quarantine environment\n+\t * because we know no new objects will be received. And that in turn\n+\t * means that we can skip connectivity checks here.\n+\t */\n+\tif (tmp_objdir) {\n+\t\tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\n+\t\tstruct async muxer;\n+\t\tint err_fd = 0;\n+\n+\t\tif (use_sideband) {\n+\t\t\tmemset(&muxer, 0, sizeof(muxer));\n+\t\t\tmuxer.proc = copy_to_sideband;\n+\t\t\tmuxer.in = -1;\n+\t\t\tif (!start_async(&muxer))\n+\t\t\t\terr_fd = muxer.in;\n+\t\t\t/* ...else, continue without relaying sideband */\n+\t\t}\n+\n+\t\topt.err_fd = err_fd;\n+\t\topt.progress = err_fd && !quiet;\n+\t\topt.env = tmp_objdir_env(tmp_objdir);\n+\t\tif (check_connected(iterate_receive_command_list, &data, &opt))\n+\t\t\tset_connectivity_errors(commands, si);\n+\n+\t\tif (use_sideband)\n+\t\t\tfinish_async(&muxer);\n+\t}\n \n \treject_updates_to_hidden(commands);\n \n-- \n2.32.0\n\n"},{"id":"428591","messageId":"7687dedd4722c39b5ecef2c2165147c25d16b8d9.1624858240.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1624858240.git.ps@pks.im","subject":"[PATCH v2 3/3] connected: implement connectivity check using bitmaps","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-06-28T05:33:15Z","receivedAt":"2021-06-28T05:33:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The connectivity checks are currently implemented via git-rev-list(1):\nwe simply ignore any objects which are reachable from preexisting refs\nvia `--not --all`, and pass all new refs which are to be checked via its\nstdin. While this works well enough for repositories which do not have a\nlot of references, it's clear that `--not --all` will linearly scale\nwith the number of refs which do exist: for each reference, we'll walk\nthrough its commit as well as its five parent commits (defined via\n`SLOP`). Given that many major hosting sites which use a pull/merge\nrequest workflow keep refs to the request's HEAD, this effectively means\nthat `--not --all` will do a nearly complete graph walk.\n\nThis commit implements an alternate strategy if the target repository\nhas bitmaps. Objects referenced by a bitmap are by definition always\nfully connected, so they form a natural boundary between old revisions\nand new revisions. With this alternate strategy, we walk all given new\nobject IDs. Whenever we hit any object which is covered by the bitmap,\nwe stop the walk.\n\nThe logic only kicks in in case we have a bitmap in the repository. If\nnot, we wouldn't be able to efficiently abort the walk because we cannot\neasily tell whether an object already exists in the target repository\nand, if it does, whether it's fully connected. As a result, we'd have to\nperform a always do graph walk in this case. Naturally, we could do the\nsame thing the previous git-rev-list(1) implementation did in that case\nand just use preexisting branch tips as boundaries. But for now, we just\nkeep the old implementation for simplicity's sake given that performance\ncharacteristics are likely not significantly different.\n\nAn easier solution may have been to simply add `--use-bitmap-index` to\nthe git-rev-list(1) invocation, but benchmarks have shown that this is\nnot effective: performance characteristics remain the same except for\nsome cases where the bitmap walks performs much worse compared to the\nnon-bitmap walk\n\nThe following benchmarks have been performed in linux.git:\n\nTest origin/master HEAD\n---------------------------------------------------------------------------------------------------------\n5400.4: empty receive-pack updated:new 176.02(387.28+175.12) 176.86(388.75+175.51) +0.5%\n5400.7: clone receive-pack updated:new 0.10(0.09+0.01) 0.08(0.06+0.03) -20.0%\n5400.9: clone receive-pack updated:main 0.09(0.08+0.01) 0.09(0.06+0.03) +0.0%\n5400.11: clone receive-pack main~10:main 0.14(0.11+0.03) 0.13(0.11+0.02) -7.1%\n5400.13: clone receive-pack :main 0.01(0.01+0.00) 0.02(0.01+0.00) +100.0%\n5400.16: clone_bitmap receive-pack updated:new 0.10(0.09+0.01) 0.28(0.13+0.15) +180.0%\n5400.18: clone_bitmap receive-pack updated:main 0.10(0.08+0.02) 0.28(0.12+0.16) +180.0%\n5400.20: clone_bitmap receive-pack main~10:main 0.13(0.11+0.02) 0.26(0.12+0.14) +100.0%\n5400.22: clone_bitmap receive-pack :main 0.01(0.01+0.00) 0.01(0.01+0.00) +0.0%\n5400.25: extrarefs receive-pack updated:new 32.14(20.76+11.59) 32.35(20.52+12.03) +0.7%\n5400.27: extrarefs receive-pack updated:main 32.08(20.54+11.75) 32.10(20.78+11.53) +0.1%\n5400.29: extrarefs receive-pack main~10:main 32.14(20.66+11.68) 32.27(20.65+11.83) +0.4%\n5400.31: extrarefs receive-pack :main 7.09(3.56+3.53) 7.10(3.70+3.40) +0.1%\n5400.34: extrarefs_bitmap receive-pack updated:new 32.41(20.59+12.02) 7.36(3.76+3.60) -77.3%\n5400.36: extrarefs_bitmap receive-pack updated:main 32.26(20.53+11.95) 7.34(3.56+3.78) -77.2%\n5400.38: extrarefs_bitmap receive-pack main~10:main 32.44(20.77+11.90) 7.40(3.70+3.70) -77.2%\n5400.40: extrarefs_bitmap receive-pack :main 7.09(3.62+3.46) 7.17(3.79+3.38) +1.1%\n\nAs expected, performance doesn't change in cases where we do not have a\nbitmap available given that the old code path still kicks in. In case we\ndo have bitmaps, this is kind of a mixed bag: while git-receive-pack(1)\nis slower in a \"normal\" clone of linux.git, it is significantly faster\nfor a clone with lots of references. The slowness can potentially be\nexplained by the overhead of loading the bitmap. On the other hand, the\nnew code is faster as expected in repos which have lots of references\ngiven that we do not have to mark all negative references anymore.\n\nSigned-off-by: Patrick Steinhardt \n---\n connected.c | 136 ++++++++++++++++++++++++++++++++++++++++++++++++++\n pack-bitmap.c | 4 +-\n pack-bitmap.h | 2 +\n 3 files changed, 140 insertions(+), 2 deletions(-)\n\ndiff --git a/connected.c b/connected.c\nindex b18299fdf0..474275a52f 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -6,6 +6,127 @@\n #include \"transport.h\"\n #include \"packfile.h\"\n #include \"promisor-remote.h\"\n+#include \"object.h\"\n+#include \"tree-walk.h\"\n+#include \"commit.h\"\n+#include \"tag.h\"\n+#include \"progress.h\"\n+#include \"oidset.h\"\n+#include \"pack-bitmap.h\"\n+\n+#define QUEUED (1u<<0)\n+\n+static int queue_object(struct repository *repo,\n+\t\t\tstruct bitmap_index *bitmap,\n+\t\t\tstruct object_array *pending,\n+\t\t\tconst struct object_id *oid)\n+{\n+\tstruct object *object;\n+\n+\t/*\n+\t * If the object ID is part of the bitmap, then we know that it must by\n+\t * definition be reachable in the target repository and be fully\n+\t * connected. We can thus skip checking the objects' referenced\n+\t * objects.\n+\t */\n+\tif (bitmap_position(bitmap, oid) >= 0)\n+\t\treturn 0;\n+\n+\t/* Otherwise the object is queued up for a connectivity check. */\n+\tobject = parse_object(repo, oid);\n+\tif (!object) {\n+\t\t/* Promisor objects do not need to be traversed. */\n+\t\tif (is_promisor_object(oid))\n+\t\t\treturn 0;\n+\t\treturn -1;\n+\t}\n+\n+\t/*\n+\t * If the object has been queued before already, then we don't need to\n+\t * do so again.\n+\t */\n+\tif (object->flags & QUEUED)\n+\t\treturn 0;\n+\tobject->flags |= QUEUED;\n+\n+\t/*\n+\t * We do not need to queue up blobs given that they don't reference any\n+\t * other objects anyway.\n+\t */\n+\tif (object->type == OBJ_BLOB)\n+\t\treturn 0;\n+\n+\tadd_object_array(object, NULL, pending);\n+\n+\treturn 0;\n+}\n+\n+static int check_object(\n+\tstruct repository *repo,\n+\tstruct bitmap_index *bitmap,\n+\tstruct object_array *pending,\n+\tconst struct object *object)\n+{\n+\tint ret = 0;\n+\n+\tif (object->type == OBJ_TREE) {\n+\t\tstruct tree *tree = (struct tree *)object;\n+\t\tstruct tree_desc tree_desc;\n+\t\tstruct name_entry entry;\n+\n+\t\tif (init_tree_desc_gently(&tree_desc, tree->buffer, tree->size))\n+\t\t\treturn error(_(\"cannot parse tree entry\"));\n+\t\twhile (tree_entry_gently(&tree_desc, &entry))\n+\t\t\tret |= queue_object(repo, bitmap, pending, &entry.oid);\n+\n+\t\tfree_tree_buffer(tree);\n+\t} else if (object->type == OBJ_COMMIT) {\n+\t\tstruct commit *commit = (struct commit *) object;\n+\t\tstruct commit_list *parents;\n+\n+\t\tfor (parents = commit->parents; parents; parents = parents->next)\n+\t\t\tret |= queue_object(repo, bitmap, pending, &parents->item->object.oid);\n+\n+\t\tfree_commit_buffer(repo->parsed_objects, commit);\n+\t} else if (object->type == OBJ_TAG) {\n+\t\tret |= queue_object(repo, bitmap, pending, get_tagged_oid((struct tag *) object));\n+\t} else {\n+\t\treturn error(_(\"unexpected object type\"));\n+\t}\n+\n+\treturn ret;\n+}\n+\n+/*\n+ * If the target repository has a bitmap, then we treat all objects reachable\n+ * via the bitmap as fully connected. Bitmapped objects thus serve as the\n+ * boundary between old and new objects.\n+ */\n+static int check_connected_with_bitmap(struct repository *repo,\n+\t\t\t\t struct bitmap_index *bitmap,\n+\t\t\t\t oid_iterate_fn fn, void *cb_data,\n+\t\t\t\t struct check_connected_options *opt)\n+{\n+\tstruct object_array pending = OBJECT_ARRAY_INIT;\n+\tstruct progress *progress = NULL;\n+\tsize_t checked_objects = 0;\n+\tstruct object_id oid;\n+\tint ret = 0;\n+\n+\tif (opt->progress)\n+\t\tprogress = start_delayed_progress(\"Checking connectivity\", 0);\n+\n+\twhile (!fn(cb_data, &oid))\n+\t\tret |= queue_object(repo, bitmap, &pending, &oid);\n+\twhile (pending.nr) {\n+\t\tret |= check_object(repo, bitmap, &pending, object_array_pop(&pending));\n+\t\tdisplay_progress(progress, ++checked_objects);\n+\t}\n+\n+\tstop_progress(&progress);\n+\tobject_array_clear(&pending);\n+\treturn ret;\n+}\n \n /*\n * If we feed all the commits we want to verify to this command\n@@ -28,12 +149,27 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \tint err = 0;\n \tstruct packed_git *new_pack = NULL;\n \tstruct transport *transport;\n+\tstruct bitmap_index *bitmap;\n \tsize_t base_len;\n \n \tif (!opt)\n \t\topt = &defaults;\n \ttransport = opt->transport;\n \n+\tbitmap = prepare_bitmap_git(the_repository);\n+\tif (bitmap) {\n+\t\t/*\n+\t\t * If we have a bitmap, then we can reuse the bitmap as\n+\t\t * boundary between old and new objects.\n+\t\t */\n+\t\terr = check_connected_with_bitmap(the_repository, bitmap,\n+\t\t\t\t\t\t fn, cb_data, opt);\n+\t\tif (opt->err_fd)\n+\t\t\tclose(opt->err_fd);\n+\t\tfree_bitmap_index(bitmap);\n+\t\treturn err;\n+\t}\n+\n \tif (fn(cb_data, &oid)) {\n \t\tif (opt->err_fd)\n \t\t\tclose(opt->err_fd);\ndiff --git a/pack-bitmap.c b/pack-bitmap.c\nindex d90e1d9d8c..d88a882ee1 100644\n--- a/pack-bitmap.c\n+++ b/pack-bitmap.c\n@@ -418,8 +418,8 @@ static inline int bitmap_position_packfile(struct bitmap_index *bitmap_git,\n \treturn pos;\n }\n \n-static int bitmap_position(struct bitmap_index *bitmap_git,\n-\t\t\t const struct object_id *oid)\n+int bitmap_position(struct bitmap_index *bitmap_git,\n+\t\t const struct object_id *oid)\n {\n \tint pos = bitmap_position_packfile(bitmap_git, oid);\n \treturn (pos >= 0) ? pos : bitmap_position_extended(bitmap_git, oid);\ndiff --git a/pack-bitmap.h b/pack-bitmap.h\nindex 99d733eb26..7b4b386107 100644\n--- a/pack-bitmap.h\n+++ b/pack-bitmap.h\n@@ -63,6 +63,8 @@ int rebuild_existing_bitmaps(struct bitmap_index *, struct packing_data *mapping\n void free_bitmap_index(struct bitmap_index *);\n int bitmap_walk_contains(struct bitmap_index *,\n \t\t\t struct bitmap *bitmap, const struct object_id *oid);\n+int bitmap_position(struct bitmap_index *bitmap_git,\n+\t\t const struct object_id *oid);\n \n /*\n * After a traversal has been performed by prepare_bitmap_walk(), this can be\n-- \n2.32.0\n\n"},{"id":"428597","messageId":"871r8mxvvx.fsf@evledraar.gmail.com","threadId":"56018","inReplyTo":"2f6c4e3d102e71104d7d00c78631b149b880609a.1624858240.git.ps@pks.im","subject":"Re: [PATCH v2 1/3] p5400: add perf tests for git-receive-pack(1)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-28T07:49:54Z","receivedAt":"2021-06-28T07:59:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jun 28 2021, Patrick Steinhardt wrote:\n\n> [[PGP Signed Part:Undecided]]\n> We'll the connectivity check logic for git-receive-pack(1) in the\n> following commits to make it perform better. As a preparatory step, add\n> some benchmarks such that we can measure these changes.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> t/perf/p5400-receive-pack.sh | 97 ++++++++++++++++++++++++++++++++++++\n> 1 file changed, 97 insertions(+)\n> create mode 100755 t/perf/p5400-receive-pack.sh\n>\n> diff --git a/t/perf/p5400-receive-pack.sh b/t/perf/p5400-receive-pack.sh\n> new file mode 100755\n> index 0000000000..a945e014a3\n> --- /dev/null\n> +++ b/t/perf/p5400-receive-pack.sh\n> @@ -0,0 +1,97 @@\n> +#!/bin/sh\n> +\n> +test_description=\"Tests performance of receive-pack\"\n> +\n> +. ./perf-lib.sh\n> +\n> +test_perf_large_repo\n\nFrom the runtime I think this just needs test_perf_default_repo, no?\nI.e. we should only have *_large_* for cases where git.git is too small\nto produce meaningful results.\n\nPart of th problem is that git.git has become larger over time...\n\n> +test_expect_success 'setup' '\n> +\t# Create a main branch such that we do not have to rely on any specific\n> +\t# branch to exist in the perf repository.\n> +\tgit switch --force-create main &&\n> +\n> +\t# Set up a pre-receive hook such that no refs will ever be changed.\n> +\t# This easily allows multiple perf runs, but still exercises\n> +\t# server-side reference negotiation and checking for consistency.\n> +\tmkdir hooks &&\n> +\twrite_script hooks/pre-receive <<-EOF &&\n> +\t\t#!/bin/sh\n\nYou don't need the #!/bin/sh here, and it won't be used. write_script()\nadds it (or the wanted shell path).\n\n> +\t\techo \"failed in pre-receive hook\"\n> +\t\texit 1\n> +\tEOF\n> +\tcat >config <<-EOF &&\n> +\t\t[core]\n> +\t\t\thooksPath=$(pwd)/hooks\n> +\tEOF\n\nEasier understood IMO as:\n\n git config -f config core.hooksPath ...\n\n> +\tGIT_CONFIG_GLOBAL=\"$(pwd)/config\" &&\n> +\texport GIT_CONFIG_GLOBAL &&\n> +\n> +\tgit switch --create updated &&\n> +\ttest_commit --no-tag updated\n> +'\n> +\n> +setup_empty() {\n> +\tgit init --bare \"$2\"\n> +}\n\nI searched ahead for setup_empty, looked unused, but...\n\n> +setup_clone() {\n> +\tgit clone --bare --no-local --branch main \"$1\" \"$2\"\n> +}\n> +\n> +setup_clone_bitmap() {\n> +\tgit clone --bare --no-local --branch main \"$1\" \"$2\" &&\n> +\tgit -C \"$2\" repack -Adb\n> +}\n> +\n> +# Create a reference for each commit in the target repository with extra-refs.\n> +# While this may be an atypical setup, biggish repositories easily end up with\n> +# hundreds of thousands of refs, and this is a good enough approximation.\n> +setup_extrarefs() {\n> +\tgit clone --bare --no-local --branch main \"$1\" \"$2\" &&\n> +\tgit -C \"$2\" log --all --format=\"tformat:create refs/commit/%h %H\" |\n> +\t\tgit -C \"$2\" update-ref --stdin\n> +}\n> +\n> +# Create a reference for each commit in the target repository with extra-refs.\n> +# While this may be an atypical setup, biggish repositories easily end up with\n> +# hundreds of thousands of refs, and this is a good enough approximation.\n> +setup_extrarefs_bitmap() {\n> +\tgit clone --bare --no-local --branch main \"$1\" \"$2\" &&\n> +\tgit -C \"$2\" log --all --format=\"tformat:create refs/commit/%h %H\" |\n> +\t\tgit -C \"$2\" update-ref --stdin &&\n> +\tgit -C \"$2\" repack -Adb\n> +}\n> +\n> +for repo in empty clone clone_bitmap extrarefs extrarefs_bitmap\n> +do\n> +\ttest_expect_success \"$repo setup\" '\n\n> +\t\trm -rf target.git &&\n> +\t\tsetup_$repo \"$(pwd)\" target.git\n\n...here we use it via interpolation.\n\nI'd find this whole pattern much easier to understand if the setups were\njust a bunch of test_expect_success that created a repo_empty.git,\nrepo_extrarefs.git etc. Then this loop would be:\n\n for repo in repo*.git ...\n\nI'd think that would also give you more meaningful perf data, as now the\nOS will churn between the clone & the subsequent push tests, better to\ndo all the setup, then all the different perf tests.\n\nPerhaps there's also a way to re-use this setup across different runs, I\ndon't know/can't remember if t/perf has a less transient thing than the\nnormal trash directory to use for that.\n"},{"id":"428598","messageId":"87y2auwh1d.fsf@evledraar.gmail.com","threadId":"56018","inReplyTo":"b3272f499e51cfc53345f9f09f8762db1a4cf0a6.1624858240.git.ps@pks.im","subject":"Re: [PATCH v2 2/3] receive-pack: skip connectivity checks on delete-only commands","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-28T08:00:26Z","receivedAt":"2021-06-28T08:05:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jun 28 2021, Patrick Steinhardt wrote:\n\n> [[PGP Signed Part:Undecided]]\n> In the case where git-receive-pack(1) receives only commands which\n> delete references, then per technical specification the client MUST NOT\n> send a packfile. As a result, we know that no new objects have been\n> received, which makes it a moot point to check whether all received\n> objects are fully connected.\n\nIs it just per specification, or do we also have assertions/tests for\nwhat happens in that case?\n\n> [...]\n> The following tests were executed on linux.git and back up above\n> expectation:\n>\n> Test origin/master HEAD\n> ---------------------------------------------------------------------------------------------------------\n> 5400.4: empty receive-pack updated:new 178.36(428.22+164.36) 177.62(421.33+164.48) -0.4%\n> 5400.7: clone receive-pack updated:new 0.10(0.08+0.02) 0.10(0.08+0.02) +0.0%\n> 5400.9: clone receive-pack updated:main 0.10(0.08+0.02) 0.11(0.08+0.02) +10.0%\n> 5400.11: clone receive-pack main~10:main 0.15(0.11+0.04) 0.15(0.10+0.05) +0.0%\n> 5400.13: clone receive-pack :main 0.01(0.00+0.01) 0.01(0.01+0.00) +0.0%\n> 5400.16: clone_bitmap receive-pack updated:new 0.10(0.07+0.02) 0.09(0.06+0.02) -10.0%\n> 5400.18: clone_bitmap receive-pack updated:main 0.10(0.07+0.02) 0.10(0.08+0.02) +0.0%\n> 5400.20: clone_bitmap receive-pack main~10:main 0.15(0.11+0.03) 0.15(0.12+0.03) +0.0%\n> 5400.22: clone_bitmap receive-pack :main 0.02(0.01+0.01) 0.01(0.00+0.00) -50.0%\n> 5400.25: extrarefs receive-pack updated:new 32.34(20.72+11.86) 32.56(20.82+11.95) +0.7%\n> 5400.27: extrarefs receive-pack updated:main 32.42(21.02+11.61) 32.52(20.64+12.10) +0.3%\n> 5400.29: extrarefs receive-pack main~10:main 32.53(20.74+12.01) 32.39(20.63+11.97) -0.4%\n> 5400.31: extrarefs receive-pack :main 7.13(3.53+3.59) 7.15(3.80+3.34) +0.3%\n> 5400.34: extrarefs_bitmap receive-pack updated:new 32.55(20.72+12.04) 32.65(20.68+12.18) +0.3%\n> 5400.36: extrarefs_bitmap receive-pack updated:main 32.50(20.90+11.86) 32.67(20.93+11.94) +0.5%\n> 5400.38: extrarefs_bitmap receive-pack main~10:main 32.43(20.88+11.75) 32.35(20.68+11.89) -0.2%\n> 5400.40: extrarefs_bitmap receive-pack :main 7.21(3.58+3.63) 7.18(3.61+3.57) -0.4%\n\nWe're doing less work so I'd expect to te be faster, but do these tests\nreally back that up? From eyeballing these I can't find a line where the\nconfidence intervals don't overlap, e.g. the +10% regresison is a\n.10->.11 \"regression\" with a [+-] 0.02 (so within the error bars) etc,\nditto for the -50% improvement.\n\nPerhaps the error bars will reduce with a high GIT_PERF_REPEAT_COUNT, or\nthe re-arrangement for keeping things hotter in cache that I suggested\nin 1/3.\n"},{"id":"428599","messageId":"87v95ywgx8.fsf@evledraar.gmail.com","threadId":"56018","inReplyTo":"87y2auwh1d.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 2/3] receive-pack: skip connectivity checks on delete-only commands","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-28T08:06:09Z","receivedAt":"2021-06-28T08:07:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jun 28 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> On Mon, Jun 28 2021, Patrick Steinhardt wrote:\n>\n>> [[PGP Signed Part:Undecided]]\n>> In the case where git-receive-pack(1) receives only commands which\n>> delete references, then per technical specification the client MUST NOT\n>> send a packfile. As a result, we know that no new objects have been\n>> received, which makes it a moot point to check whether all received\n>> objects are fully connected.\n>\n> Is it just per specification, or do we also have assertions/tests for\n> what happens in that case?\n>\n>> [...]\n>> The following tests were executed on linux.git and back up above\n>> expectation:\n>>\n>> Test origin/master HEAD\n>> ---------------------------------------------------------------------------------------------------------\n>> 5400.4: empty receive-pack updated:new 178.36(428.22+164.36) 177.62(421.33+164.48) -0.4%\n>> 5400.7: clone receive-pack updated:new 0.10(0.08+0.02) 0.10(0.08+0.02) +0.0%\n>> 5400.9: clone receive-pack updated:main 0.10(0.08+0.02) 0.11(0.08+0.02) +10.0%\n>> 5400.11: clone receive-pack main~10:main 0.15(0.11+0.04) 0.15(0.10+0.05) +0.0%\n>> 5400.13: clone receive-pack :main 0.01(0.00+0.01) 0.01(0.01+0.00) +0.0%\n>> 5400.16: clone_bitmap receive-pack updated:new 0.10(0.07+0.02) 0.09(0.06+0.02) -10.0%\n>> 5400.18: clone_bitmap receive-pack updated:main 0.10(0.07+0.02) 0.10(0.08+0.02) +0.0%\n>> 5400.20: clone_bitmap receive-pack main~10:main 0.15(0.11+0.03) 0.15(0.12+0.03) +0.0%\n>> 5400.22: clone_bitmap receive-pack :main 0.02(0.01+0.01) 0.01(0.00+0.00) -50.0%\n>> 5400.25: extrarefs receive-pack updated:new 32.34(20.72+11.86) 32.56(20.82+11.95) +0.7%\n>> 5400.27: extrarefs receive-pack updated:main 32.42(21.02+11.61) 32.52(20.64+12.10) +0.3%\n>> 5400.29: extrarefs receive-pack main~10:main 32.53(20.74+12.01) 32.39(20.63+11.97) -0.4%\n>> 5400.31: extrarefs receive-pack :main 7.13(3.53+3.59) 7.15(3.80+3.34) +0.3%\n>> 5400.34: extrarefs_bitmap receive-pack updated:new 32.55(20.72+12.04) 32.65(20.68+12.18) +0.3%\n>> 5400.36: extrarefs_bitmap receive-pack updated:main 32.50(20.90+11.86) 32.67(20.93+11.94) +0.5%\n>> 5400.38: extrarefs_bitmap receive-pack main~10:main 32.43(20.88+11.75) 32.35(20.68+11.89) -0.2%\n>> 5400.40: extrarefs_bitmap receive-pack :main 7.21(3.58+3.63) 7.18(3.61+3.57) -0.4%\n>\n> We're doing less work so I'd expect to te be faster, but do these tests\n> really back that up? From eyeballing these I can't find a line where the\n> confidence intervals don't overlap, e.g. the +10% regresison is a\n> .10->.11 \"regression\" with a [+-] 0.02 (so within the error bars) etc,\n> ditto for the -50% improvement.\n>\n> Perhaps the error bars will reduce with a high GIT_PERF_REPEAT_COUNT, or\n> the re-arrangement for keeping things hotter in cache that I suggested\n> in 1/3.\n\nUrgh, nevermind. Just after I sent this I re-read t/perf/README. I'd\nsomehow recalled that we'd emit error bars from it, but that's\nElapsed(User + System), not Time(.. + errorbar) as I thought, nevermind.\nStill, numbers are such that I wonder if the differences are getting\nlost in some noise...\n"},{"id":"428641","messageId":"YNovuzAsaEb2uIaa@nand.local","threadId":"56018","inReplyTo":"7687dedd4722c39b5ecef2c2165147c25d16b8d9.1624858240.git.ps@pks.im","subject":"Re: [PATCH v2 3/3] connected: implement connectivity check using bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-28T20:23:23Z","receivedAt":"2021-06-28T20:23:29Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jun 28, 2021 at 07:33:15AM +0200, Patrick Steinhardt wrote:\n> As expected, performance doesn't change in cases where we do not have a\n> bitmap available given that the old code path still kicks in. In case we\n> do have bitmaps, this is kind of a mixed bag: while git-receive-pack(1)\n> is slower in a \"normal\" clone of linux.git, it is significantly faster\n> for a clone with lots of references. The slowness can potentially be\n> explained by the overhead of loading the bitmap. On the other hand, the\n> new code is faster as expected in repos which have lots of references\n> given that we do not have to mark all negative references anymore.\n\nI haven't had a chance to look closely at your patches yet, but I like\nthe idea of using an object's presence in the reachability bitmap to\nperform the connectivity checks.\n\nI have wondered how much performance we could eek out by being able to\nload the .bitmap file without having to read each individual bitmap\ncontained in it. (I believe Peff mentioned this elsewhere, but) I would\nbe be interested in something as simple as an optional .bitmap extension\nwhich indicates the list of commits which have a bitmap, and their\noffset within the bitmap.\n\nI'll try this out myself and see if it's worth it. (As an aside, I'll be\noffline next week, so it may take me a little while to post something to\nthe list).\n"},{"id":"428717","messageId":"YNq7MD/fMQp05I21@ncase","threadId":"56018","inReplyTo":"871r8mxvvx.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 1/3] p5400: add perf tests for git-receive-pack(1)","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-06-29T06:18:24Z","receivedAt":"2021-06-29T06:18:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jun 28, 2021 at 09:49:54AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Jun 28 2021, Patrick Steinhardt wrote:\n> \n> > [[PGP Signed Part:Undecided]]\n> > We'll the connectivity check logic for git-receive-pack(1) in the\n> > following commits to make it perform better. As a preparatory step, add\n> > some benchmarks such that we can measure these changes.\n> >\n> > Signed-off-by: Patrick Steinhardt \n> > ---\n> > t/perf/p5400-receive-pack.sh | 97 ++++++++++++++++++++++++++++++++++++\n> > 1 file changed, 97 insertions(+)\n> > create mode 100755 t/perf/p5400-receive-pack.sh\n> >\n> > diff --git a/t/perf/p5400-receive-pack.sh b/t/perf/p5400-receive-pack.sh\n> > new file mode 100755\n> > index 0000000000..a945e014a3\n> > --- /dev/null\n> > +++ b/t/perf/p5400-receive-pack.sh\n> > @@ -0,0 +1,97 @@\n> > +#!/bin/sh\n> > +\n> > +test_description=\"Tests performance of receive-pack\"\n> > +\n> > +. ./perf-lib.sh\n> > +\n> > +test_perf_large_repo\n> \n> From the runtime I think this just needs test_perf_default_repo, no?\n> I.e. we should only have *_large_* for cases where git.git is too small\n> to produce meaningful results.\n> \n> Part of th problem is that git.git has become larger over time...\n\nI did these tests for 3/3 with git.git first, and results were\nsignificantly different. The performance issues I'm trying to fix with\nthe connectivity check really only start to show up with largish\nrepositories.\n\n> > +test_expect_success 'setup' '\n> > +\t# Create a main branch such that we do not have to rely on any specific\n> > +\t# branch to exist in the perf repository.\n> > +\tgit switch --force-create main &&\n> > +\n> > +\t# Set up a pre-receive hook such that no refs will ever be changed.\n> > +\t# This easily allows multiple perf runs, but still exercises\n> > +\t# server-side reference negotiation and checking for consistency.\n> > +\tmkdir hooks &&\n> > +\twrite_script hooks/pre-receive <<-EOF &&\n> > +\t\t#!/bin/sh\n> \n> You don't need the #!/bin/sh here, and it won't be used. write_script()\n> adds it (or the wanted shell path).\n\nMakes sense.\n\n> > +\t\techo \"failed in pre-receive hook\"\n> > +\t\texit 1\n> > +\tEOF\n> > +\tcat >config <<-EOF &&\n> > +\t\t[core]\n> > +\t\t\thooksPath=$(pwd)/hooks\n> > +\tEOF\n> \n> Easier understood IMO as:\n> \n> git config -f config core.hooksPath ...\n\nYup, will change.\n\n> > +\tGIT_CONFIG_GLOBAL=\"$(pwd)/config\" &&\n> > +\texport GIT_CONFIG_GLOBAL &&\n> > +\n> > +\tgit switch --create updated &&\n> > +\ttest_commit --no-tag updated\n> > +'\n> > +\n> > +setup_empty() {\n> > +\tgit init --bare \"$2\"\n> > +}\n> \n> I searched ahead for setup_empty, looked unused, but...\n> \n> > +setup_clone() {\n> > +\tgit clone --bare --no-local --branch main \"$1\" \"$2\"\n> > +}\n> > +\n> > +setup_clone_bitmap() {\n> > +\tgit clone --bare --no-local --branch main \"$1\" \"$2\" &&\n> > +\tgit -C \"$2\" repack -Adb\n> > +}\n> > +\n> > +# Create a reference for each commit in the target repository with extra-refs.\n> > +# While this may be an atypical setup, biggish repositories easily end up with\n> > +# hundreds of thousands of refs, and this is a good enough approximation.\n> > +setup_extrarefs() {\n> > +\tgit clone --bare --no-local --branch main \"$1\" \"$2\" &&\n> > +\tgit -C \"$2\" log --all --format=\"tformat:create refs/commit/%h %H\" |\n> > +\t\tgit -C \"$2\" update-ref --stdin\n> > +}\n> > +\n> > +# Create a reference for each commit in the target repository with extra-refs.\n> > +# While this may be an atypical setup, biggish repositories easily end up with\n> > +# hundreds of thousands of refs, and this is a good enough approximation.\n> > +setup_extrarefs_bitmap() {\n> > +\tgit clone --bare --no-local --branch main \"$1\" \"$2\" &&\n> > +\tgit -C \"$2\" log --all --format=\"tformat:create refs/commit/%h %H\" |\n> > +\t\tgit -C \"$2\" update-ref --stdin &&\n> > +\tgit -C \"$2\" repack -Adb\n> > +}\n> > +\n> > +for repo in empty clone clone_bitmap extrarefs extrarefs_bitmap\n> > +do\n> > +\ttest_expect_success \"$repo setup\" '\n> \n> > +\t\trm -rf target.git &&\n> > +\t\tsetup_$repo \"$(pwd)\" target.git\n> \n> ...here we use it via interpolation.\n> \n> I'd find this whole pattern much easier to understand if the setups were\n> just a bunch of test_expect_success that created a repo_empty.git,\n> repo_extrarefs.git etc. Then this loop would be:\n> \n> for repo in repo*.git ...\n> \n> I'd think that would also give you more meaningful perf data, as now the\n> OS will churn between the clone & the subsequent push tests, better to\n> do all the setup, then all the different perf tests.\n> \n> Perhaps there's also a way to re-use this setup across different runs, I\n> don't know/can't remember if t/perf has a less transient thing than the\n> normal trash directory to use for that.\n\nI originally had code like this, but the issue with first creating all\nthe repos is that it requires lots of disk space with large repos given\nthat we'll clone it once per setup. Combined with the fact that I\noften run tests in tmpfs, this led to out-of-memory situations quite\nfast given that I had 3x6GB repositories plus the seeded packfiles in\nRAM.\n\nThis is why I've changed the setup to do the setup as we go, to bring\ndisk usage down to something sane.\n\nPatrick\n"},{"id":"428719","messageId":"YNq8+Jlsu1UfDKav@ncase","threadId":"56018","inReplyTo":"87y2auwh1d.fsf@evledraar.gmail.com","subject":"Re: [PATCH v2 2/3] receive-pack: skip connectivity checks on delete-only commands","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-06-29T06:26:00Z","receivedAt":"2021-06-29T06:26:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jun 28, 2021 at 10:00:26AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Jun 28 2021, Patrick Steinhardt wrote:\n> \n> > [[PGP Signed Part:Undecided]]\n> > In the case where git-receive-pack(1) receives only commands which\n> > delete references, then per technical specification the client MUST NOT\n> > send a packfile. As a result, we know that no new objects have been\n> > received, which makes it a moot point to check whether all received\n> > objects are fully connected.\n> \n> Is it just per specification, or do we also have assertions/tests for\n> what happens in that case?\n\nI'm not sure whether we have any tests for this, but I've seen several\nhangs already in case the server did expect a packfile or errors in case\nthe client sent one. In any case, the technical specification in\nDocumentation/technical/pack-protocol.txt is quite clear on this:\n\n The packfile MUST NOT be sent if the only command used is 'delete'.\n\n> > [...]\n> > The following tests were executed on linux.git and back up above\n> > expectation:\n> >\n> > Test origin/master HEAD\n> > ---------------------------------------------------------------------------------------------------------\n> > 5400.4: empty receive-pack updated:new 178.36(428.22+164.36) 177.62(421.33+164.48) -0.4%\n> > 5400.7: clone receive-pack updated:new 0.10(0.08+0.02) 0.10(0.08+0.02) +0.0%\n> > 5400.9: clone receive-pack updated:main 0.10(0.08+0.02) 0.11(0.08+0.02) +10.0%\n> > 5400.11: clone receive-pack main~10:main 0.15(0.11+0.04) 0.15(0.10+0.05) +0.0%\n> > 5400.13: clone receive-pack :main 0.01(0.00+0.01) 0.01(0.01+0.00) +0.0%\n> > 5400.16: clone_bitmap receive-pack updated:new 0.10(0.07+0.02) 0.09(0.06+0.02) -10.0%\n> > 5400.18: clone_bitmap receive-pack updated:main 0.10(0.07+0.02) 0.10(0.08+0.02) +0.0%\n> > 5400.20: clone_bitmap receive-pack main~10:main 0.15(0.11+0.03) 0.15(0.12+0.03) +0.0%\n> > 5400.22: clone_bitmap receive-pack :main 0.02(0.01+0.01) 0.01(0.00+0.00) -50.0%\n> > 5400.25: extrarefs receive-pack updated:new 32.34(20.72+11.86) 32.56(20.82+11.95) +0.7%\n> > 5400.27: extrarefs receive-pack updated:main 32.42(21.02+11.61) 32.52(20.64+12.10) +0.3%\n> > 5400.29: extrarefs receive-pack main~10:main 32.53(20.74+12.01) 32.39(20.63+11.97) -0.4%\n> > 5400.31: extrarefs receive-pack :main 7.13(3.53+3.59) 7.15(3.80+3.34) +0.3%\n> > 5400.34: extrarefs_bitmap receive-pack updated:new 32.55(20.72+12.04) 32.65(20.68+12.18) +0.3%\n> > 5400.36: extrarefs_bitmap receive-pack updated:main 32.50(20.90+11.86) 32.67(20.93+11.94) +0.5%\n> > 5400.38: extrarefs_bitmap receive-pack main~10:main 32.43(20.88+11.75) 32.35(20.68+11.89) -0.2%\n> > 5400.40: extrarefs_bitmap receive-pack :main 7.21(3.58+3.63) 7.18(3.61+3.57) -0.4%\n> \n> We're doing less work so I'd expect to te be faster, but do these tests\n> really back that up? From eyeballing these I can't find a line where the\n> confidence intervals don't overlap, e.g. the +10% regresison is a\n> .10->.11 \"regression\" with a [+-] 0.02 (so within the error bars) etc,\n> ditto for the -50% improvement.\n> \n> Perhaps the error bars will reduce with a high GIT_PERF_REPEAT_COUNT, or\n> the re-arrangement for keeping things hotter in cache that I suggested\n> in 1/3.\n\nAs I've layed out in the commit message, all we save now is spawning\ngit-rev-list(1). The command list iterator which is used to feed data\ninto git-rev-list(1) wouldn't provide any references given that it\nskips over all queued updates whose new OID is the null OID. So\ngit-rev-list(1) doesn't receive any input except `--not --all` and thus\ncan exit without doing a graph walk.\n\nAbove numbers simply show that this saving is not significant and gets\nlost in the noise, at least on Linux. Windows may show slightly\ndifferent numbers given that spawning of processes is slower there, but\nI don't expect it to matter much there, either.\n\nPatrick\n"},{"id":"428742","messageId":"87r1gkg8rf.fsf@evledraar.gmail.com","threadId":"56018","inReplyTo":"YNq7MD/fMQp05I21@ncase","subject":"Re: [PATCH v2 1/3] p5400: add perf tests for git-receive-pack(1)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-06-29T12:09:25Z","receivedAt":"2021-06-29T12:23:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jun 29 2021, Patrick Steinhardt wrote:\n\n> [[PGP Signed Part:Undecided]]\n> On Mon, Jun 28, 2021 at 09:49:54AM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> On Mon, Jun 28 2021, Patrick Steinhardt wrote:\n>> \n>> > [[PGP Signed Part:Undecided]]\n>> > We'll the connectivity check logic for git-receive-pack(1) in the\n>> > following commits to make it perform better. As a preparatory step, add\n>> > some benchmarks such that we can measure these changes.\n>> >\n>> > Signed-off-by: Patrick Steinhardt \n>> > ---\n>> > t/perf/p5400-receive-pack.sh | 97 ++++++++++++++++++++++++++++++++++++\n>> > 1 file changed, 97 insertions(+)\n>> > create mode 100755 t/perf/p5400-receive-pack.sh\n>> >\n>> > diff --git a/t/perf/p5400-receive-pack.sh b/t/perf/p5400-receive-pack.sh\n>> > new file mode 100755\n>> > index 0000000000..a945e014a3\n>> > --- /dev/null\n>> > +++ b/t/perf/p5400-receive-pack.sh\n>> > @@ -0,0 +1,97 @@\n>> > +#!/bin/sh\n>> > +\n>> > +test_description=\"Tests performance of receive-pack\"\n>> > +\n>> > +. ./perf-lib.sh\n>> > +\n>> > +test_perf_large_repo\n>> \n>> From the runtime I think this just needs test_perf_default_repo, no?\n>> I.e. we should only have *_large_* for cases where git.git is too small\n>> to produce meaningful results.\n>> \n>> Part of th problem is that git.git has become larger over time...\n>\n> I did these tests for 3/3 with git.git first, and results were\n> significantly different. The performance issues I'm trying to fix with\n> the connectivity check really only start to show up with largish\n> repositories.\n>\n>> > +test_expect_success 'setup' '\n>> > +\t# Create a main branch such that we do not have to rely on any specific\n>> > +\t# branch to exist in the perf repository.\n>> > +\tgit switch --force-create main &&\n>> > +\n>> > +\t# Set up a pre-receive hook such that no refs will ever be changed.\n>> > +\t# This easily allows multiple perf runs, but still exercises\n>> > +\t# server-side reference negotiation and checking for consistency.\n>> > +\tmkdir hooks &&\n>> > +\twrite_script hooks/pre-receive <<-EOF &&\n>> > +\t\t#!/bin/sh\n>> \n>> You don't need the #!/bin/sh here, and it won't be used. write_script()\n>> adds it (or the wanted shell path).\n>\n> Makes sense.\n>\n>> > +\t\techo \"failed in pre-receive hook\"\n>> > +\t\texit 1\n>> > +\tEOF\n>> > +\tcat >config <<-EOF &&\n>> > +\t\t[core]\n>> > +\t\t\thooksPath=$(pwd)/hooks\n>> > +\tEOF\n>> \n>> Easier understood IMO as:\n>> \n>> git config -f config core.hooksPath ...\n>\n> Yup, will change.\n>\n>> > +\tGIT_CONFIG_GLOBAL=\"$(pwd)/config\" &&\n>> > +\texport GIT_CONFIG_GLOBAL &&\n>> > +\n>> > +\tgit switch --create updated &&\n>> > +\ttest_commit --no-tag updated\n>> > +'\n>> > +\n>> > +setup_empty() {\n>> > +\tgit init --bare \"$2\"\n>> > +}\n>> \n>> I searched ahead for setup_empty, looked unused, but...\n>> \n>> > +setup_clone() {\n>> > +\tgit clone --bare --no-local --branch main \"$1\" \"$2\"\n>> > +}\n>> > +\n>> > +setup_clone_bitmap() {\n>> > +\tgit clone --bare --no-local --branch main \"$1\" \"$2\" &&\n>> > +\tgit -C \"$2\" repack -Adb\n>> > +}\n>> > +\n>> > +# Create a reference for each commit in the target repository with extra-refs.\n>> > +# While this may be an atypical setup, biggish repositories easily end up with\n>> > +# hundreds of thousands of refs, and this is a good enough approximation.\n>> > +setup_extrarefs() {\n>> > +\tgit clone --bare --no-local --branch main \"$1\" \"$2\" &&\n>> > +\tgit -C \"$2\" log --all --format=\"tformat:create refs/commit/%h %H\" |\n>> > +\t\tgit -C \"$2\" update-ref --stdin\n>> > +}\n>> > +\n>> > +# Create a reference for each commit in the target repository with extra-refs.\n>> > +# While this may be an atypical setup, biggish repositories easily end up with\n>> > +# hundreds of thousands of refs, and this is a good enough approximation.\n>> > +setup_extrarefs_bitmap() {\n>> > +\tgit clone --bare --no-local --branch main \"$1\" \"$2\" &&\n>> > +\tgit -C \"$2\" log --all --format=\"tformat:create refs/commit/%h %H\" |\n>> > +\t\tgit -C \"$2\" update-ref --stdin &&\n>> > +\tgit -C \"$2\" repack -Adb\n>> > +}\n>> > +\n>> > +for repo in empty clone clone_bitmap extrarefs extrarefs_bitmap\n>> > +do\n>> > +\ttest_expect_success \"$repo setup\" '\n>> \n>> > +\t\trm -rf target.git &&\n>> > +\t\tsetup_$repo \"$(pwd)\" target.git\n>> \n>> ...here we use it via interpolation.\n>> \n>> I'd find this whole pattern much easier to understand if the setups were\n>> just a bunch of test_expect_success that created a repo_empty.git,\n>> repo_extrarefs.git etc. Then this loop would be:\n>> \n>> for repo in repo*.git ...\n>> \n>> I'd think that would also give you more meaningful perf data, as now the\n>> OS will churn between the clone & the subsequent push tests, better to\n>> do all the setup, then all the different perf tests.\n>> \n>> Perhaps there's also a way to re-use this setup across different runs, I\n>> don't know/can't remember if t/perf has a less transient thing than the\n>> normal trash directory to use for that.\n>\n> I originally had code like this, but the issue with first creating all\n> the repos is that it requires lots of disk space with large repos given\n> that we'll clone it once per setup. Combined with the fact that I\n> often run tests in tmpfs, this led to out-of-memory situations quite\n> fast given that I had 3x6GB repositories plus the seeded packfiles in\n> RAM.\n>\n> This is why I've changed the setup to do the setup as we go, to bring\n> disk usage down to something sane.\n>\n> Patrick\n>\n> [[End of PGP Signed Part]]\n\nAh, I see. In that case wouldn't it be even better/faster with/without\nmy suggestion to not use \"clone\" here, which would either be manually\nset up with alternates, or removing the --no-local flag.\n\nYou'd then share bulk of the object database, and just have different\nreferences. B.t.w. you'll probably get less noise/more relevant results\nif you then do a \"pack-refs\" after creating those N references.\n\nSo you should have:\n\n 0. Your \"big\" test repop (not used directly)\n 1. An empty repo\n 2. The \"big\" test repo itself, but just the HEAD branch, using\n alternates to point to #0\n 3. Ditto, but we create a crapload of refs for each commit for a\n version of #2.\n 4. Ditto #3 (could even \"cp\" over the packed refs file to save time),\n but add a bitmap on top.\n\nWell, presumably for #4 we'd actually want to do the \"git repack -Adb\"\nfor #2 (or enforce that #0 must have it), then just move the *.bitmap\nfile(s) to #4. Now the test case conflates whether we have bitmaps with\nhow well (re)packed something is.\n\nI think this might also allow you to get rid of the pre-receive hook for\na \"real\" push test, since the side-repos would be so cheap at this point\nthat you could perhaps setup N of them to push into.\n"},{"id":"428787","messageId":"YNuiM8TR5evSeNsN@nand.local","threadId":"56018","inReplyTo":"YNovuzAsaEb2uIaa@nand.local","subject":"Re: [PATCH v2 3/3] connected: implement connectivity check using bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-29T22:44:03Z","receivedAt":"2021-06-29T22:44:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Jun 28, 2021 at 04:23:23PM -0400, Taylor Blau wrote:\n> I'll try this out myself and see if it's worth it. (As an aside, I'll be\n> offline next week, so it may take me a little while to post something to\n> the list).\n\nI gave implementing this a shot and it seems to have produced some good\nimprovements, although there are definitely some areas where it does\nbetter than others.\n\nHere are some results running on linux.git with a cold cache, counting\nobjects for commit 2ab38c17aa, which I picked deliberately since I know\nit has a bitmap:\n\n $ hyperfine \\\n 'GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2' \\\n 'GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2' \\\n --prepare='sync; echo 3 | sudo tee /proc/sys/vm/drop_caches'\n\n\t\tBenchmark #1: GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2\n\t\t\tTime (mean ± σ): 141.1 ms ± 2.5 ms [User: 13.0 ms, System: 64.3 ms]\n\t\t\tRange (min … max): 136.2 ms … 143.4 ms 10 runs\n\n\t\tBenchmark #2: GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2\n\t\t\tTime (mean ± σ): 28.7 ms ± 3.2 ms [User: 6.5 ms, System: 10.0 ms]\n\t\t\tRange (min … max): 22.0 ms … 31.0 ms 21 runs\n\n\t\tSummary\n\t\t\t'GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2' ran\n\t\t\t\t4.91 ± 0.55 times faster than 'GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2'\n\nThat's sort of a best-case scenario, because we're not doing any\ntraversal between the bitmapped commits and the traversal tips. But even\nif we do have some traversal, the results are still pretty good.\nSwapping out 2ab38c17aa for `--branches` yields a 5.02x improvement from\n141.0ms down to 28.1ms.\n\nAdding in `--tags` basically negates any improvement (having the commit\ntable extension eeks out a 1.03x improvement from 645.7ms down to\n626.0ms. `perf record` shows that 30% of time is spent outside of the\nbitmap code.\n\nIf you want to give this a try yourself, I highly recommend generating\nyour bitmap while packing with `-c pack.writeReverseIndex`. Building a\nreverse index on-the-fly also seems to negate any performance\nimprovements here, so having an on-disk reverse index is more or less a\nprerequisite to testing this out.\n\nExtremely gross and inscrutable code can be found on the\n'tb/bitmap-commit-table' branch of my fork [1].\n\nThanks,\nTaylor\n\n[1]: https://github.com/git/git/compare/master...ttaylorr:tb/bitmap-commit-table\n\n"},{"id":"428792","messageId":"YNvJj42ALg4/t+0l@coredump.intra.peff.net","threadId":"56018","inReplyTo":"b3272f499e51cfc53345f9f09f8762db1a4cf0a6.1624858240.git.ps@pks.im","subject":"Re: [PATCH v2 2/3] receive-pack: skip connectivity checks on delete-only commands","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-06-30T01:31:59Z","receivedAt":"2021-06-30T01:32:02Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 28, 2021 at 07:33:11AM +0200, Patrick Steinhardt wrote:\n\n> Fix this by not doing a connectivity check in case there were no pushed\n> objects. Given that git-rev-walk(1) with only negative references will\n> not do any graph walk, no performance improvements are to be expected.\n> Conceptionally, it is still the right thing to do though.\n\nEven though it's not producing any exciting results, I agree this is\nstill a reasonable thing to do.\n\nI'm actually surprised it didn't help in your many-ref cases, just\nbecause I think the traversal machinery is pretty eager to parse tags\nand commits which are fed as tips.\n\nIf I run \"git rev-list --not --all --stdin +\t/*\n> +\t * If received commands only consist of deletions, then the client MUST\n> +\t * NOT send a packfile because there cannot be any new objects in the\n> +\t * first place. As a result, we do not set up a quarantine environment\n> +\t * because we know no new objects will be received. And that in turn\n> +\t * means that we can skip connectivity checks here.\n> +\t */\n> +\tif (tmp_objdir) {\n\nI think this will work, but we're now making assumptions about how\ntmp_objdir will be initialized by the rest of the code.\n\nCould we make a more direct check of: skip calling rev-list if we have\nno positive tips to feed? If we call iterate_receive_command() and it\nreturns end-of-list on the first call, then we know there is nothing to\nfeed (and as a bonus, this catches some more noop cases around shallow\nrepos; see iterate_receive_command).\n\nBut that made me think that check_connected() could be doing this\nitself. And indeed, it seems to already. At the top of that function is:\n\n if (fn(cb_data, &oid)) {\n if (opt->err_fd)\n close(opt->err_fd);\n return err;\n }\n\nSo before we even spawn rev-list, if there's nothing to feed it, we'll\nreturn right away (\"err\" will be \"0\" at this point). And I think that\nhas been there since the beginning of the function (it is even in the\nold versions in builtin/fetch.c before it was factored out).\n\nAnd that explains why you didn't see any performance improvement. We're\nalready doing this optimization. :)\n\n-Peff\n"},{"id":"428793","messageId":"YNvKbtE+EWKvf2Qe@coredump.intra.peff.net","threadId":"56018","inReplyTo":"YNvJj42ALg4/t+0l@coredump.intra.peff.net","subject":"Re: [PATCH v2 2/3] receive-pack: skip connectivity checks on delete-only commands","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-06-30T01:35:42Z","receivedAt":"2021-06-30T01:35:45Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 29, 2021 at 09:31:59PM -0400, Jeff King wrote:\n\n> If I run \"git rev-list --not --all --stdin takes about 35ms. But if I make a ton of refs, like:\n\nThe \"--stdin\" should be before \"--not\", of course, to mimic a real\nconnectivity check. But it doesn't matter in this case because we're\nfeeding an empty input. :)\n\n-Peff\n"},{"id":"428794","messageId":"YNvOJQDYRWpFa+4S@coredump.intra.peff.net","threadId":"56018","inReplyTo":"7687dedd4722c39b5ecef2c2165147c25d16b8d9.1624858240.git.ps@pks.im","subject":"Re: [PATCH v2 3/3] connected: implement connectivity check using bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-06-30T01:51:33Z","receivedAt":"2021-06-30T01:51:36Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 28, 2021 at 07:33:15AM +0200, Patrick Steinhardt wrote:\n\n> As expected, performance doesn't change in cases where we do not have a\n> bitmap available given that the old code path still kicks in. In case we\n> do have bitmaps, this is kind of a mixed bag: while git-receive-pack(1)\n> is slower in a \"normal\" clone of linux.git, it is significantly faster\n> for a clone with lots of references. The slowness can potentially be\n> explained by the overhead of loading the bitmap. On the other hand, the\n> new code is faster as expected in repos which have lots of references\n> given that we do not have to mark all negative references anymore.\n\nHmm. We _do_ still have to mark those negative references now, though\n(the bitmap code still considers each as a reachability tip for the\n\"have\" side of the traversal). It's just that we may have to do less\ntraversal on them, if they're mentioned by other bitmaps.\n\nSo in that sense I don't think your \"a ref for every commit\" cases are\nall that interesting. Any bitmap near the tip of history is going to\ninclude a bit for all those old commits, because our fake set of refs\nare all reachable. A much more interesting history is when you have a\nbunch of little unreachable spikes coming off the main history.\n\nThis is common if you have a lot of branches in the repo, but also if\nyou maintain a lot of book-keeping refs (like the refs/pull/* we do at\nGitHub; I assume GitLab does something similar).\n\nHere are some real-world numbers from one of the repos that gives us\nfrequent problems with bitmaps. refs/pull/9937/head in this case is an\nunmerged PR with 8 commits on it.\n\n [without bitmaps, full check but with count to suppress output]\n $ time git rev-list --count refs/pull/9937/head --objects --not --all\n 0\n real\t0m1.280s\n user\t0m1.131s\n sys\t0m0.148s\n\n [same, but with bitmaps]\n $ time git rev-list --count refs/pull/9937/head --objects --not --all --use-bitmap-index\n 0\n \n real\t1m38.146s\n user\t1m30.015s\n sys\t0m3.443s\n\nYikes. Now this is a pretty extreme case, as it has a lot of bookkeeping\nrefs. If we limited ourselves to just the branches (in which case our\nunmerged PR will appear to have a couple new commits), though, we still\nget:\n\n $ time git rev-list --count refs/pull/9937/head --objects --not --branches\n 64\n real\t0m0.366s\n user\t0m0.272s\n sys\t0m0.093s\n\n $ time git rev-list --count refs/pull/9937/head --objects --not --branches --use-bitmap-index\n 61\n real\t0m10.372s\n user\t0m9.633s\n sys\t0m0.736s\n\nwhich is still a pretty bad regression (the difference in the output is\nexpected; the regular traversal is not as thorough at finding objects\nwhich appear in non-contiguous sections of history).\n\nAgain, this is one of the repositories that routinely gives us problems.\nBut even on git/git, which is usually not a problematic repo, I get:\n\n $ time git rev-list refs/pull/986/head --objects --not --all\n real\t0m0.121s\n user\t0m0.024s\n sys\t0m0.097s\n\n $ time git rev-list refs/pull/986/head --objects --not --all --use-bitmap-index\n real\t0m12.406s\n user\t0m5.843s\n sys\t0m0.734s\n\nSo even if this tradeoff might help on balance, it really makes some\ncases pathologically bad.\n\n-Peff\n"},{"id":"428796","messageId":"YNvRSJI1oWN9W25Y@coredump.intra.peff.net","threadId":"56018","inReplyTo":"YNuiM8TR5evSeNsN@nand.local","subject":"Re: [PATCH v2 3/3] connected: implement connectivity check using bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-06-30T02:04:56Z","receivedAt":"2021-06-30T02:04:59Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 29, 2021 at 06:44:03PM -0400, Taylor Blau wrote:\n\n> $ hyperfine \\\n> 'GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2' \\\n> 'GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2' \\\n> --prepare='sync; echo 3 | sudo tee /proc/sys/vm/drop_caches'\n> \n> \t\tBenchmark #1: GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2\n> \t\t\tTime (mean ± σ): 141.1 ms ± 2.5 ms [User: 13.0 ms, System: 64.3 ms]\n> \t\t\tRange (min … max): 136.2 ms … 143.4 ms 10 runs\n> \n> \t\tBenchmark #2: GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2\n> \t\t\tTime (mean ± σ): 28.7 ms ± 3.2 ms [User: 6.5 ms, System: 10.0 ms]\n> \t\t\tRange (min … max): 22.0 ms … 31.0 ms 21 runs\n> \n> \t\tSummary\n> \t\t\t'GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2' ran\n> \t\t\t\t4.91 ± 0.55 times faster than 'GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2'\n\nI was curious why your machine is so much slower than mine. With the\ncurrent bitmap format, I can run that command pretty consistently in\n~22ms. But I think the trick here is the cache-dropping. The cold-cache\nperformance is going to be very dependent on faulting in the extra\nbytes (and you can see that the actual CPU time in the first case is\nmuch smaller than the runtime, so it really is waiting on the disk).\n\nIn the warm-cache case, the improvement seems to go away (or maybe I'm\nholding it wrong; even in the cold-cache case I don't get anywhere near\nas impressive a speedup as you showed above). Which isn't to say that\ncold-cache isn't sometimes important, and this may still be worth doing.\nBut it really seems like the CPU involve in walking over the file isn't\nactually that much.\n\nI got an even more curious result when adding in \"--not --all\" (which\nthe connectivity check under discussion would do). There the improvement\nfrom your patch should be even less, because we'd end up reading most of\nthe bitmaps anyway. But I got:\n\n\n $ hyperfine \\\n 'GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2 --not --all' \\\n 'GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2 --not --all' \\\n --prepare='sync; echo 3 | sudo tee /proc/sys/vm/drop_caches'\n\n Benchmark #1: GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2 --not --all\n Time (mean ± σ): 4.197 s ± 0.823 s [User: 284.7 ms, System: 734.5 ms]\n Range (min … max): 2.612 s … 5.009 s 10 runs\n \n Benchmark #2: GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2 --not --all\n Time (mean ± σ): 4.498 s ± 0.612 s [User: 315.3 ms, System: 829.7 ms]\n Range (min … max): 3.010 s … 5.072 s 10 runs\n \n Summary\n 'GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2 --not --all' ran\n 1.07 ± 0.26 times faster than 'GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2 --not --all'\n\nwhich was actually faster _without_ the extra table. 7% isn't a lot,\nespecially for a cold-cache test, so that may just be noise. But it\ndoesn't seem like a clear win to me.\n\n-Peff\n"},{"id":"428802","messageId":"YNvgA6RLIMdD77Hk@nand.local","threadId":"56018","inReplyTo":"YNvRSJI1oWN9W25Y@coredump.intra.peff.net","subject":"Re: [PATCH v2 3/3] connected: implement connectivity check using bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-06-30T03:07:47Z","receivedAt":"2021-06-30T03:07:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 29, 2021 at 10:04:56PM -0400, Jeff King wrote:\n> In the warm-cache case, the improvement seems to go away (or maybe I'm\n> holding it wrong; even in the cold-cache case I don't get anywhere near\n> as impressive a speedup as you showed above). Which isn't to say that\n> cold-cache isn't sometimes important, and this may still be worth doing.\n> But it really seems like the CPU involve in walking over the file isn't\n> actually that much.\n\nHmm. I think that you might be holding it wrong, or at least I'm able to\nreproduce some substantial improvements in the warm cache case with\nlimited traversals. Here are a few runs of the same hyperfine\ninvocation, just swapping the `--prepare` which drops the disk cache\nwith `--warmup 3` which populates them.\n\n $ hyperfine \\\n 'GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2' \\\n 'GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2' \\\n --warmup 3\n Benchmark #1: GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2\n Time (mean ± σ): 23.1 ms ± 6.4 ms [User: 9.4 ms, System: 13.6 ms]\n Range (min … max): 13.8 ms … 35.8 ms 161 runs\n\n Benchmark #2: GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index 2ab38c17aac10bf55ab3efde4c4db3893d8691d2\n Time (mean ± σ): 11.2 ms ± 1.8 ms [User: 7.5 ms, System: 3.7 ms]\n Range (min … max): 4.7 ms … 12.6 ms 238 runs\n\nSwapping just loading an individual commit to look at all branches, I\nget the following 2.01x improvement:\n\n Benchmark #1: GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index --branches\n Time (mean ± σ): 22.5 ms ± 5.8 ms [User: 8.5 ms, System: 14.0 ms]\n Range (min … max): 14.1 ms … 34.9 ms 157 runs\n\n Benchmark #2: GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index --branches\n Time (mean ± σ): 11.2 ms ± 1.9 ms [User: 7.1 ms, System: 4.1 ms]\n Range (min … max): 4.7 ms … 13.4 ms 239 runs\n\nBut there are some diminishing returns when I include --tags, too, since\nI assume that there is some more traversal involved in older parts of\nthe kernel's history which aren't as well covered by bitmaps. But it's\nstill an improvement of 1.17x (give or take .31x, according to\nhyperfine).\n\n Benchmark #1: GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index --branches --tags\n Time (mean ± σ): 66.8 ms ± 12.4 ms [User: 43.6 ms, System: 23.1 ms]\n Range (min … max): 54.4 ms … 92.3 ms 39 runs\n\n Benchmark #2: GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index --branches --tags\n Time (mean ± σ): 57.3 ms ± 10.9 ms [User: 37.5 ms, System: 19.8 ms]\n Range (min … max): 44.0 ms … 79.5 ms 45 runs\n\n\n> I got an even more curious result when adding in \"--not --all\" (which\n> the connectivity check under discussion would do). There the improvement\n> from your patch should be even less, because we'd end up reading most of\n> the bitmaps anyway. But I got:\n\nInteresting. Discussion about that series aside, I go from 28.6ms\nwithout reading the table to 35.1ms reading it, which is much better in\nabsolute timings, but much worse relatively speaking.\n\nI can't quite seem to make sense of the perf report on that command.\nMost of the time is spent faulting pages in, but most of the time\nmeasured in the \"git\" object is in ubc_check. I don't really know how to\ninterpret that, but I'd be curious if you had any thoughts.\n\nI was just looking at:\n\n $ GIT_READ_COMMIT_TABLE=1 perf record -F997 -g \\\n git.compile rev-list --count --objects \\\n --use-bitmap-index 2ab38c17aac --not --all\n $ perf report\n\nThanks,\nTaylor\n"},{"id":"428804","messageId":"YNwE3wES3iv+Xynp@coredump.intra.peff.net","threadId":"56018","inReplyTo":"YNvgA6RLIMdD77Hk@nand.local","subject":"Re: [PATCH v2 3/3] connected: implement connectivity check using bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-06-30T05:45:03Z","receivedAt":"2021-06-30T05:45:24Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 29, 2021 at 11:07:47PM -0400, Taylor Blau wrote:\n\n> On Tue, Jun 29, 2021 at 10:04:56PM -0400, Jeff King wrote:\n> > In the warm-cache case, the improvement seems to go away (or maybe I'm\n> > holding it wrong; even in the cold-cache case I don't get anywhere near\n> > as impressive a speedup as you showed above). Which isn't to say that\n> > cold-cache isn't sometimes important, and this may still be worth doing.\n> > But it really seems like the CPU involve in walking over the file isn't\n> > actually that much.\n> \n> Hmm. I think that you might be holding it wrong, or at least I'm able to\n> reproduce some substantial improvements in the warm cache case with\n> limited traversals.\n\nOK, I definitely was holding it wrong. It turns out that it helps to run\nthe custom version of Git when passing in the pack.writebitmapcomittable\noption. (I regret there is no portable way to communicate a facepalm\nimage over plain text).\n\nSo that helped, but I did seem some other interesting bits.\n\nHere's my cold-cache time for the commit you selected:\n\n $ export commit=2ab38c17aac10bf55ab3efde4c4db3893d8691d2\n $ hyperfine \\\n -L table 0,1 \\\n 'GIT_READ_COMMIT_TABLE={table} git.compile rev-list --count --objects --use-bitmap-index $commit' \\\n --prepare='sync; echo 3 | sudo tee /proc/sys/vm/drop_caches'\n\n Benchmark #1: GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index $commit\n Time (mean ± σ): 1.420 s ± 0.162 s [User: 196.1 ms, System: 293.7 ms]\n Range (min … max): 1.083 s … 1.540 s 10 runs\n \n Benchmark #2: GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index $commit\n Time (mean ± σ): 1.319 s ± 0.256 s [User: 171.1 ms, System: 237.1 ms]\n Range (min … max): 0.773 s … 1.588 s 10 runs\n \n Summary\n 'GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index $commit' ran\n 1.08 ± 0.24 times faster than 'GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index $commit'\n\nSo better, but well within the noise (and I had a few runs where it\nactually performed worse). But you picked that commit because you knew\nit was bitmapped, and it's not in my repo. If I switch to a commit that\nis covered in my repo, then I get similar results to yours:\n\n $ export commit=9b1ea29bc0d7b94d420f96a0f4121403efc3dd85\n $ hyperfine \\\n -L table 0,1 \\\n 'GIT_READ_COMMIT_TABLE={table} git.compile rev-list --count --objects --use-bitmap-index $commit' \\\n --prepare='sync; echo 3 | sudo tee /proc/sys/vm/drop_caches'\n\n Benchmark #1: GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index $commit\n Time (mean ± σ): 309.3 ms ± 61.0 ms [User: 19.4 ms, System: 79.0 ms]\n Range (min … max): 154.4 ms … 386.7 ms 10 runs\n \n Benchmark #2: GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index $commit\n Time (mean ± σ): 33.7 ms ± 2.5 ms [User: 3.3 ms, System: 3.6 ms]\n Range (min … max): 31.5 ms … 46.5 ms 33 runs\n\n Summary\n 'GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index $commit' ran\n 9.19 ± 1.94 times faster than 'GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index $commit'\n\nAnd the effect continues in the warm cache case, though the absolute\nnumbers are much tinier:\n\n Benchmark #1: GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index $commit\n Time (mean ± σ): 12.0 ms ± 0.3 ms [User: 4.6 ms, System: 7.4 ms]\n Range (min … max): 11.4 ms … 13.2 ms 219 runs\n\n Benchmark #2: GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index $commit\n Time (mean ± σ): 3.0 ms ± 0.4 ms [User: 2.3 ms, System: 0.8 ms]\n Range (min … max): 2.6 ms … 5.5 ms 851 runs\n\n Summary\n 'GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index $commit' ran\n 3.95 ± 0.55 times faster than 'GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index $commit'\n\nThat implies to me that yes, it really is saving time, especially in the\ncold-cache case. But if you have to do any actual fill-in traversal, the\nbenefits get totally lost in the noise. _Especially_ in the cold-cache\ncase, because then we're faulting in the actual object data, etc.\n\nBy the way, one other thing I noticed is that having a fully-build\ncommit-graph also made a big difference (much bigger than this patch),\nespecially for the non-bitmapped commit. Which makes sense, since it is\nsaving us from loading commit objects from disk during fill-in\ntraversal.\n\nSo I dunno. There's absolutely savings for some cases, but I suspect in\npractice it's not going to really be noticeable. Part of me says \"well,\nif it ever provides a benefit and there isn't a downside, why not?\". So\njust devil's advocating on downsides for a moment:\n\n - there's some extra complexity in the file format and code to read\n and write these (and still fall back to the old system when they're\n absent). I don't think it's a deal-breaker, as it's really not that\n complicated a feature.\n\n - we're using extra bytes on disk (and the associated cold block cache\n effects there). It's not very many bytes, though (I guess 20 for the\n hash, plus a few offset integers; if we wanted to really\n penny-pinch, we could probably store 32-bit pointers to the hashes\n in the associated .idx file, at the cost of an extra level of\n indirection while binary searching). But that is only for a few\n hundred commits that are bitmapped, not all of them. And it's\n balanced by not needing to allocate a similar in-memory lookup table\n in each command. So it's probably a net win.\n\n> > I got an even more curious result when adding in \"--not --all\" (which\n> > the connectivity check under discussion would do). There the improvement\n> > from your patch should be even less, because we'd end up reading most of\n> > the bitmaps anyway. But I got:\n> \n> Interesting. Discussion about that series aside, I go from 28.6ms\n> without reading the table to 35.1ms reading it, which is much better in\n> absolute timings, but much worse relatively speaking.\n\nI suspect that's mostly noise. With \"--not --all\" in a regular linux.git\nrepo, I don't find any statistical difference.\n\nIn a fake repo with one ref per commit, everything is even more lost in\nthe noise (because we spend like 2000ms loading up all the tip commits).\nI suspect it's worse on a real repo with lots of refs (the \"spiky\nbranches\" thing I mentioned earlier in the thread), since there we'd\nhave to do even more fill-in traversal.\n\n> I can't quite seem to make sense of the perf report on that command.\n> Most of the time is spent faulting pages in, but most of the time\n> measured in the \"git\" object is in ubc_check. I don't really know how to\n> interpret that, but I'd be curious if you had any thoughts.\n\nubc_check() is basically \"computing sha1\" (we check the sha1 on every\nobject we call parse_object() for). I'd guess it's just time spent\nloading the negative tips (commits if you don't have a commit graph,\nplus tags to peel, though I guess we should be using the packed-refs\npeel optimization here).\n\n-Peff\n"},{"id":"428820","messageId":"YNx3DYLLmRlGLjBU@ncase","threadId":"56018","inReplyTo":"YNvJj42ALg4/t+0l@coredump.intra.peff.net","subject":"Re: [PATCH v2 2/3] receive-pack: skip connectivity checks on delete-only commands","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-06-30T13:52:13Z","receivedAt":"2021-06-30T13:54:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Jun 29, 2021 at 09:31:59PM -0400, Jeff King wrote:\n> On Mon, Jun 28, 2021 at 07:33:11AM +0200, Patrick Steinhardt wrote:\n> \n> > Fix this by not doing a connectivity check in case there were no pushed\n> > objects. Given that git-rev-walk(1) with only negative references will\n> > not do any graph walk, no performance improvements are to be expected.\n> > Conceptionally, it is still the right thing to do though.\n> \n> Even though it's not producing any exciting results, I agree this is\n> still a reasonable thing to do.\n> \n> I'm actually surprised it didn't help in your many-ref cases, just\n> because I think the traversal machinery is pretty eager to parse tags\n> and commits which are fed as tips.\n> \n> If I run \"git rev-list --not --all --stdin takes about 35ms. But if I make a ton of refs, like:\n> \n> git rev-list HEAD |\n> perl -lpe 's{.*}{update refs/foo/commit$. $&}' |\n> git update-ref --stdin\n> git pack-refs --all --prune\n> \n> then it takes about 2000ms (if you don't pack it's even worse, as you\n> might expect).\n> \n> So how come we don't see that improvement in your \"extrarefs\" cases?\n> Looking at patch 1, they also seem to make one ref for every commit.\n> \n> I think the answer may be below...\n> \n> > +\t/*\n> > +\t * If received commands only consist of deletions, then the client MUST\n> > +\t * NOT send a packfile because there cannot be any new objects in the\n> > +\t * first place. As a result, we do not set up a quarantine environment\n> > +\t * because we know no new objects will be received. And that in turn\n> > +\t * means that we can skip connectivity checks here.\n> > +\t */\n> > +\tif (tmp_objdir) {\n> \n> I think this will work, but we're now making assumptions about how\n> tmp_objdir will be initialized by the rest of the code.\n> \n> Could we make a more direct check of: skip calling rev-list if we have\n> no positive tips to feed? If we call iterate_receive_command() and it\n> returns end-of-list on the first call, then we know there is nothing to\n> feed (and as a bonus, this catches some more noop cases around shallow\n> repos; see iterate_receive_command).\n> \n> But that made me think that check_connected() could be doing this\n> itself. And indeed, it seems to already. At the top of that function is:\n> \n> if (fn(cb_data, &oid)) {\n> if (opt->err_fd)\n> close(opt->err_fd);\n> return err;\n> }\n> \n> So before we even spawn rev-list, if there's nothing to feed it, we'll\n> return right away (\"err\" will be \"0\" at this point). And I think that\n> has been there since the beginning of the function (it is even in the\n> old versions in builtin/fetch.c before it was factored out).\n> \n> And that explains why you didn't see any performance improvement. We're\n> already doing this optimization. :)\n> \n> -Peff\n\nHah, thanks for solving this riddle. I always kind of wondered what this\nwas supposed to do, where my assumption simply was \"It's in case the\ncallback returns an error\". But your explanation is obviously correct\nand neatly explains what's going on. So yes, I'll drop this patch.\n\nPatrick\n"},{"id":"429063","messageId":"YN9QbEaWgP09PfeD@nand.local","threadId":"56018","inReplyTo":"YNwE3wES3iv+Xynp@coredump.intra.peff.net","subject":"Re: [PATCH v2 3/3] connected: implement connectivity check using bitmaps","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2021-07-02T17:44:12Z","receivedAt":"2021-07-02T17:44:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jun 30, 2021 at 01:45:03AM -0400, Jeff King wrote:\n> That implies to me that yes, it really is saving time, especially in the\n> cold-cache case. But if you have to do any actual fill-in traversal, the\n> benefits get totally lost in the noise. _Especially_ in the cold-cache\n> case, because then we're faulting in the actual object data, etc.\n\nThat's definitely true. I would say that any patches in this direction\nwould have the general sense of \"this helps in some cases where we don't\nhave to do much traversal by eliminating an unnecessarily eager part of\nloading bitmaps, and does not make anything worse when the bitmap\ncoverage is incomplete (and requires traversing).\"\n\nI would add that these effects change with the size of the bitmap.\nLet's just consider the \"count the number of objects in a bitmapped\ncommit\". On my local copy of the kernel, I see a relatively modest\nimprovement:\n\n $ tip=2ab38c17aac10bf55ab3efde4c4db3893d8691d2\n $ hyperfine \\\n 'GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index $tip' \\\n 'GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index $tip' \\\n --warmup=3\n Benchmark #1: GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index $tip\n Time (mean ± σ): 21.5 ms ± 5.6 ms [User: 8.7 ms, System: 12.7 ms]\n Range (min … max): 12.4 ms … 34.2 ms 170 runs\n\n Benchmark #2: GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index $tip\n Time (mean ± σ): 10.6 ms ± 1.6 ms [User: 7.1 ms, System: 3.5 ms]\n Range (min … max): 4.5 ms … 11.9 ms 258 runs\n\nbut on my copy of the kernel's fork network repo (that containing all of\ntorvalds/linux's objects, as well as all of its fork's objects, too),\nthe magnitude of the effect is much bigger:\n\n Benchmark #1: GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index $tip\n Time (mean ± σ): 332.3 ms ± 12.6 ms [User: 210.4 ms, System: 121.8 ms]\n Range (min … max): 322.7 ms … 362.4 ms 10 runs\n\n Benchmark #2: GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index $tip\n Time (mean ± σ): 260.0 ms ± 9.3 ms [User: 191.0 ms, System: 69.0 ms]\n Range (min … max): 250.4 ms … 272.8 ms 11 runs\n\nThat's a more modest 1.28x improvement (versus 2.03x in just linux.git),\nbut the overall magnitude is much bigger.\n\nThis is basically an effect of the bitmaps themselves. In the latter\nexample, there are more bitmaps (around 1.6k of them, versus just over\n500 in my copy of just the kernel), and each of them are much wider\n(because there are far more objects, 40.2M versus just 7.8M). So there\nis more work to do, and the page cache is less efficient for us because\nwe can't fit as much of the .bitmap file in the page cache at once.\n\n> By the way, one other thing I noticed is that having a fully-build\n> commit-graph also made a big difference (much bigger than this patch),\n> especially for the non-bitmapped commit. Which makes sense, since it is\n> saving us from loading commit objects from disk during fill-in\n> traversal.\n\nLikewise having an reverse index helps a lot, too. That radix sort\nscales linearly with the number of objects in the bitmapped pack (plus\nyou're paying the cost to allocate more heap, etc).\n\nThis clouded up some of my timings in p5310, which made me think that it\nwould be a good idea to `git config pack.writeReverseIndex true` in the\nsetup for those tests, but an even better direction would be to change\nthe default of pack.writeReverseIndex to true everywhere.\n\n> So I dunno. There's absolutely savings for some cases, but I suspect in\n> practice it's not going to really be noticeable. Part of me says \"well,\n> if it ever provides a benefit and there isn't a downside, why not?\". So\n> just devil's advocating on downsides for a moment:\n>\n> - there's some extra complexity in the file format and code to read\n> and write these (and still fall back to the old system when they're\n> absent). I don't think it's a deal-breaker, as it's really not that\n> complicated a feature.\n\nI agree with both of these. The complexity is manageable, I think,\nespecially since I dropped support for the extended offset table (having\na bitmap file that is >2GiB seems extremely unlikely to me, and it's\npossible to add support for it in the future) and\nfanout table (there are usually less than <1k commits with bitmaps, so\na 256-entry fanout table doesn't seem to help much in benchmarking).\n\nSo what's left of the format is really just:\n\n - a table of object id's\n - a table of (uint32_t, uint32_t) tuples describing the (short) offset\n of the bitmap, and an index position of the xor'd bitmap (if one\n exists).\n\n> - we're using extra bytes on disk (and the associated cold block cache\n> effects there). It's not very many bytes, though (I guess 20 for the\n> hash, plus a few offset integers; if we wanted to really\n> penny-pinch, we could probably store 32-bit pointers to the hashes\n> in the associated .idx file, at the cost of an extra level of\n> indirection while binary searching). But that is only for a few\n> hundred commits that are bitmapped, not all of them. And it's\n> balanced by not needing to allocate a similar in-memory lookup table\n> in each command. So it's probably a net win.\n\nWorth benchmarking, at least.\n\nI'll be offline for the next ~week and a half for my wedding, but I'll\npost some patches to the list shortly after I get back.\n\nThanks,\nTaylor\n"},{"id":"429075","messageId":"YN+DbDhMrqOnJbcO@coredump.intra.peff.net","threadId":"56018","inReplyTo":"YN9QbEaWgP09PfeD@nand.local","subject":"Re: [PATCH v2 3/3] connected: implement connectivity check using bitmaps","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-07-02T21:21:48Z","receivedAt":"2021-07-02T21:21:50Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jul 02, 2021 at 01:44:12PM -0400, Taylor Blau wrote:\n\n> I would add that these effects change with the size of the bitmap.\n> Let's just consider the \"count the number of objects in a bitmapped\n> commit\". On my local copy of the kernel, I see a relatively modest\n> improvement:\n> \n> $ tip=2ab38c17aac10bf55ab3efde4c4db3893d8691d2\n> $ hyperfine \\\n> 'GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index $tip' \\\n> 'GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index $tip' \\\n> --warmup=3\n> Benchmark #1: GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index $tip\n> Time (mean ± σ): 21.5 ms ± 5.6 ms [User: 8.7 ms, System: 12.7 ms]\n> Range (min … max): 12.4 ms … 34.2 ms 170 runs\n> \n> Benchmark #2: GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index $tip\n> Time (mean ± σ): 10.6 ms ± 1.6 ms [User: 7.1 ms, System: 3.5 ms]\n> Range (min … max): 4.5 ms … 11.9 ms 258 runs\n> \n> but on my copy of the kernel's fork network repo (that containing all of\n> torvalds/linux's objects, as well as all of its fork's objects, too),\n> the magnitude of the effect is much bigger:\n> \n> Benchmark #1: GIT_READ_COMMIT_TABLE=0 git.compile rev-list --count --objects --use-bitmap-index $tip\n> Time (mean ± σ): 332.3 ms ± 12.6 ms [User: 210.4 ms, System: 121.8 ms]\n> Range (min … max): 322.7 ms … 362.4 ms 10 runs\n> \n> Benchmark #2: GIT_READ_COMMIT_TABLE=1 git.compile rev-list --count --objects --use-bitmap-index $tip\n> Time (mean ± σ): 260.0 ms ± 9.3 ms [User: 191.0 ms, System: 69.0 ms]\n> Range (min … max): 250.4 ms … 272.8 ms 11 runs\n> \n> That's a more modest 1.28x improvement (versus 2.03x in just linux.git),\n> but the overall magnitude is much bigger.\n\nThanks, this is much more compelling. 70ms is a lot of startup time to\nsave. I am a little surprised that a no-traversal bitmap query like this\nwould still take 300ms. I wonder if 2ab38c17aac actually got a bitmap in\nyour second example (and if not, then there are probably cases where the\nrelative speedup would be even more impressive).\n\n> This clouded up some of my timings in p5310, which made me think that it\n> would be a good idea to `git config pack.writeReverseIndex true` in the\n> setup for those tests, but an even better direction would be to change\n> the default of pack.writeReverseIndex to true everywhere.\n\nYes, I'd be in favor of that. IMHO the reason to make it configurable at\nall was not because it's ever a bad idea, but just to phase it in and\nget experience with it (and to give an escape hatch for debugging it).\n\nIt's probably _less_ useful for local clones that are not serving\nfetches. But every push is already generating the same thing in-memory,\nso it seems like a good tradeoff to just use it everywhere.\n\n> > - there's some extra complexity in the file format and code to read\n> > and write these (and still fall back to the old system when they're\n> > absent). I don't think it's a deal-breaker, as it's really not that\n> > complicated a feature.\n> \n> I agree with both of these. The complexity is manageable, I think,\n> especially since I dropped support for the extended offset table (having\n> a bitmap file that is >2GiB seems extremely unlikely to me, and it's\n> possible to add support for it in the future) and\n> fanout table (there are usually less than <1k commits with bitmaps, so\n> a 256-entry fanout table doesn't seem to help much in benchmarking).\n> \n> So what's left of the format is really just:\n> \n> - a table of object id's\n> - a table of (uint32_t, uint32_t) tuples describing the (short) offset\n> of the bitmap, and an index position of the xor'd bitmap (if one\n> exists).\n\nYeah, that really seems quite simple. I'd have to judge after seeing the\ncleaned up code, but I suspect it's not going to be a burden.\n\n> I'll be offline for the next ~week and a half for my wedding, but I'll\n> post some patches to the list shortly after I get back.\n\nYep, no rush. Thanks for looking into this.\n\n-Peff\n"},{"id":"430629","messageId":"YPbdJFfykDo1diJK@ncase","threadId":"56018","inReplyTo":"YNvOJQDYRWpFa+4S@coredump.intra.peff.net","subject":"Re: [PATCH v2 3/3] connected: implement connectivity check using bitmaps","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-07-20T14:26:44Z","receivedAt":"2021-07-20T14:28:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Jun 29, 2021 at 09:51:33PM -0400, Jeff King wrote:\n> On Mon, Jun 28, 2021 at 07:33:15AM +0200, Patrick Steinhardt wrote:\n> \n> > As expected, performance doesn't change in cases where we do not have a\n> > bitmap available given that the old code path still kicks in. In case we\n> > do have bitmaps, this is kind of a mixed bag: while git-receive-pack(1)\n> > is slower in a \"normal\" clone of linux.git, it is significantly faster\n> > for a clone with lots of references. The slowness can potentially be\n> > explained by the overhead of loading the bitmap. On the other hand, the\n> > new code is faster as expected in repos which have lots of references\n> > given that we do not have to mark all negative references anymore.\n> \n> Hmm. We _do_ still have to mark those negative references now, though\n> (the bitmap code still considers each as a reachability tip for the\n> \"have\" side of the traversal). It's just that we may have to do less\n> traversal on them, if they're mentioned by other bitmaps.\n> \n> So in that sense I don't think your \"a ref for every commit\" cases are\n> all that interesting. Any bitmap near the tip of history is going to\n> include a bit for all those old commits, because our fake set of refs\n> are all reachable. A much more interesting history is when you have a\n> bunch of little unreachable spikes coming off the main history.\n> \n> This is common if you have a lot of branches in the repo, but also if\n> you maintain a lot of book-keeping refs (like the refs/pull/* we do at\n> GitHub; I assume GitLab does something similar).\n> \n> Here are some real-world numbers from one of the repos that gives us\n> frequent problems with bitmaps. refs/pull/9937/head in this case is an\n> unmerged PR with 8 commits on it.\n\nYeah, this kind of brings us back to the old topic of pathological\nperformance combined with bitmaps. As I said in the cover letter, I\nhaven't been particularly happy with the results of this version, but\nrather intended it as an RFC. Taylor's extension does look quite\ninteresting, but ultimately I'm not sure whether we want to use bitmaps\nfor connectivity checks. Your spiky-branches example neatly highlights\nthat it cannot really work in the general case.\n\nI wonder where that leaves us. I'm out of ideas on how to solve this in\nthe general case for any push/connectivity check, so I guess that any\nalternative approach would instead make use of heuristics.\n\nIn the current context, I care mostly about the user-side context, which\nis interactive pushes. Without knowing the numbers, my bet is that the\nmost frequent usecase here is the push of a single branch with only a\nbunch of commits. If the pushed commit is a descendant of any existing\ncommit, then we can limit the connectivity check to `git rev-list\n--objects $newoid --not $oldoid` instead of `--not --all`. There's two\nissues:\n\n - \"descendant of any existing commit\" is again the same territory\n performance-wise as `--all`. So we can heuristically limit this\n either to the to-be-updated-target reference if it exists, or\n HEAD.\n\n - Calculating ancestry can be expensive if there's too many commits\n in between or if history is unrelated. We may limit this check to\n a small number like only checking the most recent 16 commits.\n\nIf these conditions hold, then we can do above optimized check,\notherwise we fall back to the old check.\n\nDo we actually gain anything by this? The following was executed with\nlinux.git and stable tags. afeb391 is an empty commit on top of master.\n\n Benchmark #1: git rev-list afeb391 --not --all\n Time (mean ± σ): 64.1 ms ± 8.0 ms [User: 52.8 ms, System: 11.1 ms]\n Range (min … max): 58.2 ms … 79.5 ms 37 runs\n\n Benchmark #2: git rev-list afeb391 --not master\n Time (mean ± σ): 1.6 ms ± 0.5 ms [User: 1.0 ms, System: 1.0 ms]\n Range (min … max): 0.4 ms … 2.2 ms 1678 runs\n\nObviously not a real-world example, but it serves as a hint that it\nwould help in some cases and potentially pay out quite well.\n\nPatrick\n"},{"id":"431668","messageId":"cover.1627896460.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1624858240.git.ps@pks.im","subject":"[PATCH v3 0/4] Speed up connectivity checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-02T09:37:57Z","receivedAt":"2021-08-02T09:38:10Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nI finally found some time again to have another look at my old problem\nof slow connectivity checks. After my previous two approaches of using\nthe quarantine directory and using bitmaps proved to not really be\nviable, I've taken a step back yet again. The result is this series,\nwhich speeds up the connectivity checks by optimizing \"revison.c\". More\nspecifically, I'm mostly tweaking how we're queueing up references,\nwhich is the most pressing issue we've observed at GitLab when doing\nconnectivity checks in repos with many references.\n\nThe following optimizations are part of this series. All benchmarks were\ndone on [1], which is a repository with about 2.2 million references\n(even though most of them are hidden to public users) with `git rev-list\n--objects --quiet --unsorted-input --not --all --not $newrev`.\n\n 1. We used to sort the input references in git-rev-list(1). This is\n moot in the context of connectivity checks, so a new flag\n suppresses this sorting. This improves the command by ~30% from\n 7.6s to 4.9s.\n\n 2. We did some busy-work, loading each reference twice via\n `get_reference()`. We now don't anymore, resulting in a ~8%\n speedup from 5.0s to 4.6s.\n\n 3. An optimization was done to how we load objects. Previously, we\n always called `oid_object_info()`, even if we had already loaded\n the object. This was tweaked to use `lookup_unknown_object()`,\n which is a performance-memory tradeoff. This saves us another 7%,\n going from 4.7s to 4.4s, but it's a prereq for (4).\n\n 4. We now make better use of the commit-graph in that we first try\n loading from there before we load it from the ODB. This is a 40%\n speedup, going from 4.4s to 2.8s.\n\nThe result is a speedup of about 65%. The nice thing compared to\nprevious versions is that this should also be visible when directly\nexecuting git-rev-list(1) or doing a revwalk.\n\nPatch #1 still needs some polishing if we agree that this patch series\nmakes sense, given that it's still missing documentation.\n\nPatrick\n\n[1]: https://gitlab.com/gitlab-org/gitlab.git\n\nPatrick Steinhardt (4):\n connected: do not sort input revisions\n revision: stop retrieving reference twice\n revision: avoid loading object headers multiple times\n revision: avoid hitting packfiles when commits are in commit-graph\n\n commit-graph.c | 55 +++++++++++++++++++++++++++++++++++++++-----------\n commit-graph.h | 2 ++\n connected.c | 1 +\n revision.c | 23 ++++++++++++++++-----\n revision.h | 1 +\n 5 files changed, 65 insertions(+), 17 deletions(-)\n\n-- \n2.32.0\n\n"},{"id":"431669","messageId":"1fd83f726a04dfb5be27c74cb116618cb76be923.1627896460.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1627896460.git.ps@pks.im","subject":"[PATCH v3 1/4] connected: do not sort input revisions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-02T09:38:01Z","receivedAt":"2021-08-02T09:38:11Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In order to compute whether objects reachable from a set of tips are all\nconnected, we do a revision walk with these tips as positive references\nand `--not --all`. `--not --all` will cause the revision walk to load\nall preexisting references as uninteresting, which can be very expensive\nin repositories with many references.\n\nBenchmarking the git-rev-list(1) command highlights that by far the most\nexpensive single phase is initial sorting of the input revisions: after\nall references have been loaded, we first sort commits by author date.\nIn a real-world repository with about 2.2 million references, it makes\nup about 40% of the total runtime of git-rev-list(1).\n\nUltimately, the connectivity check shouldn't really bother about the\norder of input revisions at all. We only care whether we can actually\nwalk all objects until we hit the cut-off point. So sorting the input is\na complete waste of time.\n\nIntroduce a new \"--unsorted-input\" flag to git-rev-list(1) which will\ncause it to not sort the commits and adjust the connectivity check to\nalways pass the flag. This results in the following speedups, executed\nin a clone of gitlab-org/gitlab [1]:\n\n Benchmark #1: git rev-list --objects --quiet --not --all --not $(cat newrev)\n Time (mean ± σ): 7.639 s ± 0.065 s [User: 7.304 s, System: 0.335 s]\n Range (min … max): 7.543 s … 7.742 s 10 runs\n\n Benchmark #2: git rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n Time (mean ± σ): 4.995 s ± 0.044 s [User: 4.657 s, System: 0.337 s]\n Range (min … max): 4.909 s … 5.048 s 10 runs\n\n Summary\n 'git rev-list --unsorted-input --objects --quiet --not --all --not $(cat newrev)' ran\n 1.53 ± 0.02 times faster than 'git rev-list --objects --quiet --not --all --not $newrev'\n\n[1]: https://gitlab.com/gitlab-org/gitlab.git. Note that not all refs\n are visible to clients.\n\nSigned-off-by: Patrick Steinhardt \n---\n connected.c | 1 +\n revision.c | 4 +++-\n revision.h | 1 +\n 3 files changed, 5 insertions(+), 1 deletion(-)\n\ndiff --git a/connected.c b/connected.c\nindex b18299fdf0..b5f9523a5f 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -106,6 +106,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \tif (opt->progress)\n \t\tstrvec_pushf(&rev_list.args, \"--progress=%s\",\n \t\t\t _(\"Checking connectivity\"));\n+\tstrvec_push(&rev_list.args, \"--unsorted-input\");\n \n \trev_list.git_cmd = 1;\n \trev_list.env = opt->env;\ndiff --git a/revision.c b/revision.c\nindex cddd0542a6..7eb09009ba 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -2256,6 +2256,8 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg\n \t} else if (!strcmp(arg, \"--author-date-order\")) {\n \t\trevs->sort_order = REV_SORT_BY_AUTHOR_DATE;\n \t\trevs->topo_order = 1;\n+\t} else if (!strcmp(arg, \"--unsorted-input\")) {\n+\t\trevs->unsorted_input = 1;\n \t} else if (!strcmp(arg, \"--early-output\")) {\n \t\trevs->early_output = 100;\n \t\trevs->topo_order = 1;\n@@ -3584,7 +3586,7 @@ int prepare_revision_walk(struct rev_info *revs)\n \n \tif (!revs->reflog_info)\n \t\tprepare_to_use_bloom_filter(revs);\n-\tif (revs->no_walk != REVISION_WALK_NO_WALK_UNSORTED)\n+\tif (revs->no_walk != REVISION_WALK_NO_WALK_UNSORTED && !revs->unsorted_input)\n \t\tcommit_list_sort_by_date(&revs->commits);\n \tif (revs->no_walk)\n \t\treturn 0;\ndiff --git a/revision.h b/revision.h\nindex fbb068da9f..ef6998509a 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -134,6 +134,7 @@ struct rev_info {\n \t\t\tsimplify_history:1,\n \t\t\tshow_pulls:1,\n \t\t\ttopo_order:1,\n+\t\t\tunsorted_input:1,\n \t\t\tsimplify_merges:1,\n \t\t\tsimplify_by_decoration:1,\n \t\t\tsingle_worktree:1,\n-- \n2.32.0\n\n"},{"id":"431670","messageId":"db854806498b316b4f59c33d70a1ea9c1401fdb7.1627896460.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1627896460.git.ps@pks.im","subject":"[PATCH v3 2/4] revision: stop retrieving reference twice","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-02T09:38:06Z","receivedAt":"2021-08-02T09:38:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When queueing up references for the revision walk, `handle_one_ref()`\nwill resolve the reference's object ID and then queue the ID as pending\nobject via `add_pending_oid()`. But `add_pending_oid()` will again try\nto resolve the object ID to an object, effectively duplicating the work\nits caller already did before.\n\nFix the issue by instead calling `add_pending_object()`, which takes the\nalready-resolved object as input. In a repository with lots of refs,\nthis translates in a nearly 10% speedup:\n\n Benchmark #1: HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n Time (mean ± σ): 5.015 s ± 0.038 s [User: 4.698 s, System: 0.316 s]\n Range (min … max): 4.970 s … 5.089 s 10 runs\n\n Benchmark #2: HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n Time (mean ± σ): 4.606 s ± 0.029 s [User: 4.260 s, System: 0.345 s]\n Range (min … max): 4.565 s … 4.657 s 10 runs\n\n Summary\n 'HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev' ran\n 1.09 ± 0.01 times faster than 'HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev'\n\nSigned-off-by: Patrick Steinhardt \n---\n revision.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/revision.c b/revision.c\nindex 7eb09009ba..f06a5d63a3 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1534,7 +1534,7 @@ static int handle_one_ref(const char *path, const struct object_id *oid,\n \n \tobject = get_reference(cb->all_revs, path, oid, cb->all_flags);\n \tadd_rev_cmdline(cb->all_revs, object, path, REV_CMD_REF, cb->all_flags);\n-\tadd_pending_oid(cb->all_revs, path, oid, cb->all_flags);\n+\tadd_pending_object(cb->all_revs, object, path);\n \treturn 0;\n }\n \n-- \n2.32.0\n\n"},{"id":"431671","messageId":"b9897e102afbcab3bfee58ed8bda24257d8b54fb.1627896460.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1627896460.git.ps@pks.im","subject":"[PATCH v3 3/4] revision: avoid loading object headers multiple times","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-02T09:38:10Z","receivedAt":"2021-08-02T09:38:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When loading references, we try to optimize loading of commits by using\nthe commit graph. To do so, we first need to determine whether the\nobject actually is a commit or not, which is why we always execute\n`oid_object_info()` first. Like this, we'll unpack the object header of\neach object first.\n\nThis pattern can be quite inefficient in case many references point to\nthe same commit: if the object didn't end up in the cached objects, then\nwe'll repeatedly unpack the same object header, even if we've already\nseen the object before.\n\nOptimize this pattern by using `lookup_unknown_object()` first in order\nto determine whether we've seen the object before. If so, then we don't\nneed to re-parse the header but can directly use its object information\nand thus gain a modest performance improvement. Executed in a real-world\nrepository with around 2.2 million references:\n\n Benchmark #1: HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n Time (mean ± σ): 4.771 s ± 0.238 s [User: 4.440 s, System: 0.330 s]\n Range (min … max): 4.539 s … 5.219 s 10 runs\n\n Benchmark #2: HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n Time (mean ± σ): 4.454 s ± 0.037 s [User: 4.122 s, System: 0.332 s]\n Range (min … max): 4.375 s … 4.496 s 10 runs\n\n Summary\n 'HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev' ran\n 1.07 ± 0.05 times faster than 'HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev'\n\nThe downside is that `lookup_unknown_object()` is forced to always\nallocate an object such that it's big enough to host all object types'\nstructs and thus we may waste memory here. This tradeoff is probably\nworth it though considering the following struct sizes:\n\n - commit: 72 bytes\n - tree: 56 bytes\n - blob: 40 bytes\n - tag: 64 bytes\n\nAssuming that in almost all repositories, most references will point to\neither a tag or a commit, we'd have a modest increase in memory\nconsumption of about 12.5% here.\n\nSigned-off-by: Patrick Steinhardt \n---\n revision.c | 15 ++++++++++++---\n 1 file changed, 12 insertions(+), 3 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex f06a5d63a3..671b6d6513 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -359,14 +359,22 @@ static struct object *get_reference(struct rev_info *revs, const char *name,\n \t\t\t\t const struct object_id *oid,\n \t\t\t\t unsigned int flags)\n {\n-\tstruct object *object;\n+\tstruct object *object = lookup_unknown_object(revs->repo, oid);\n+\n+\tif (object->type == OBJ_NONE) {\n+\t\tint type = oid_object_info(revs->repo, oid, NULL);\n+\t\tif (type < 0 || !object_as_type(object, type, 1)) {\n+\t\t\tobject = NULL;\n+\t\t\tgoto out;\n+\t\t}\n+\t}\n \n \t/*\n \t * If the repository has commit graphs, repo_parse_commit() avoids\n \t * reading the object buffer, so use it whenever possible.\n \t */\n-\tif (oid_object_info(revs->repo, oid, NULL) == OBJ_COMMIT) {\n-\t\tstruct commit *c = lookup_commit(revs->repo, oid);\n+\tif (object->type == OBJ_COMMIT) {\n+\t\tstruct commit *c = (struct commit *) object;\n \t\tif (!repo_parse_commit(revs->repo, c))\n \t\t\tobject = (struct object *) c;\n \t\telse\n@@ -375,6 +383,7 @@ static struct object *get_reference(struct rev_info *revs, const char *name,\n \t\tobject = parse_object(revs->repo, oid);\n \t}\n \n+out:\n \tif (!object) {\n \t\tif (revs->ignore_missing)\n \t\t\treturn object;\n-- \n2.32.0\n\n"},{"id":"431672","messageId":"f6fc2a5e6d94befa915fb59b6296ce3153820c13.1627896460.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1627896460.git.ps@pks.im","subject":"[PATCH v3 4/4] revision: avoid hitting packfiles when commits are in commit-graph","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-02T09:38:15Z","receivedAt":"2021-08-02T09:38:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When queueing references in git-rev-list(1), we try to either reuse an\nalready parsed object or alternatively we load the object header from\ndisk in order to determine its type. This is inefficient for commits\nthough in cases where we have a commit graph available: instead of\nhitting the real object on disk to determine its type, we may instead\nsearch the object graph for the object ID. In case it's found, we can\ndirectly fill in the commit object, otherwise we can still hit the disk\nto determine the object's type.\n\nExpose a new function `find_object_in_graph()`, which fills in an object\nof unknown type in case we find its object ID in the graph. This\nprovides a big performance win in cases where there is a commit-graph\navailable in the repository in case we load lots of references. The\nfollowing has been executed in a real-world repository with about 2.2\nmillion refs:\n\n Benchmark #1: HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n Time (mean ± σ): 4.465 s ± 0.037 s [User: 4.144 s, System: 0.320 s]\n Range (min … max): 4.411 s … 4.514 s 10 runs\n\n Benchmark #2: HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n Time (mean ± σ): 2.886 s ± 0.032 s [User: 2.570 s, System: 0.316 s]\n Range (min … max): 2.826 s … 2.933 s 10 runs\n\n Summary\n 'HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev' ran\n 1.55 ± 0.02 times faster than 'HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev'\n\nSigned-off-by: Patrick Steinhardt \n---\n commit-graph.c | 55 +++++++++++++++++++++++++++++++++++++++-----------\n commit-graph.h | 2 ++\n revision.c | 10 +++++----\n 3 files changed, 51 insertions(+), 16 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 3860a0d847..a81d5cebc0 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -864,6 +864,48 @@ static int fill_commit_in_graph(struct repository *r,\n \treturn 1;\n }\n \n+static int find_object_id_in_graph(const struct object_id *id, struct commit_graph *g, uint32_t *pos)\n+{\n+\tstruct commit_graph *cur_g = g;\n+\tuint32_t lex_index;\n+\n+\twhile (cur_g && !bsearch_graph(cur_g, (struct object_id *)id, &lex_index))\n+\t\tcur_g = cur_g->base_graph;\n+\n+\tif (cur_g) {\n+\t\t*pos = lex_index + cur_g->num_commits_in_base;\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+int find_object_in_graph(struct repository *repo, struct object *object)\n+{\n+\tstruct commit *commit;\n+\tuint32_t pos;\n+\n+\tif (object->parsed) {\n+\t\tif (object->type != OBJ_COMMIT)\n+\t\t\treturn -1;\n+\t\treturn 0;\n+\t}\n+\n+\tif (!repo->objects->commit_graph)\n+\t\treturn -1;\n+\n+\tif (!find_object_id_in_graph(&object->oid, repo->objects->commit_graph, &pos))\n+\t\treturn -1;\n+\n+\tcommit = object_as_type(object, OBJ_COMMIT, 1);\n+\tif (!commit)\n+\t\treturn -1;\n+\tif (!fill_commit_in_graph(repo, commit, repo->objects->commit_graph, pos))\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t *pos)\n {\n \tuint32_t graph_pos = commit_graph_position(item);\n@@ -871,18 +913,7 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n \t\t*pos = graph_pos;\n \t\treturn 1;\n \t} else {\n-\t\tstruct commit_graph *cur_g = g;\n-\t\tuint32_t lex_index;\n-\n-\t\twhile (cur_g && !bsearch_graph(cur_g, &(item->object.oid), &lex_index))\n-\t\t\tcur_g = cur_g->base_graph;\n-\n-\t\tif (cur_g) {\n-\t\t\t*pos = lex_index + cur_g->num_commits_in_base;\n-\t\t\treturn 1;\n-\t\t}\n-\n-\t\treturn 0;\n+\t\treturn find_object_id_in_graph(&item->object.oid, g, pos);\n \t}\n }\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex 96c24fb577..f373fab4c0 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -139,6 +139,8 @@ int write_commit_graph(struct object_directory *odb,\n \t\t enum commit_graph_write_flags flags,\n \t\t const struct commit_graph_opts *opts);\n \n+int find_object_in_graph(struct repository *repo, struct object *object);\n+\n #define COMMIT_GRAPH_VERIFY_SHALLOW\t(1 << 0)\n \n int verify_commit_graph(struct repository *r, struct commit_graph *g, int flags);\ndiff --git a/revision.c b/revision.c\nindex 671b6d6513..c3f9cf2998 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -362,10 +362,12 @@ static struct object *get_reference(struct rev_info *revs, const char *name,\n \tstruct object *object = lookup_unknown_object(revs->repo, oid);\n \n \tif (object->type == OBJ_NONE) {\n-\t\tint type = oid_object_info(revs->repo, oid, NULL);\n-\t\tif (type < 0 || !object_as_type(object, type, 1)) {\n-\t\t\tobject = NULL;\n-\t\t\tgoto out;\n+\t\tif (find_object_in_graph(revs->repo, object) < 0) {\n+\t\t\tint type = oid_object_info(revs->repo, oid, NULL);\n+\t\t\tif (type < 0 || !object_as_type(object, type, 1)) {\n+\t\t\t\tobject = NULL;\n+\t\t\t\tgoto out;\n+\t\t\t}\n \t\t}\n \t}\n \n-- \n2.32.0\n\n"},{"id":"431680","messageId":"871r7covsa.fsf@evledraar.gmail.com","threadId":"56018","inReplyTo":"1fd83f726a04dfb5be27c74cb116618cb76be923.1627896460.git.ps@pks.im","subject":"Re: [PATCH v3 1/4] connected: do not sort input revisions","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-02T12:49:29Z","receivedAt":"2021-08-02T12:50:34Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Aug 02 2021, Patrick Steinhardt wrote:\n\n> [[PGP Signed Part:Undecided]]\n> In order to compute whether objects reachable from a set of tips are all\n> connected, we do a revision walk with these tips as positive references\n> and `--not --all`. `--not --all` will cause the revision walk to load\n> all preexisting references as uninteresting, which can be very expensive\n> in repositories with many references.\n>\n> Benchmarking the git-rev-list(1) command highlights that by far the most\n> expensive single phase is initial sorting of the input revisions: after\n> all references have been loaded, we first sort commits by author date.\n> In a real-world repository with about 2.2 million references, it makes\n> up about 40% of the total runtime of git-rev-list(1).\n>\n> Ultimately, the connectivity check shouldn't really bother about the\n> order of input revisions at all. We only care whether we can actually\n> walk all objects until we hit the cut-off point. So sorting the input is\n> a complete waste of time.\n\nReally good results:\n\n> Introduce a new \"--unsorted-input\" flag to git-rev-list(1) which will\n> cause it to not sort the commits and adjust the connectivity check to\n> always pass the flag. This results in the following speedups, executed\n> in a clone of gitlab-org/gitlab [1]:\n>\n> Benchmark #1: git rev-list --objects --quiet --not --all --not $(cat newrev)\n> Time (mean ± σ): 7.639 s ± 0.065 s [User: 7.304 s, System: 0.335 s]\n> Range (min … max): 7.543 s … 7.742 s 10 runs\n>\n> Benchmark #2: git rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n> Time (mean ± σ): 4.995 s ± 0.044 s [User: 4.657 s, System: 0.337 s]\n> Range (min … max): 4.909 s … 5.048 s 10 runs\n>\n> Summary\n> 'git rev-list --unsorted-input --objects --quiet --not --all --not $(cat newrev)' ran\n> 1.53 ± 0.02 times faster than 'git rev-list --objects --quiet --not --all --not $newrev'\n\nJust bikeshedding for a potential re-roll, perhaps --unordered-input, so\nthat it matches/rhymes with the existing \"git cat-file --unordered\",\nwhich serves the same conceptual purpose (except this one's input, that\none's output).\n"},{"id":"431681","messageId":"87y29knh0x.fsf@evledraar.gmail.com","threadId":"56018","inReplyTo":"db854806498b316b4f59c33d70a1ea9c1401fdb7.1627896460.git.ps@pks.im","subject":"Re: [PATCH v3 2/4] revision: stop retrieving reference twice","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-02T12:53:30Z","receivedAt":"2021-08-02T12:54:45Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Aug 02 2021, Patrick Steinhardt wrote:\n\n> [[PGP Signed Part:Undecided]]\n> When queueing up references for the revision walk, `handle_one_ref()`\n> will resolve the reference's object ID and then queue the ID as pending\n> object via `add_pending_oid()`. But `add_pending_oid()` will again try\n> to resolve the object ID to an object, effectively duplicating the work\n> its caller already did before.\n>\n> Fix the issue by instead calling `add_pending_object()`, which takes the\n> already-resolved object as input. In a repository with lots of refs,\n> this translates in a nearly 10% speedup:\n>\n> Benchmark #1: HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n> Time (mean ± σ): 5.015 s ± 0.038 s [User: 4.698 s, System: 0.316 s]\n> Range (min … max): 4.970 s … 5.089 s 10 runs\n>\n> Benchmark #2: HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n> Time (mean ± σ): 4.606 s ± 0.029 s [User: 4.260 s, System: 0.345 s]\n> Range (min … max): 4.565 s … 4.657 s 10 runs\n>\n> Summary\n> 'HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev' ran\n> 1.09 ± 0.01 times faster than 'HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev'\n\nIt might be worth calling out explicitly that it's not just\n\"effectively\", but that add_pending_oid() is just a thin wrapper for\nget_reference() followed by add_pending_object(), so we're guaranteed to\nget the exact same result here as before, just without the duplicate\nwork.\n\nI.e. we're not going down some other lookup path that uses object\nlookups with different flags or whatever as a result of this change.\n"},{"id":"431682","messageId":"87v94onguw.fsf@evledraar.gmail.com","threadId":"56018","inReplyTo":"b9897e102afbcab3bfee58ed8bda24257d8b54fb.1627896460.git.ps@pks.im","subject":"Re: [PATCH v3 3/4] revision: avoid loading object headers multiple times","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-02T12:55:09Z","receivedAt":"2021-08-02T12:58:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Aug 02 2021, Patrick Steinhardt wrote:\n\n> [[PGP Signed Part:Undecided]]\n> When loading references, we try to optimize loading of commits by using\n> the commit graph. To do so, we first need to determine whether the\n> object actually is a commit or not, which is why we always execute\n> `oid_object_info()` first. Like this, we'll unpack the object header of\n> each object first.\n>\n> This pattern can be quite inefficient in case many references point to\n> the same commit: if the object didn't end up in the cached objects, then\n> we'll repeatedly unpack the same object header, even if we've already\n> seen the object before.\n>\n> Optimize this pattern by using `lookup_unknown_object()` first in order\n> to determine whether we've seen the object before. If so, then we don't\n> need to re-parse the header but can directly use its object information\n> and thus gain a modest performance improvement. Executed in a real-world\n> repository with around 2.2 million references:\n>\n> Benchmark #1: HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n> Time (mean ± σ): 4.771 s ± 0.238 s [User: 4.440 s, System: 0.330 s]\n> Range (min … max): 4.539 s … 5.219 s 10 runs\n>\n> Benchmark #2: HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n> Time (mean ± σ): 4.454 s ± 0.037 s [User: 4.122 s, System: 0.332 s]\n> Range (min … max): 4.375 s … 4.496 s 10 runs\n>\n> Summary\n> 'HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev' ran\n> 1.07 ± 0.05 times faster than 'HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev'\n>\n> The downside is that `lookup_unknown_object()` is forced to always\n> allocate an object such that it's big enough to host all object types'\n> structs and thus we may waste memory here. This tradeoff is probably\n> worth it though considering the following struct sizes:\n>\n> - commit: 72 bytes\n> - tree: 56 bytes\n> - blob: 40 bytes\n> - tag: 64 bytes\n>\n> Assuming that in almost all repositories, most references will point to\n> either a tag or a commit, we'd have a modest increase in memory\n> consumption of about 12.5% here.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> revision.c | 15 ++++++++++++---\n> 1 file changed, 12 insertions(+), 3 deletions(-)\n>\n> diff --git a/revision.c b/revision.c\n> index f06a5d63a3..671b6d6513 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -359,14 +359,22 @@ static struct object *get_reference(struct rev_info *revs, const char *name,\n> \t\t\t\t const struct object_id *oid,\n> \t\t\t\t unsigned int flags)\n> {\n> -\tstruct object *object;\n> +\tstruct object *object = lookup_unknown_object(revs->repo, oid);\n> +\n> +\tif (object->type == OBJ_NONE) {\n> +\t\tint type = oid_object_info(revs->repo, oid, NULL);\n> +\t\tif (type < 0 || !object_as_type(object, type, 1)) {\n\nLet's s/int type/enum object_type, personally I think we should never do\n\"type < 0\" either, and check OBJ_BAD explicitly, but I've seemingly lost\nthat discussion on-list before.\n\nBut I think the consensus is that we should not do !type, but rather\ntype == OBJ_NONE.\n"},{"id":"431744","messageId":"xmqqa6lzwu31.fsf@gitster.g","threadId":"56018","inReplyTo":"1fd83f726a04dfb5be27c74cb116618cb76be923.1627896460.git.ps@pks.im","subject":"Re: [PATCH v3 1/4] connected: do not sort input revisions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-02T19:00:02Z","receivedAt":"2021-08-02T19:00:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> In order to compute whether objects reachable from a set of tips are all\n> connected, we do a revision walk with these tips as positive references\n> and `--not --all`. `--not --all` will cause the revision walk to load\n> all preexisting references as uninteresting, which can be very expensive\n> in repositories with many references.\n>\n> Benchmarking the git-rev-list(1) command highlights that by far the most\n> expensive single phase is initial sorting of the input revisions: after\n> all references have been loaded, we first sort commits by author date.\n> In a real-world repository with about 2.2 million references, it makes\n> up about 40% of the total runtime of git-rev-list(1).\n\nNice finding.\n\n> Ultimately, the connectivity check shouldn't really bother about the\n> order of input revisions at all. We only care whether we can actually\n> walk all objects until we hit the cut-off point. So sorting the input is\n> a complete waste of time.\n\nSorting of positive side is done to help both performance and\ncorrectness in regular use of the traversal machinery, especially\nwhen reachability bitmap is not in effect, but on the negative side\nI do not think there is any downside to omit sorting offhand. The\nonly case that may get affected is when the revision.c::SLOP kicks\nin to deal with oddball commits with incorrect committer timestamps,\nbut then the result of the sorting isn't to be trusted anyway, so...\n\n> Introduce a new \"--unsorted-input\" flag to git-rev-list(1) which will\n> cause it to not sort the commits and adjust the connectivity check to\n> always pass the flag. This results in the following speedups, executed\n> in a clone of gitlab-org/gitlab [1]:\n> ...\n> [1]: https://gitlab.com/gitlab-org/gitlab.git. Note that not all refs\n> are visible to clients.\n\nSo is this the 2.2 million refs thing?\n\n> @@ -3584,7 +3586,7 @@ int prepare_revision_walk(struct rev_info *revs)\n> \n> \tif (!revs->reflog_info)\n> \t\tprepare_to_use_bloom_filter(revs);\n> -\tif (revs->no_walk != REVISION_WALK_NO_WALK_UNSORTED)\n> +\tif (revs->no_walk != REVISION_WALK_NO_WALK_UNSORTED && !revs->unsorted_input)\n> \t\tcommit_list_sort_by_date(&revs->commits);\n\nLooks quite straight-forward.\n\nI however suspect that in the longer term it may be cleaner to get\nrid of REVSISION_WALK_NO_WALK_UNSORTED if we do this. The knob that\ncontrols if we sort the initial traversal tips and the knob that\ncontrols if we walk from these tips used to be tied to --no-walk\nonly because ca92e59e30b wanted to affect only no-walk case, but\nwith your new finding, it clearly is not limited to the no-walk case\nto want to avoid sorting.\n\n\n"},{"id":"431750","messageId":"xmqq5ywnws6v.fsf@gitster.g","threadId":"56018","inReplyTo":"b9897e102afbcab3bfee58ed8bda24257d8b54fb.1627896460.git.ps@pks.im","subject":"Re: [PATCH v3 3/4] revision: avoid loading object headers multiple times","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-02T19:40:56Z","receivedAt":"2021-08-02T19:41:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> When loading references, we try to optimize loading of commits by using\n> the commit graph. To do so, we first need to determine whether the\n> object actually is a commit or not, which is why we always execute\n> `oid_object_info()` first. Like this, we'll unpack the object header of\n> each object first.\n>\n> This pattern can be quite inefficient in case many references point to\n> the same commit: if the object didn't end up in the cached objects, then\n> we'll repeatedly unpack the same object header, even if we've already\n> seen the object before.\n> ...\n> Assuming that in almost all repositories, most references will point to\n> either a tag or a commit, we'd have a modest increase in memory\n> consumption of about 12.5% here.\n\nI wonder if we can also say almost all repositories, the majority of\nrefs point at the same object. If that holds, this would certainly\nbe a win, but otherwise, it is not so clear.\n"},{"id":"431751","messageId":"xmqqwnp3vcow.fsf@gitster.g","threadId":"56018","inReplyTo":"f6fc2a5e6d94befa915fb59b6296ce3153820c13.1627896460.git.ps@pks.im","subject":"Re: [PATCH v3 4/4] revision: avoid hitting packfiles when commits are in commit-graph","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-02T20:01:03Z","receivedAt":"2021-08-02T20:01:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 3860a0d847..a81d5cebc0 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -864,6 +864,48 @@ static int fill_commit_in_graph(struct repository *r,\n> \treturn 1;\n> }\n\n\nDescribe return-value here. 0 for not-found, !0 for found?\n\n> +static int find_object_id_in_graph(const struct object_id *id, struct commit_graph *g, uint32_t *pos)\n> +{\n> +\tstruct commit_graph *cur_g = g;\n> +\tuint32_t lex_index;\n> +\n> +\twhile (cur_g && !bsearch_graph(cur_g, (struct object_id *)id, &lex_index))\n> +\t\tcur_g = cur_g->base_graph;\n> +\n> +\tif (cur_g) {\n> +\t\t*pos = lex_index + cur_g->num_commits_in_base;\n> +\t\treturn 1;\n> +\t}\n> +\n> +\treturn 0;\n> +}\n\nLikewise, or as this is public, perhaps in commit-graph.h next to\nits declaration.\n\n> +int find_object_in_graph(struct repository *repo, struct object *object)\n> +{\n> +\tstruct commit *commit;\n> +\tuint32_t pos;\n> +\n> +\tif (object->parsed) {\n> +\t\tif (object->type != OBJ_COMMIT)\n> +\t\t\treturn -1;\n> +\t\treturn 0;\n\nThis is puzzling---at least it is not consistent with what the\nfunction name says (\"please say if you find _this_ object in the\ncommit-graph file\"---if that is not what this function does, it\nneeds a comment before the implementation).\n\nThe caller had object and we has already been parsed. If the\nfunction were \"with help from commit-graph, please tell me if you\ncan positively say this is a commit\", the above is understandable.\nIf we know positively that it is not commit, we say \"no, it is not a\ncommit\" (which may be suboptimal---if the caller falls back to\nanother codepath, the object will still not be a commit) and if we\nknow it is a commit, we can say \"yes, it definitely is a commit\" and\nthe caller can stop there.\n\nI guess my only problem with this function is that its name and what\nit does does not align. If the caller uses it for the real purpose\nof the function I guessed, then the logic itself may be OK.\n\n> +\t}\n> +\n> +\tif (!repo->objects->commit_graph)\n> +\t\treturn -1;\n\nThere is no commit-graph, then we decline to make a decision, which\nmakes sense.\n\n> +\tif (!find_object_id_in_graph(&object->oid, repo->objects->commit_graph, &pos))\n> +\t\treturn -1;\n\nIf it does not exist in the graph, we cannot tell, either.\n\n> +\tcommit = object_as_type(object, OBJ_COMMIT, 1);\n> +\tif (!commit)\n> +\t\treturn -1;\n> +\tif (!fill_commit_in_graph(repo, commit, repo->objects->commit_graph, pos))\n> +\t\treturn -1;\n> +\n> +\treturn 0;\n> +}\n> +\n> static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t *pos)\n> {\n> \tuint32_t graph_pos = commit_graph_position(item);\n> @@ -871,18 +913,7 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n> \t\t*pos = graph_pos;\n> \t\treturn 1;\n> \t} else {\n> -\t\tstruct commit_graph *cur_g = g;\n> -\t\tuint32_t lex_index;\n> -\n> -\t\twhile (cur_g && !bsearch_graph(cur_g, &(item->object.oid), &lex_index))\n> -\t\t\tcur_g = cur_g->base_graph;\n> -\n> -\t\tif (cur_g) {\n> -\t\t\t*pos = lex_index + cur_g->num_commits_in_base;\n> -\t\t\treturn 1;\n> -\t\t}\n> -\n> -\t\treturn 0;\n> +\t\treturn find_object_id_in_graph(&item->object.oid, g, pos);\n\nAnd I think this one is a op-op refactoring that does not change the\nbehaviour of find_commit_in_graph()? It might be easier if done in\na separate preparatory step, but it is small enough.\n\n> diff --git a/commit-graph.h b/commit-graph.h\n> index 96c24fb577..f373fab4c0 100644\n> --- a/commit-graph.h\n> +++ b/commit-graph.h\n> @@ -139,6 +139,8 @@ int write_commit_graph(struct object_directory *odb,\n> \t\t enum commit_graph_write_flags flags,\n> \t\t const struct commit_graph_opts *opts);\n> \n> +int find_object_in_graph(struct repository *repo, struct object *object);\n> +\n> #define COMMIT_GRAPH_VERIFY_SHALLOW\t(1 << 0)\n> \n> int verify_commit_graph(struct repository *r, struct commit_graph *g, int flags);\n> diff --git a/revision.c b/revision.c\n> index 671b6d6513..c3f9cf2998 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -362,10 +362,12 @@ static struct object *get_reference(struct rev_info *revs, const char *name,\n> \tstruct object *object = lookup_unknown_object(revs->repo, oid);\n> \n> \tif (object->type == OBJ_NONE) {\n> -\t\tint type = oid_object_info(revs->repo, oid, NULL);\n> -\t\tif (type < 0 || !object_as_type(object, type, 1)) {\n> -\t\t\tobject = NULL;\n> -\t\t\tgoto out;\n> +\t\tif (find_object_in_graph(revs->repo, object) < 0) {\n> +\t\t\tint type = oid_object_info(revs->repo, oid, NULL);\n> +\t\t\tif (type < 0 || !object_as_type(object, type, 1)) {\n> +\t\t\t\tobject = NULL;\n> +\t\t\t\tgoto out;\n> +\t\t\t}\n> \t\t}\n> \t}\n"},{"id":"431804","messageId":"YQkDQY9Qyqi7FCg6@ncase","threadId":"56018","inReplyTo":"871r7covsa.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 1/4] connected: do not sort input revisions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-03T08:50:09Z","receivedAt":"2021-08-03T08:50:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Aug 02, 2021 at 02:49:29PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> On Mon, Aug 02 2021, Patrick Steinhardt wrote:\n[snip]\n> > Introduce a new \"--unsorted-input\" flag to git-rev-list(1) which will\n> > cause it to not sort the commits and adjust the connectivity check to\n> > always pass the flag. This results in the following speedups, executed\n> > in a clone of gitlab-org/gitlab [1]:\n> >\n> > Benchmark #1: git rev-list --objects --quiet --not --all --not $(cat newrev)\n> > Time (mean ± σ): 7.639 s ± 0.065 s [User: 7.304 s, System: 0.335 s]\n> > Range (min … max): 7.543 s … 7.742 s 10 runs\n> >\n> > Benchmark #2: git rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n> > Time (mean ± σ): 4.995 s ± 0.044 s [User: 4.657 s, System: 0.337 s]\n> > Range (min … max): 4.909 s … 5.048 s 10 runs\n> >\n> > Summary\n> > 'git rev-list --unsorted-input --objects --quiet --not --all --not $(cat newrev)' ran\n> > 1.53 ± 0.02 times faster than 'git rev-list --objects --quiet --not --all --not $newrev'\n> \n> Just bikeshedding for a potential re-roll, perhaps --unordered-input, so\n> that it matches/rhymes with the existing \"git cat-file --unordered\",\n> which serves the same conceptual purpose (except this one's input, that\n> one's output).\n\nYeah, I wasn't quite sure how to name it myself either. Internally, we\ntypically use \"unsorted\" instead of \"unordered\", and there's also the\npreexisting \"--no-walk=(sorted|unsorted)\" flag for git-rev-list(1). With\nthe latter in mind, I think that \"unsorted\" fits a bit better given that\nwe already use it in the same command, but I don't particularly mind.\n\nFor now, I'll keep \"--unsorted-input\", but please feel free to give me\nanother shout if you disagree with my reasoning.\n\nPatrick\n"},{"id":"431805","messageId":"YQkEdYi/7vrS9ydC@ncase","threadId":"56018","inReplyTo":"xmqqa6lzwu31.fsf@gitster.g","subject":"Re: [PATCH v3 1/4] connected: do not sort input revisions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-03T08:55:17Z","receivedAt":"2021-08-03T08:55:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Aug 02, 2021 at 12:00:02PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n[snip]\n> > Introduce a new \"--unsorted-input\" flag to git-rev-list(1) which will\n> > cause it to not sort the commits and adjust the connectivity check to\n> > always pass the flag. This results in the following speedups, executed\n> > in a clone of gitlab-org/gitlab [1]:\n> > ...\n> > [1]: https://gitlab.com/gitlab-org/gitlab.git. Note that not all refs\n> > are visible to clients.\n> \n> So is this the 2.2 million refs thing?\n\nYeah, it is. The repo itself got 2.2 million refs, even though only 800k\nare publicly visible. It got even more when one considers its alternate,\nwhere it grows to 3.4 million in total.\n\n> > @@ -3584,7 +3586,7 @@ int prepare_revision_walk(struct rev_info *revs)\n> > \n> > \tif (!revs->reflog_info)\n> > \t\tprepare_to_use_bloom_filter(revs);\n> > -\tif (revs->no_walk != REVISION_WALK_NO_WALK_UNSORTED)\n> > +\tif (revs->no_walk != REVISION_WALK_NO_WALK_UNSORTED && !revs->unsorted_input)\n> > \t\tcommit_list_sort_by_date(&revs->commits);\n> \n> Looks quite straight-forward.\n> \n> I however suspect that in the longer term it may be cleaner to get\n> rid of REVSISION_WALK_NO_WALK_UNSORTED if we do this. The knob that\n> controls if we sort the initial traversal tips and the knob that\n> controls if we walk from these tips used to be tied to --no-walk\n> only because ca92e59e30b wanted to affect only no-walk case, but\n> with your new finding, it clearly is not limited to the no-walk case\n> to want to avoid sorting.\n\nRight. The question also is what to do when the user calls `git rev-list\n--no-walk=sorted --unsorted-input`. Do we sort? Don't we? Should we mark\nthese options as incompatible with each other and bail out? I guess just\nbailing out would be the easiest solution for now.\n\nPatrick\n"},{"id":"431806","messageId":"YQkHUZKZ6gkFkaD0@ncase","threadId":"56018","inReplyTo":"xmqq5ywnws6v.fsf@gitster.g","subject":"Re: [PATCH v3 3/4] revision: avoid loading object headers multiple times","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-03T09:07:29Z","receivedAt":"2021-08-03T09:07:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Aug 02, 2021 at 12:40:56PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > When loading references, we try to optimize loading of commits by using\n> > the commit graph. To do so, we first need to determine whether the\n> > object actually is a commit or not, which is why we always execute\n> > `oid_object_info()` first. Like this, we'll unpack the object header of\n> > each object first.\n> >\n> > This pattern can be quite inefficient in case many references point to\n> > the same commit: if the object didn't end up in the cached objects, then\n> > we'll repeatedly unpack the same object header, even if we've already\n> > seen the object before.\n> > ...\n> > Assuming that in almost all repositories, most references will point to\n> > either a tag or a commit, we'd have a modest increase in memory\n> > consumption of about 12.5% here.\n> \n> I wonder if we can also say almost all repositories, the majority of\n> refs point at the same object. If that holds, this would certainly\n> be a win, but otherwise, it is not so clear.\n\nI doubt that's the case in general. I rather assume that it's typically\ngoing to be a smallish subset that points to the same commit, but for\nthese cases we at least avoid doing the lookup multiple times. As I\nsaid, it's definitely a tradeoff between memory and performance: in the\nworst case (all references point to different blobs) we allocate 33%\nmore memory without having any speedups. A more realistic scenario would\nprobably be something like a trunk-based development repo, where there's\na single branch only and the rest is tags. There we'd allocate 11% more\nmemory without any speedups. In general, it's going to be various shades\nof gray, where we allocate something from 0% to 11% more memory while\ngetting some modest speedups in some cases.\n\nSo if we only inspect this commit as a standalone it's definitely\ndebatable whether we'd want to take it or not. But one important thing\nis that it's a prerequisite for patch 4/4: in order to not parse commits\nin case they're part of the commit-graph, we need to first obtain an\nobject such that we can fill it in via the graph. So we have to call\n`lookup_unknown_object()` anyway. Might be sensible to document this as\npart of the commit message.\n\nPatrick\n"},{"id":"431807","messageId":"YQkJdDvRtyOPzszU@ncase","threadId":"56018","inReplyTo":"xmqqwnp3vcow.fsf@gitster.g","subject":"Re: [PATCH v3 4/4] revision: avoid hitting packfiles when commits are in commit-graph","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-03T09:16:36Z","receivedAt":"2021-08-03T09:16:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Aug 02, 2021 at 01:01:03PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n[snip]\n> > +int find_object_in_graph(struct repository *repo, struct object *object)\n> > +{\n> > +\tstruct commit *commit;\n> > +\tuint32_t pos;\n> > +\n> > +\tif (object->parsed) {\n> > +\t\tif (object->type != OBJ_COMMIT)\n> > +\t\t\treturn -1;\n> > +\t\treturn 0;\n> \n> This is puzzling---at least it is not consistent with what the\n> function name says (\"please say if you find _this_ object in the\n> commit-graph file\"---if that is not what this function does, it\n> needs a comment before the implementation).\n> \n> The caller had object and we has already been parsed. If the\n> function were \"with help from commit-graph, please tell me if you\n> can positively say this is a commit\", the above is understandable.\n> If we know positively that it is not commit, we say \"no, it is not a\n> commit\" (which may be suboptimal---if the caller falls back to\n> another codepath, the object will still not be a commit) and if we\n> know it is a commit, we can say \"yes, it definitely is a commit\" and\n> the caller can stop there.\n> \n> I guess my only problem with this function is that its name and what\n> it does does not align. If the caller uses it for the real purpose\n> of the function I guessed, then the logic itself may be OK.\n\nFair point. The only caller for now only calls the function if the\nobject's type is unknown, so it really is \"Resolve the commit if it is\none\". I'll adjust the function's name.\n\n> > static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t *pos)\n> > {\n> > \tuint32_t graph_pos = commit_graph_position(item);\n> > @@ -871,18 +913,7 @@ static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n> > \t\t*pos = graph_pos;\n> > \t\treturn 1;\n> > \t} else {\n> > -\t\tstruct commit_graph *cur_g = g;\n> > -\t\tuint32_t lex_index;\n> > -\n> > -\t\twhile (cur_g && !bsearch_graph(cur_g, &(item->object.oid), &lex_index))\n> > -\t\t\tcur_g = cur_g->base_graph;\n> > -\n> > -\t\tif (cur_g) {\n> > -\t\t\t*pos = lex_index + cur_g->num_commits_in_base;\n> > -\t\t\treturn 1;\n> > -\t\t}\n> > -\n> > -\t\treturn 0;\n> > +\t\treturn find_object_id_in_graph(&item->object.oid, g, pos);\n> \n> And I think this one is a op-op refactoring that does not change the\n> behaviour of find_commit_in_graph()? It might be easier if done in\n> a separate preparatory step, but it is small enough.\n\nWill do.\n\nOne thing that occurred to me this morning after waking up is that this\ncommit changes semantics: if we're able to look up the commit via the\ncommit-graph, then we'll happily consider it to exist in the repository.\nBut given that we don't hit the object database at all anymore, it may\nbe that the commit-graph was out of date while the commit got\nunreachable and thus pruned. So it may not even exist anymore in the\nrepository.\n\nI wonder what our stance on this is. I can definitely understand the\nangle that this would be a deal breaker given that we now claim commits\nexist which don't anymore. On the other hand, we update commit-graphs\nvia git-gc(1), which makes this scenario a lot less likely nowadays. Is\nthere any precedent in our codebase where we treat commits part of the\ncommit-graph as existing? If not, do we want to make that assumption?\n\nPatrick\n"},{"id":"431886","messageId":"xmqqr1fatd3h.fsf@gitster.g","threadId":"56018","inReplyTo":"YQkEdYi/7vrS9ydC@ncase","subject":"Re: [PATCH v3 1/4] connected: do not sort input revisions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-03T21:47:30Z","receivedAt":"2021-08-03T21:47:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> I however suspect that in the longer term it may be cleaner to get\n>> rid of REVSISION_WALK_NO_WALK_UNSORTED if we do this. The knob that\n>> controls if we sort the initial traversal tips and the knob that\n>> controls if we walk from these tips used to be tied to --no-walk\n>> only because ca92e59e30b wanted to affect only no-walk case, but\n>> with your new finding, it clearly is not limited to the no-walk case\n>> to want to avoid sorting.\n>\n> Right. The question also is what to do when the user calls `git rev-list\n> --no-walk=sorted --unsorted-input`. Do we sort? Don't we? Should we mark\n> these options as incompatible with each other and bail out? I guess just\n> bailing out would be the easiest solution for now.\n\nI'd say so. Even without the future clean-up to get rid of the\nNO_WALK_UNSORTED, the issue already exists with this series, and\nwhen in doubt, it is easiest to start tight and take our time to\nfigure out what the right behaviour should be while we initially\ndo not allow both to be used at the same time.\n"},{"id":"431887","messageId":"xmqqmtpytcny.fsf@gitster.g","threadId":"56018","inReplyTo":"YQkJdDvRtyOPzszU@ncase","subject":"Re: [PATCH v3 4/4] revision: avoid hitting packfiles when commits are in commit-graph","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-03T21:56:49Z","receivedAt":"2021-08-03T21:56:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> I wonder what our stance on this is. I can definitely understand the\n> angle that this would be a deal breaker given that we now claim commits\n> exist which don't anymore.\n\nAn optimization that produces a wrong result very fast is a useless\noptimization that has no place in our codebase. But don't we have\nsome clue recorded in the commit graph file that tells us with what\npackfile the graph is to be used (iow, if the named packfile still\nexists there, the objects recorded in the graph file are to be found\nthere) or something?\n"},{"id":"431950","messageId":"87tuk5mq4h.fsf@evledraar.gmail.com","threadId":"56018","inReplyTo":"YQkJdDvRtyOPzszU@ncase","subject":"Re: [PATCH v3 4/4] revision: avoid hitting packfiles when commits are in commit-graph","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-04T10:51:18Z","receivedAt":"2021-08-04T11:00:21Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Aug 03 2021, Patrick Steinhardt wrote:\n\n> I wonder what our stance on this is. I can definitely understand the\n> angle that this would be a deal breaker given that we now claim commits\n> exist which don't anymore. On the other hand, we update commit-graphs\n> via git-gc(1), which makes this scenario a lot less likely nowadays. Is\n> there any precedent in our codebase where we treat commits part of the\n> commit-graph as existing? If not, do we want to make that assumption?\n\nI don't think there is, but don't see why given the performance benefits\nit should not at least be exposed in this form for those that think they\nknow what they're doing.\n\nBut right now the way we write the commit graph seems guaranteed to\nproduce races in this area, i.e. we expire objects first, and then we\nwrite a new graph (see cmd_gc()). I.e. something like:\n\n 1. (Re)pack reachable objects\n 2. Prune unrechable and old\n 3. Write a new commit graph using discovered tips\n\nI don't see a good reason other than just that this needs some\nrefactoring to not instead do:\n\n 1. Discover reachable refs\n 2. (Re)pack reachable objects from those refs\n 3. Write a new commit graph using those ref tips, i.e. don't re-run\n for_each_ref() in write_commit_graph_reachable() but call\n write_commit_graph() with those OIDs directly.\n 4. Prune unreachable and old\n\nWhich I think would nicely get around this particular race condition,\nwhich also doesn't exist if you call \"git commit-graph write\", it's just\nif we write a new graph and then expire objects without being aware that\nthe graph needs updating.\n"},{"id":"431951","messageId":"87r1f9mq1j.fsf@evledraar.gmail.com","threadId":"56018","inReplyTo":"YQkDQY9Qyqi7FCg6@ncase","subject":"Re: [PATCH v3 1/4] connected: do not sort input revisions","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-08-04T11:01:19Z","receivedAt":"2021-08-04T11:02:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Aug 03 2021, Patrick Steinhardt wrote:\n\n> [[PGP Signed Part:Undecided]]\n> On Mon, Aug 02, 2021 at 02:49:29PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> On Mon, Aug 02 2021, Patrick Steinhardt wrote:\n> [snip]\n>> > Introduce a new \"--unsorted-input\" flag to git-rev-list(1) which will\n>> > cause it to not sort the commits and adjust the connectivity check to\n>> > always pass the flag. This results in the following speedups, executed\n>> > in a clone of gitlab-org/gitlab [1]:\n>> >\n>> > Benchmark #1: git rev-list --objects --quiet --not --all --not $(cat newrev)\n>> > Time (mean ± σ): 7.639 s ± 0.065 s [User: 7.304 s, System: 0.335 s]\n>> > Range (min … max): 7.543 s … 7.742 s 10 runs\n>> >\n>> > Benchmark #2: git rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n>> > Time (mean ± σ): 4.995 s ± 0.044 s [User: 4.657 s, System: 0.337 s]\n>> > Range (min … max): 4.909 s … 5.048 s 10 runs\n>> >\n>> > Summary\n>> > 'git rev-list --unsorted-input --objects --quiet --not --all --not $(cat newrev)' ran\n>> > 1.53 ± 0.02 times faster than 'git rev-list --objects --quiet --not --all --not $newrev'\n>> \n>> Just bikeshedding for a potential re-roll, perhaps --unordered-input, so\n>> that it matches/rhymes with the existing \"git cat-file --unordered\",\n>> which serves the same conceptual purpose (except this one's input, that\n>> one's output).\n>\n> Yeah, I wasn't quite sure how to name it myself either. Internally, we\n> typically use \"unsorted\" instead of \"unordered\", and there's also the\n> preexisting \"--no-walk=(sorted|unsorted)\" flag for git-rev-list(1). With\n> the latter in mind, I think that \"unsorted\" fits a bit better given that\n> we already use it in the same command, but I don't particularly mind.\n>\n> For now, I'll keep \"--unsorted-input\", but please feel free to give me\n> another shout if you disagree with my reasoning.\n\nSounds good. I didn't mean it as a strong suggestion, unsorted does\nindeed sounds better in this context, just a gentle poke to check if you\nknew about that similar option...\n\nThanks.\n"},{"id":"432076","messageId":"YQu5pyjcaye3P0N3@ncase","threadId":"56018","inReplyTo":"87v94onguw.fsf@evledraar.gmail.com","subject":"Re: [PATCH v3 3/4] revision: avoid loading object headers multiple times","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-05T10:12:55Z","receivedAt":"2021-08-05T10:13:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Aug 02, 2021 at 02:55:09PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> On Mon, Aug 02 2021, Patrick Steinhardt wrote:\n[snip]\n> > diff --git a/revision.c b/revision.c\n> > index f06a5d63a3..671b6d6513 100644\n> > --- a/revision.c\n> > +++ b/revision.c\n> > @@ -359,14 +359,22 @@ static struct object *get_reference(struct rev_info *revs, const char *name,\n> > \t\t\t\t const struct object_id *oid,\n> > \t\t\t\t unsigned int flags)\n> > {\n> > -\tstruct object *object;\n> > +\tstruct object *object = lookup_unknown_object(revs->repo, oid);\n> > +\n> > +\tif (object->type == OBJ_NONE) {\n> > +\t\tint type = oid_object_info(revs->repo, oid, NULL);\n> > +\t\tif (type < 0 || !object_as_type(object, type, 1)) {\n> \n> Let's s/int type/enum object_type, personally I think we should never do\n> \"type < 0\" either, and check OBJ_BAD explicitly, but I've seemingly lost\n> that discussion on-list before.\n> \n> But I think the consensus is that we should not do !type, but rather\n> type == OBJ_NONE.\n\n`oid_object_info()` does return an `int` though, so it feels kind of\nweird to me to stuff it into an enum right away. Furthermore, while\n`OBJ_BAD` does map to `-1`, the documentation of `oid_object_info()`\ncurrently asks for what I'm doing: \"\"\"returns enum object_type or\nnegative\"\"\".\n\nPatrick\n"},{"id":"432092","messageId":"YQvFGuf3Ba/5UNXk@ncase","threadId":"56018","inReplyTo":"xmqqmtpytcny.fsf@gitster.g","subject":"Re: [PATCH v3 4/4] revision: avoid hitting packfiles when commits are in commit-graph","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-05T11:01:46Z","receivedAt":"2021-08-05T11:01:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Aug 03, 2021 at 02:56:49PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > I wonder what our stance on this is. I can definitely understand the\n> > angle that this would be a deal breaker given that we now claim commits\n> > exist which don't anymore.\n> \n> An optimization that produces a wrong result very fast is a useless\n> optimization that has no place in our codebase. But don't we have\n> some clue recorded in the commit graph file that tells us with what\n> packfile the graph is to be used (iow, if the named packfile still\n> exists there, the objects recorded in the graph file are to be found\n> there) or something?\n\nUnfortunately, no. For bitmaps we have this information given that a\nbitmap is tied to a specific pack anyway. But for commit-graphs, the\nstory is different given that they don't really care about the packs per\nse, but only about the commits.\n\nI was briefly wondering whether we can somehow use generation numbers to\ncut off parsing some commits: given we have already observed a commit\nwith generation number N and we have determined that this commit's\nobject exists, and we now see a commit with generation number M with\nMrepo, oid)) {\n+\t\t\tdie(\"bad object %s\", name);\n \t\t}\n \t}\n\nWe assert that the object exists, but `repo_has_object_file()` won't try\nto unpack the object header given that we request no info about the\nobject. And because the object ID has been part of the commit-graph, we\nknow that it's a commit. It's a bit slower compared to the version where\nwe don't assert object existence, but still a lot faster compared to\nlooking up the object type via the ODB:\n\n Benchmark #1: HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $(cat newrev)\n Time (mean ± σ): 4.512 s ± 0.057 s [User: 4.131 s, System: 0.381 s]\n Range (min … max): 4.435 s … 4.632 s 10 runs\n\n Benchmark #2: without-existence rev-list --unsorted-input --objects --quiet --not --all --not $(cat newrev)\n Time (mean ± σ): 2.903 s ± 0.022 s [User: 2.533 s, System: 0.369 s]\n Range (min … max): 2.878 s … 2.954 s 10 runs\n\n Benchmark #3: with-existence: rev-list --unsorted-input --objects --quiet --not --all --not $(cat newrev)\n Time (mean ± σ): 3.071 s ± 0.014 s [User: 2.712 s, System: 0.358 s]\n Range (min … max): 3.050 s … 3.088 s 10 runs\n\n Summary\n 'without-existence: rev-list --unsorted-input --objects --quiet --not --all --not $(cat newrev)' ran\n 1.06 ± 0.01 times faster than 'with-existance: rev-list --unsorted-input --objects --quiet --not --all --not $(cat newrev)'\n 1.55 ± 0.02 times faster than 'HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $(cat newrev)'\n\nPatrick\n"},{"id":"432093","messageId":"cover.1628162156.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1627896460.git.ps@pks.im","subject":"[PATCH v4 0/6] Speed up connectivity checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-05T11:25:19Z","receivedAt":"2021-08-05T11:25:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is version 4 of my series to speed up connectivity checks. Given\nthat v3 has received positive feedback, I finally stuck to the approach\nand only have a bunch of iterative changes based on your feedback.\n\nChanges compared to v3:\n\n - Patch 1/6 is new and splits up revs->no_walk into revs->no_walk\n which now only indicates whether to walk and revs->unserted_input,\n which indicates whether to sort input.\n\n - Patch 2/6 got some documentation for the new `--unsorted-input`\n option. Furthermore, we refuse `--no-walk`/`--no-walk=sorted` and\n `--unsorted-input` if used together. I've also added some tests\n for the new option.\n\n - Patch 3/6 has an updated commit message, detailing that\n `add_pending_oid()` only is a thin wrapper around\n `add_pending_object()`.\n\n - Patch 4/6 has an update commit message, stating that it's a\n prerequisite for 6/6.\n\n - Patch 5/6 is new, splitting out the logic to find a commit ID in\n the commit graph as a prerequisite for 6/6.\n\n - Patch 6/6 now also verifies that commits parsed only via the\n commit-graph exist in the ODB. I've also renamed\n `find_object_in_graph()` to `parse_commit_in_graph_gently()` to\n better reflect what the function does.\n\nWith the added existence check in 6/6, the speedup is not as big as\nbefore (1.47x faster instead of 1.55x). But it's still very much worth\nit. In total, this patch series decreases `git rev-list --objects\n--unsorted --not --all --not $newrev` from 7.6s to 3.0s in my test\nrepository.\n\nThanks for all your feedback!\n\nPatrick\n\nPatrick Steinhardt (6):\n revision: separate walk and unsorted flags\n connected: do not sort input revisions\n revision: stop retrieving reference twice\n revision: avoid loading object headers multiple times\n commit-graph: split out function to search commit position\n revision: avoid hitting packfiles when commits are in commit-graph\n\n Documentation/rev-list-options.txt | 8 +++-\n builtin/log.c | 2 +-\n builtin/revert.c | 3 +-\n commit-graph.c | 75 +++++++++++++++++++++---------\n commit-graph.h | 7 +++\n connected.c | 1 +\n revision.c | 52 +++++++++++++++++----\n revision.h | 7 +--\n t/t6000-rev-list-misc.sh | 38 +++++++++++++++\n 9 files changed, 153 insertions(+), 40 deletions(-)\n\nRange-diff against v3:\n-: ---------- > 1: 67232910ac revision: separate walk and unsorted flags\n1: 1fd83f726a ! 2: 9d7f484907 connected: do not sort input revisions\n @@ Commit message\n \n Signed-off-by: Patrick Steinhardt \n \n + ## Documentation/rev-list-options.txt ##\n +@@ Documentation/rev-list-options.txt: list of the missing objects. Object IDs are prefixed with a ``?'' character.\n + \tobjects.\n + endif::git-rev-list[]\n + \n ++--unsorted-input::\n ++\tShow commits in the order they were given on the command line instead\n ++\tof sorting them in reverse chronological order by commit time. Cannot\n ++\tbe combined with `--no-walk` or `--no-walk=sorted`.\n ++\n + --no-walk[=(sorted|unsorted)]::\n + \tOnly show the given commits, but do not traverse their ancestors.\n + \tThis has no effect if a range is specified. If the argument\n +@@ Documentation/rev-list-options.txt: endif::git-rev-list[]\n + \tgiven on the command line. Otherwise (if `sorted` or no argument\n + \twas given), the commits are shown in reverse chronological order\n + \tby commit time.\n +-\tCannot be combined with `--graph`.\n ++\tCannot be combined with `--graph`. Cannot be combined with\n ++\t`--unsorted-input` if `sorted` or no argument was given.\n + \n + --do-walk::\n + \tOverrides a previous `--no-walk`.\n +\n ## connected.c ##\n @@ connected.c: int check_connected(oid_iterate_fn fn, void *cb_data,\n \tif (opt->progress)\n @@ revision.c: static int handle_revision_opt(struct rev_info *revs, int argc, cons\n \t\trevs->sort_order = REV_SORT_BY_AUTHOR_DATE;\n \t\trevs->topo_order = 1;\n +\t} else if (!strcmp(arg, \"--unsorted-input\")) {\n ++\t\tif (revs->no_walk && !revs->unsorted_input)\n ++\t\t\tdie(_(\"--unsorted-input is incompatible with --no-walk and --no-walk=sorted\"));\n +\t\trevs->unsorted_input = 1;\n \t} else if (!strcmp(arg, \"--early-output\")) {\n \t\trevs->early_output = 100;\n \t\trevs->topo_order = 1;\n -@@ revision.c: int prepare_revision_walk(struct rev_info *revs)\n - \n - \tif (!revs->reflog_info)\n - \t\tprepare_to_use_bloom_filter(revs);\n --\tif (revs->no_walk != REVISION_WALK_NO_WALK_UNSORTED)\n -+\tif (revs->no_walk != REVISION_WALK_NO_WALK_UNSORTED && !revs->unsorted_input)\n - \t\tcommit_list_sort_by_date(&revs->commits);\n - \tif (revs->no_walk)\n - \t\treturn 0;\n +@@ revision.c: static int handle_revision_pseudo_opt(const char *submodule,\n + \t} else if (!strcmp(arg, \"--not\")) {\n + \t\t*flags ^= UNINTERESTING | BOTTOM;\n + \t} else if (!strcmp(arg, \"--no-walk\")) {\n ++\t\tif (revs->unsorted_input)\n ++\t\t\tdie(_(\"--no-walk is incompatible with --no-walk=unsorted and --unsorted-input\"));\n + \t\trevs->no_walk = 1;\n + \t} else if (skip_prefix(arg, \"--no-walk=\", &optarg)) {\n + \t\t/*\n +@@ revision.c: static int handle_revision_pseudo_opt(const char *submodule,\n + \t\t * not allowed, since the argument is optional.\n + \t\t */\n + \t\trevs->no_walk = 1;\n +-\t\tif (!strcmp(optarg, \"sorted\"))\n ++\t\tif (!strcmp(optarg, \"sorted\")) {\n ++\t\t\tif (revs->unsorted_input)\n ++\t\t\t\tdie(_(\"--no-walk=sorted is incompatible with --no-walk=unsorted \"\n ++\t\t\t\t \"and --unsorted-input\"));\n + \t\t\trevs->unsorted_input = 0;\n +-\t\telse if (!strcmp(optarg, \"unsorted\"))\n ++\t\t} else if (!strcmp(optarg, \"unsorted\"))\n + \t\t\trevs->unsorted_input = 1;\n + \t\telse\n + \t\t\treturn error(\"invalid argument to --no-walk\");\n \n - ## revision.h ##\n -@@ revision.h: struct rev_info {\n - \t\t\tsimplify_history:1,\n - \t\t\tshow_pulls:1,\n - \t\t\ttopo_order:1,\n -+\t\t\tunsorted_input:1,\n - \t\t\tsimplify_merges:1,\n - \t\t\tsimplify_by_decoration:1,\n - \t\t\tsingle_worktree:1,\n + ## t/t6000-rev-list-misc.sh ##\n +@@ t/t6000-rev-list-misc.sh: test_expect_success 'rev-list --count --objects' '\n + \ttest_line_count = $count actual\n + '\n + \n ++test_expect_success 'rev-list --unsorted-input results in different sorting' '\n ++\tgit rev-list --unsorted-input HEAD HEAD~ >first &&\n ++\tgit rev-list --unsorted-input HEAD~ HEAD >second &&\n ++\t! test_cmp first second &&\n ++\tsort first >first.sorted &&\n ++\tsort second >second.sorted &&\n ++\ttest_cmp first.sorted second.sorted\n ++'\n ++\n ++test_expect_success 'rev-list --unsorted-input compatible with --no-walk=unsorted' '\n ++\tgit rev-list --unsorted-input --no-walk=unsorted HEAD HEAD~ >actual &&\n ++\tgit rev-parse HEAD >expect &&\n ++\tgit rev-parse HEAD~ >>expect &&\n ++\ttest_cmp expect actual\n ++'\n ++\n ++test_expect_success 'rev-list --unsorted-input incompatible with --no-walk=sorted' '\n ++\tcat >expect <<-EOF &&\n ++\t\tfatal: --no-walk is incompatible with --no-walk=unsorted and --unsorted-input\n ++\tEOF\n ++\ttest_must_fail git rev-list --unsorted-input --no-walk HEAD 2>error &&\n ++\ttest_cmp expect error &&\n ++\n ++\tcat >expect <<-EOF &&\n ++\t\tfatal: --no-walk=sorted is incompatible with --no-walk=unsorted and --unsorted-input\n ++\tEOF\n ++\ttest_must_fail git rev-list --unsorted-input --no-walk=sorted HEAD 2>error &&\n ++\ttest_cmp expect error &&\n ++\n ++\tcat >expect <<-EOF &&\n ++\t\tfatal: --unsorted-input is incompatible with --no-walk and --no-walk=sorted\n ++\tEOF\n ++\ttest_must_fail git rev-list --no-walk --unsorted-input HEAD 2>error &&\n ++\ttest_cmp expect error &&\n ++\ttest_must_fail git rev-list --no-walk=sorted --unsorted-input HEAD 2>error &&\n ++\ttest_cmp expect error\n ++'\n ++\n + test_done\n2: db85480649 ! 3: d8e63d0943 revision: stop retrieving reference twice\n @@ Commit message\n revision: stop retrieving reference twice\n \n When queueing up references for the revision walk, `handle_one_ref()`\n - will resolve the reference's object ID and then queue the ID as pending\n - object via `add_pending_oid()`. But `add_pending_oid()` will again try\n - to resolve the object ID to an object, effectively duplicating the work\n - its caller already did before.\n + will resolve the reference's object ID via `get_reference()` and then\n + queue the ID as pending object via `add_pending_oid()`. But given that\n + `add_pending_oid()` is only a thin wrapper around `add_pending_object()`\n + which fist calls `get_reference()`, we effectively resolve the reference\n + twice and thus duplicate some of the work.\n \n - Fix the issue by instead calling `add_pending_object()`, which takes the\n - already-resolved object as input. In a repository with lots of refs,\n - this translates in a nearly 10% speedup:\n + Fix the issue by instead calling `add_pending_object()` directly, which\n + takes the already-resolved object as input. In a repository with lots of\n + refs, this translates into a near 10% speedup:\n \n Benchmark #1: HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n Time (mean ± σ): 5.015 s ± 0.038 s [User: 4.698 s, System: 0.316 s]\n3: b9897e102a ! 4: ba8df5cad0 revision: avoid loading object headers multiple times\n @@ Commit message\n either a tag or a commit, we'd have a modest increase in memory\n consumption of about 12.5% here.\n \n + Note that on its own, this patch may not seem like a clear win. But it\n + is a prerequisite for the following patch, which will result in another\n + 37% speedup.\n +\n Signed-off-by: Patrick Steinhardt \n \n ## revision.c ##\n-: ---------- > 5: e33cd51ebf commit-graph: split out function to search commit position\n4: f6fc2a5e6d ! 6: 900c5a9c60 revision: avoid hitting packfiles when commits are in commit-graph\n @@ Commit message\n directly fill in the commit object, otherwise we can still hit the disk\n to determine the object's type.\n \n - Expose a new function `find_object_in_graph()`, which fills in an object\n - of unknown type in case we find its object ID in the graph. This\n - provides a big performance win in cases where there is a commit-graph\n - available in the repository in case we load lots of references. The\n - following has been executed in a real-world repository with about 2.2\n - million refs:\n + Expose a new function `parse_commit_in_graph_gently()`, which fills in\n + an object of unknown type in case we find its object ID in the graph.\n + This provides a big performance win in cases where there is a\n + commit-graph available in the repository in case we load lots of\n + references. The following has been executed in a real-world repository\n + with about 2.2 million refs:\n \n Benchmark #1: HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n - Time (mean ± σ): 4.465 s ± 0.037 s [User: 4.144 s, System: 0.320 s]\n - Range (min … max): 4.411 s … 4.514 s 10 runs\n + Time (mean ± σ): 4.508 s ± 0.039 s [User: 4.131 s, System: 0.377 s]\n + Range (min … max): 4.455 s … 4.576 s 10 runs\n \n Benchmark #2: HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n - Time (mean ± σ): 2.886 s ± 0.032 s [User: 2.570 s, System: 0.316 s]\n - Range (min … max): 2.826 s … 2.933 s 10 runs\n + Time (mean ± σ): 3.072 s ± 0.031 s [User: 2.707 s, System: 0.365 s]\n + Range (min … max): 3.040 s … 3.144 s 10 runs\n \n Summary\n 'HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev' ran\n - 1.55 ± 0.02 times faster than 'HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev'\n + 1.47 ± 0.02 times faster than 'HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev'\n \n Signed-off-by: Patrick Steinhardt \n \n ## commit-graph.c ##\n -@@ commit-graph.c: static int fill_commit_in_graph(struct repository *r,\n - \treturn 1;\n +@@ commit-graph.c: static int find_commit_pos_in_graph(struct commit *item, struct commit_graph *g,\n + \t}\n }\n \n -+static int find_object_id_in_graph(const struct object_id *id, struct commit_graph *g, uint32_t *pos)\n -+{\n -+\tstruct commit_graph *cur_g = g;\n -+\tuint32_t lex_index;\n -+\n -+\twhile (cur_g && !bsearch_graph(cur_g, (struct object_id *)id, &lex_index))\n -+\t\tcur_g = cur_g->base_graph;\n -+\n -+\tif (cur_g) {\n -+\t\t*pos = lex_index + cur_g->num_commits_in_base;\n -+\t\treturn 1;\n -+\t}\n -+\n -+\treturn 0;\n -+}\n -+\n -+int find_object_in_graph(struct repository *repo, struct object *object)\n ++int parse_commit_in_graph_gently(struct repository *repo, struct object *object)\n +{\n +\tstruct commit *commit;\n +\tuint32_t pos;\n @@ commit-graph.c: static int fill_commit_in_graph(struct repository *r,\n +\tif (!repo->objects->commit_graph)\n +\t\treturn -1;\n +\n -+\tif (!find_object_id_in_graph(&object->oid, repo->objects->commit_graph, &pos))\n ++\tif (!search_commit_pos_in_graph(&object->oid, repo->objects->commit_graph, &pos))\n +\t\treturn -1;\n +\n +\tcommit = object_as_type(object, OBJ_COMMIT, 1);\n @@ commit-graph.c: static int fill_commit_in_graph(struct repository *r,\n +\treturn 0;\n +}\n +\n - static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t *pos)\n - {\n - \tuint32_t graph_pos = commit_graph_position(item);\n -@@ commit-graph.c: static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uin\n - \t\t*pos = graph_pos;\n - \t\treturn 1;\n - \t} else {\n --\t\tstruct commit_graph *cur_g = g;\n --\t\tuint32_t lex_index;\n --\n --\t\twhile (cur_g && !bsearch_graph(cur_g, &(item->object.oid), &lex_index))\n --\t\t\tcur_g = cur_g->base_graph;\n --\n --\t\tif (cur_g) {\n --\t\t\t*pos = lex_index + cur_g->num_commits_in_base;\n --\t\t\treturn 1;\n --\t\t}\n --\n --\t\treturn 0;\n -+\t\treturn find_object_id_in_graph(&item->object.oid, g, pos);\n - \t}\n - }\n - \n + static int parse_commit_in_graph_one(struct repository *r,\n + \t\t\t\t struct commit_graph *g,\n + \t\t\t\t struct commit *item)\n \n ## commit-graph.h ##\n -@@ commit-graph.h: int write_commit_graph(struct object_directory *odb,\n - \t\t enum commit_graph_write_flags flags,\n - \t\t const struct commit_graph_opts *opts);\n +@@ commit-graph.h: int open_commit_graph(const char *graph_file, int *fd, struct stat *st);\n + */\n + int parse_commit_in_graph(struct repository *r, struct commit *item);\n \n -+int find_object_in_graph(struct repository *repo, struct object *object);\n ++/*\n ++ * Given an object of unknown type, try to fill in the object in case it is a\n ++ * commit part of the commit-graph. Returns 0 if the object is a parsed commit\n ++ * or if it could be filled in via the commit graph, otherwise it returns -1.\n ++ */\n ++int parse_commit_in_graph_gently(struct repository *repo, struct object *object);\n +\n - #define COMMIT_GRAPH_VERIFY_SHALLOW\t(1 << 0)\n - \n - int verify_commit_graph(struct repository *r, struct commit_graph *g, int flags);\n + /*\n + * It is possible that we loaded commit contents from the commit buffer,\n + * but we also want to ensure the commit-graph content is correctly\n \n ## revision.c ##\n @@ revision.c: static struct object *get_reference(struct rev_info *revs, const char *name,\n @@ revision.c: static struct object *get_reference(struct rev_info *revs, const cha\n \tif (object->type == OBJ_NONE) {\n -\t\tint type = oid_object_info(revs->repo, oid, NULL);\n -\t\tif (type < 0 || !object_as_type(object, type, 1)) {\n --\t\t\tobject = NULL;\n --\t\t\tgoto out;\n -+\t\tif (find_object_in_graph(revs->repo, object) < 0) {\n ++\t\t/*\n ++\t\t * It's likely that the reference points to a commit, so we\n ++\t\t * first try to look it up via the commit-graph. If successful,\n ++\t\t * then we know it's a commit and don't have to unpack the\n ++\t\t * object header. We still need to assert that the object\n ++\t\t * exists, but given that we don't request any info about the\n ++\t\t * object this is a lot faster than `oid_object_info()`.\n ++\t\t */\n ++\t\tif (parse_commit_in_graph_gently(revs->repo, object) < 0) {\n +\t\t\tint type = oid_object_info(revs->repo, oid, NULL);\n +\t\t\tif (type < 0 || !object_as_type(object, type, 1)) {\n +\t\t\t\tobject = NULL;\n +\t\t\t\tgoto out;\n +\t\t\t}\n ++\t\t} else if (!repo_has_object_file(revs->repo, oid)) {\n + \t\t\tobject = NULL;\n + \t\t\tgoto out;\n \t\t}\n - \t}\n - \n-- \n2.32.0\n\n"},{"id":"432094","messageId":"67232910acf4a248654060ad51a17af37acba0fb.1628162156.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1628162156.git.ps@pks.im","subject":"[PATCH v4 1/6] revision: separate walk and unsorted flags","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-05T11:25:24Z","receivedAt":"2021-08-05T11:25:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `--no-walk` flag supports two modes: either it sorts the revisions\ngiven as input input or it doesn't. This is reflected in a single\n`no_walk` flag, which reflects one of the three states \"walk\", \"don't\nwalk but without sorting\" and \"don't walk but with sorting\".\n\nSplit up the flag into two separate bits, one indicating whether we\nshould walk or not and one indicating whether the input should be sorted\nor not. This will allow us to more easily introduce a new flag\n`--unsorted-input`, which only impacts the sorting bit.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/log.c | 2 +-\n builtin/revert.c | 3 ++-\n revision.c | 9 +++++----\n revision.h | 7 ++-----\n 4 files changed, 10 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/log.c b/builtin/log.c\nindex 3d7717ba5c..f75d87e8d7 100644\n--- a/builtin/log.c\n+++ b/builtin/log.c\n@@ -637,7 +637,7 @@ int cmd_show(int argc, const char **argv, const char *prefix)\n \trepo_init_revisions(the_repository, &rev, prefix);\n \trev.diff = 1;\n \trev.always_show_header = 1;\n-\trev.no_walk = REVISION_WALK_NO_WALK_SORTED;\n+\trev.no_walk = 1;\n \trev.diffopt.stat_width = -1; \t/* Scale to real terminal size */\n \n \tmemset(&opt, 0, sizeof(opt));\ndiff --git a/builtin/revert.c b/builtin/revert.c\nindex 237f2f18d4..2e13660e4b 100644\n--- a/builtin/revert.c\n+++ b/builtin/revert.c\n@@ -191,7 +191,8 @@ static int run_sequencer(int argc, const char **argv, struct replay_opts *opts)\n \t\tstruct setup_revision_opt s_r_opt;\n \t\topts->revs = xmalloc(sizeof(*opts->revs));\n \t\trepo_init_revisions(the_repository, opts->revs, NULL);\n-\t\topts->revs->no_walk = REVISION_WALK_NO_WALK_UNSORTED;\n+\t\topts->revs->no_walk = 1;\n+\t\topts->revs->unsorted_input = 1;\n \t\tif (argc < 2)\n \t\t\tusage_with_options(usage_str, options);\n \t\tif (!strcmp(argv[1], \"-\"))\ndiff --git a/revision.c b/revision.c\nindex cddd0542a6..86bbcd10d2 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -2651,16 +2651,17 @@ static int handle_revision_pseudo_opt(const char *submodule,\n \t} else if (!strcmp(arg, \"--not\")) {\n \t\t*flags ^= UNINTERESTING | BOTTOM;\n \t} else if (!strcmp(arg, \"--no-walk\")) {\n-\t\trevs->no_walk = REVISION_WALK_NO_WALK_SORTED;\n+\t\trevs->no_walk = 1;\n \t} else if (skip_prefix(arg, \"--no-walk=\", &optarg)) {\n \t\t/*\n \t\t * Detached form (\"--no-walk X\" as opposed to \"--no-walk=X\")\n \t\t * not allowed, since the argument is optional.\n \t\t */\n+\t\trevs->no_walk = 1;\n \t\tif (!strcmp(optarg, \"sorted\"))\n-\t\t\trevs->no_walk = REVISION_WALK_NO_WALK_SORTED;\n+\t\t\trevs->unsorted_input = 0;\n \t\telse if (!strcmp(optarg, \"unsorted\"))\n-\t\t\trevs->no_walk = REVISION_WALK_NO_WALK_UNSORTED;\n+\t\t\trevs->unsorted_input = 1;\n \t\telse\n \t\t\treturn error(\"invalid argument to --no-walk\");\n \t} else if (!strcmp(arg, \"--do-walk\")) {\n@@ -3584,7 +3585,7 @@ int prepare_revision_walk(struct rev_info *revs)\n \n \tif (!revs->reflog_info)\n \t\tprepare_to_use_bloom_filter(revs);\n-\tif (revs->no_walk != REVISION_WALK_NO_WALK_UNSORTED)\n+\tif (!revs->unsorted_input)\n \t\tcommit_list_sort_by_date(&revs->commits);\n \tif (revs->no_walk)\n \t\treturn 0;\ndiff --git a/revision.h b/revision.h\nindex fbb068da9f..0c65a760ee 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -79,10 +79,6 @@ struct rev_cmdline_info {\n \t} *rev;\n };\n \n-#define REVISION_WALK_WALK 0\n-#define REVISION_WALK_NO_WALK_SORTED 1\n-#define REVISION_WALK_NO_WALK_UNSORTED 2\n-\n struct oidset;\n struct topo_walk_info;\n \n@@ -129,7 +125,8 @@ struct rev_info {\n \t/* Traversal flags */\n \tunsigned int\tdense:1,\n \t\t\tprune:1,\n-\t\t\tno_walk:2,\n+\t\t\tno_walk:1,\n+\t\t\tunsorted_input:1,\n \t\t\tremove_empty_trees:1,\n \t\t\tsimplify_history:1,\n \t\t\tshow_pulls:1,\n-- \n2.32.0\n\n"},{"id":"432095","messageId":"9d7f484907e2bd2492e6676238579e9f0c6ed374.1628162156.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1628162156.git.ps@pks.im","subject":"[PATCH v4 2/6] connected: do not sort input revisions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-05T11:25:28Z","receivedAt":"2021-08-05T11:25:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In order to compute whether objects reachable from a set of tips are all\nconnected, we do a revision walk with these tips as positive references\nand `--not --all`. `--not --all` will cause the revision walk to load\nall preexisting references as uninteresting, which can be very expensive\nin repositories with many references.\n\nBenchmarking the git-rev-list(1) command highlights that by far the most\nexpensive single phase is initial sorting of the input revisions: after\nall references have been loaded, we first sort commits by author date.\nIn a real-world repository with about 2.2 million references, it makes\nup about 40% of the total runtime of git-rev-list(1).\n\nUltimately, the connectivity check shouldn't really bother about the\norder of input revisions at all. We only care whether we can actually\nwalk all objects until we hit the cut-off point. So sorting the input is\na complete waste of time.\n\nIntroduce a new \"--unsorted-input\" flag to git-rev-list(1) which will\ncause it to not sort the commits and adjust the connectivity check to\nalways pass the flag. This results in the following speedups, executed\nin a clone of gitlab-org/gitlab [1]:\n\n Benchmark #1: git rev-list --objects --quiet --not --all --not $(cat newrev)\n Time (mean ± σ): 7.639 s ± 0.065 s [User: 7.304 s, System: 0.335 s]\n Range (min … max): 7.543 s … 7.742 s 10 runs\n\n Benchmark #2: git rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n Time (mean ± σ): 4.995 s ± 0.044 s [User: 4.657 s, System: 0.337 s]\n Range (min … max): 4.909 s … 5.048 s 10 runs\n\n Summary\n 'git rev-list --unsorted-input --objects --quiet --not --all --not $(cat newrev)' ran\n 1.53 ± 0.02 times faster than 'git rev-list --objects --quiet --not --all --not $newrev'\n\n[1]: https://gitlab.com/gitlab-org/gitlab.git. Note that not all refs\n are visible to clients.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/rev-list-options.txt | 8 ++++++-\n connected.c | 1 +\n revision.c | 13 ++++++++--\n t/t6000-rev-list-misc.sh | 38 ++++++++++++++++++++++++++++++\n 4 files changed, 57 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 24569b06d1..b7bd27e171 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -968,6 +968,11 @@ list of the missing objects. Object IDs are prefixed with a ``?'' character.\n \tobjects.\n endif::git-rev-list[]\n \n+--unsorted-input::\n+\tShow commits in the order they were given on the command line instead\n+\tof sorting them in reverse chronological order by commit time. Cannot\n+\tbe combined with `--no-walk` or `--no-walk=sorted`.\n+\n --no-walk[=(sorted|unsorted)]::\n \tOnly show the given commits, but do not traverse their ancestors.\n \tThis has no effect if a range is specified. If the argument\n@@ -975,7 +980,8 @@ endif::git-rev-list[]\n \tgiven on the command line. Otherwise (if `sorted` or no argument\n \twas given), the commits are shown in reverse chronological order\n \tby commit time.\n-\tCannot be combined with `--graph`.\n+\tCannot be combined with `--graph`. Cannot be combined with\n+\t`--unsorted-input` if `sorted` or no argument was given.\n \n --do-walk::\n \tOverrides a previous `--no-walk`.\ndiff --git a/connected.c b/connected.c\nindex b18299fdf0..b5f9523a5f 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -106,6 +106,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \tif (opt->progress)\n \t\tstrvec_pushf(&rev_list.args, \"--progress=%s\",\n \t\t\t _(\"Checking connectivity\"));\n+\tstrvec_push(&rev_list.args, \"--unsorted-input\");\n \n \trev_list.git_cmd = 1;\n \trev_list.env = opt->env;\ndiff --git a/revision.c b/revision.c\nindex 86bbcd10d2..793f76a509 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -2256,6 +2256,10 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg\n \t} else if (!strcmp(arg, \"--author-date-order\")) {\n \t\trevs->sort_order = REV_SORT_BY_AUTHOR_DATE;\n \t\trevs->topo_order = 1;\n+\t} else if (!strcmp(arg, \"--unsorted-input\")) {\n+\t\tif (revs->no_walk && !revs->unsorted_input)\n+\t\t\tdie(_(\"--unsorted-input is incompatible with --no-walk and --no-walk=sorted\"));\n+\t\trevs->unsorted_input = 1;\n \t} else if (!strcmp(arg, \"--early-output\")) {\n \t\trevs->early_output = 100;\n \t\trevs->topo_order = 1;\n@@ -2651,6 +2655,8 @@ static int handle_revision_pseudo_opt(const char *submodule,\n \t} else if (!strcmp(arg, \"--not\")) {\n \t\t*flags ^= UNINTERESTING | BOTTOM;\n \t} else if (!strcmp(arg, \"--no-walk\")) {\n+\t\tif (revs->unsorted_input)\n+\t\t\tdie(_(\"--no-walk is incompatible with --no-walk=unsorted and --unsorted-input\"));\n \t\trevs->no_walk = 1;\n \t} else if (skip_prefix(arg, \"--no-walk=\", &optarg)) {\n \t\t/*\n@@ -2658,9 +2664,12 @@ static int handle_revision_pseudo_opt(const char *submodule,\n \t\t * not allowed, since the argument is optional.\n \t\t */\n \t\trevs->no_walk = 1;\n-\t\tif (!strcmp(optarg, \"sorted\"))\n+\t\tif (!strcmp(optarg, \"sorted\")) {\n+\t\t\tif (revs->unsorted_input)\n+\t\t\t\tdie(_(\"--no-walk=sorted is incompatible with --no-walk=unsorted \"\n+\t\t\t\t \"and --unsorted-input\"));\n \t\t\trevs->unsorted_input = 0;\n-\t\telse if (!strcmp(optarg, \"unsorted\"))\n+\t\t} else if (!strcmp(optarg, \"unsorted\"))\n \t\t\trevs->unsorted_input = 1;\n \t\telse\n \t\t\treturn error(\"invalid argument to --no-walk\");\ndiff --git a/t/t6000-rev-list-misc.sh b/t/t6000-rev-list-misc.sh\nindex 12def7bcbf..8e213eb413 100755\n--- a/t/t6000-rev-list-misc.sh\n+++ b/t/t6000-rev-list-misc.sh\n@@ -169,4 +169,42 @@ test_expect_success 'rev-list --count --objects' '\n \ttest_line_count = $count actual\n '\n \n+test_expect_success 'rev-list --unsorted-input results in different sorting' '\n+\tgit rev-list --unsorted-input HEAD HEAD~ >first &&\n+\tgit rev-list --unsorted-input HEAD~ HEAD >second &&\n+\t! test_cmp first second &&\n+\tsort first >first.sorted &&\n+\tsort second >second.sorted &&\n+\ttest_cmp first.sorted second.sorted\n+'\n+\n+test_expect_success 'rev-list --unsorted-input compatible with --no-walk=unsorted' '\n+\tgit rev-list --unsorted-input --no-walk=unsorted HEAD HEAD~ >actual &&\n+\tgit rev-parse HEAD >expect &&\n+\tgit rev-parse HEAD~ >>expect &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'rev-list --unsorted-input incompatible with --no-walk=sorted' '\n+\tcat >expect <<-EOF &&\n+\t\tfatal: --no-walk is incompatible with --no-walk=unsorted and --unsorted-input\n+\tEOF\n+\ttest_must_fail git rev-list --unsorted-input --no-walk HEAD 2>error &&\n+\ttest_cmp expect error &&\n+\n+\tcat >expect <<-EOF &&\n+\t\tfatal: --no-walk=sorted is incompatible with --no-walk=unsorted and --unsorted-input\n+\tEOF\n+\ttest_must_fail git rev-list --unsorted-input --no-walk=sorted HEAD 2>error &&\n+\ttest_cmp expect error &&\n+\n+\tcat >expect <<-EOF &&\n+\t\tfatal: --unsorted-input is incompatible with --no-walk and --no-walk=sorted\n+\tEOF\n+\ttest_must_fail git rev-list --no-walk --unsorted-input HEAD 2>error &&\n+\ttest_cmp expect error &&\n+\ttest_must_fail git rev-list --no-walk=sorted --unsorted-input HEAD 2>error &&\n+\ttest_cmp expect error\n+'\n+\n test_done\n-- \n2.32.0\n\n"},{"id":"432096","messageId":"d8e63d0943bbe3829a53561a3a1cb4f9e6963337.1628162156.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1628162156.git.ps@pks.im","subject":"[PATCH v4 3/6] revision: stop retrieving reference twice","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-05T11:25:32Z","receivedAt":"2021-08-05T11:25:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When queueing up references for the revision walk, `handle_one_ref()`\nwill resolve the reference's object ID via `get_reference()` and then\nqueue the ID as pending object via `add_pending_oid()`. But given that\n`add_pending_oid()` is only a thin wrapper around `add_pending_object()`\nwhich fist calls `get_reference()`, we effectively resolve the reference\ntwice and thus duplicate some of the work.\n\nFix the issue by instead calling `add_pending_object()` directly, which\ntakes the already-resolved object as input. In a repository with lots of\nrefs, this translates into a near 10% speedup:\n\n Benchmark #1: HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n Time (mean ± σ): 5.015 s ± 0.038 s [User: 4.698 s, System: 0.316 s]\n Range (min … max): 4.970 s … 5.089 s 10 runs\n\n Benchmark #2: HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n Time (mean ± σ): 4.606 s ± 0.029 s [User: 4.260 s, System: 0.345 s]\n Range (min … max): 4.565 s … 4.657 s 10 runs\n\n Summary\n 'HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev' ran\n 1.09 ± 0.01 times faster than 'HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev'\n\nSigned-off-by: Patrick Steinhardt \n---\n revision.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/revision.c b/revision.c\nindex 793f76a509..0d99413856 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1534,7 +1534,7 @@ static int handle_one_ref(const char *path, const struct object_id *oid,\n \n \tobject = get_reference(cb->all_revs, path, oid, cb->all_flags);\n \tadd_rev_cmdline(cb->all_revs, object, path, REV_CMD_REF, cb->all_flags);\n-\tadd_pending_oid(cb->all_revs, path, oid, cb->all_flags);\n+\tadd_pending_object(cb->all_revs, object, path);\n \treturn 0;\n }\n \n-- \n2.32.0\n\n"},{"id":"432097","messageId":"ba8df5cad00c5dace1d3abfbd563197dbb6b4b32.1628162156.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1628162156.git.ps@pks.im","subject":"[PATCH v4 4/6] revision: avoid loading object headers multiple times","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-05T11:25:36Z","receivedAt":"2021-08-05T11:25:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When loading references, we try to optimize loading of commits by using\nthe commit graph. To do so, we first need to determine whether the\nobject actually is a commit or not, which is why we always execute\n`oid_object_info()` first. Like this, we'll unpack the object header of\neach object first.\n\nThis pattern can be quite inefficient in case many references point to\nthe same commit: if the object didn't end up in the cached objects, then\nwe'll repeatedly unpack the same object header, even if we've already\nseen the object before.\n\nOptimize this pattern by using `lookup_unknown_object()` first in order\nto determine whether we've seen the object before. If so, then we don't\nneed to re-parse the header but can directly use its object information\nand thus gain a modest performance improvement. Executed in a real-world\nrepository with around 2.2 million references:\n\n Benchmark #1: HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n Time (mean ± σ): 4.771 s ± 0.238 s [User: 4.440 s, System: 0.330 s]\n Range (min … max): 4.539 s … 5.219 s 10 runs\n\n Benchmark #2: HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n Time (mean ± σ): 4.454 s ± 0.037 s [User: 4.122 s, System: 0.332 s]\n Range (min … max): 4.375 s … 4.496 s 10 runs\n\n Summary\n 'HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev' ran\n 1.07 ± 0.05 times faster than 'HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev'\n\nThe downside is that `lookup_unknown_object()` is forced to always\nallocate an object such that it's big enough to host all object types'\nstructs and thus we may waste memory here. This tradeoff is probably\nworth it though considering the following struct sizes:\n\n - commit: 72 bytes\n - tree: 56 bytes\n - blob: 40 bytes\n - tag: 64 bytes\n\nAssuming that in almost all repositories, most references will point to\neither a tag or a commit, we'd have a modest increase in memory\nconsumption of about 12.5% here.\n\nNote that on its own, this patch may not seem like a clear win. But it\nis a prerequisite for the following patch, which will result in another\n37% speedup.\n\nSigned-off-by: Patrick Steinhardt \n---\n revision.c | 15 ++++++++++++---\n 1 file changed, 12 insertions(+), 3 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex 0d99413856..25f4784fdd 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -359,14 +359,22 @@ static struct object *get_reference(struct rev_info *revs, const char *name,\n \t\t\t\t const struct object_id *oid,\n \t\t\t\t unsigned int flags)\n {\n-\tstruct object *object;\n+\tstruct object *object = lookup_unknown_object(revs->repo, oid);\n+\n+\tif (object->type == OBJ_NONE) {\n+\t\tint type = oid_object_info(revs->repo, oid, NULL);\n+\t\tif (type < 0 || !object_as_type(object, type, 1)) {\n+\t\t\tobject = NULL;\n+\t\t\tgoto out;\n+\t\t}\n+\t}\n \n \t/*\n \t * If the repository has commit graphs, repo_parse_commit() avoids\n \t * reading the object buffer, so use it whenever possible.\n \t */\n-\tif (oid_object_info(revs->repo, oid, NULL) == OBJ_COMMIT) {\n-\t\tstruct commit *c = lookup_commit(revs->repo, oid);\n+\tif (object->type == OBJ_COMMIT) {\n+\t\tstruct commit *c = (struct commit *) object;\n \t\tif (!repo_parse_commit(revs->repo, c))\n \t\t\tobject = (struct object *) c;\n \t\telse\n@@ -375,6 +383,7 @@ static struct object *get_reference(struct rev_info *revs, const char *name,\n \t\tobject = parse_object(revs->repo, oid);\n \t}\n \n+out:\n \tif (!object) {\n \t\tif (revs->ignore_missing)\n \t\t\treturn object;\n-- \n2.32.0\n\n"},{"id":"432098","messageId":"e33cd51ebf9751a156198ff489304c897923b64c.1628162156.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1628162156.git.ps@pks.im","subject":"[PATCH v4 5/6] commit-graph: split out function to search commit position","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-05T11:25:41Z","receivedAt":"2021-08-05T11:25:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The function `find_commit_in_graph()` assumes that the caller has passed\nan object which was already determined to be a commit given that it will\naccess the commit's graph position, which is stored in a commit slab. In\na subsequent patch, we want to search for an object ID though without\nknowing whether it is a commit or not, which is not currently possible.\n\nSplit out the logic to search the commit graph for a given object ID to\nprepare for this change. This commit also renames the function to\n`find_commit_pos_in_graph()`, which more accurately reflects what this\nfunction does. Furthermore, in order to allow for the searched object ID\nto be const, we need to adjust `bsearch_graph()`'s signature to accept a\nconstant object ID as input, too.\n\nSigned-off-by: Patrick Steinhardt \n---\n commit-graph.c | 55 +++++++++++++++++++++++++++-----------------------\n 1 file changed, 30 insertions(+), 25 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 3860a0d847..8c4c7262c8 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -723,7 +723,7 @@ void close_commit_graph(struct raw_object_store *o)\n \to->commit_graph = NULL;\n }\n \n-static int bsearch_graph(struct commit_graph *g, struct object_id *oid, uint32_t *pos)\n+static int bsearch_graph(struct commit_graph *g, const struct object_id *oid, uint32_t *pos)\n {\n \treturn bsearch_hash(oid->hash, g->chunk_oid_fanout,\n \t\t\t g->chunk_oid_lookup, g->hash_len, pos);\n@@ -864,25 +864,30 @@ static int fill_commit_in_graph(struct repository *r,\n \treturn 1;\n }\n \n-static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t *pos)\n+static int search_commit_pos_in_graph(const struct object_id *id, struct commit_graph *g, uint32_t *pos)\n+{\n+\tstruct commit_graph *cur_g = g;\n+\tuint32_t lex_index;\n+\n+\twhile (cur_g && !bsearch_graph(cur_g, id, &lex_index))\n+\t\tcur_g = cur_g->base_graph;\n+\n+\tif (cur_g) {\n+\t\t*pos = lex_index + cur_g->num_commits_in_base;\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int find_commit_pos_in_graph(struct commit *item, struct commit_graph *g, uint32_t *pos)\n {\n \tuint32_t graph_pos = commit_graph_position(item);\n \tif (graph_pos != COMMIT_NOT_FROM_GRAPH) {\n \t\t*pos = graph_pos;\n \t\treturn 1;\n \t} else {\n-\t\tstruct commit_graph *cur_g = g;\n-\t\tuint32_t lex_index;\n-\n-\t\twhile (cur_g && !bsearch_graph(cur_g, &(item->object.oid), &lex_index))\n-\t\t\tcur_g = cur_g->base_graph;\n-\n-\t\tif (cur_g) {\n-\t\t\t*pos = lex_index + cur_g->num_commits_in_base;\n-\t\t\treturn 1;\n-\t\t}\n-\n-\t\treturn 0;\n+\t\treturn search_commit_pos_in_graph(&item->object.oid, g, pos);\n \t}\n }\n \n@@ -895,7 +900,7 @@ static int parse_commit_in_graph_one(struct repository *r,\n \tif (item->object.parsed)\n \t\treturn 1;\n \n-\tif (find_commit_in_graph(item, g, &pos))\n+\tif (find_commit_pos_in_graph(item, g, &pos))\n \t\treturn fill_commit_in_graph(r, item, g, pos);\n \n \treturn 0;\n@@ -921,7 +926,7 @@ void load_commit_graph_info(struct repository *r, struct commit *item)\n \tuint32_t pos;\n \tif (!prepare_commit_graph(r))\n \t\treturn;\n-\tif (find_commit_in_graph(item, r->objects->commit_graph, &pos))\n+\tif (find_commit_pos_in_graph(item, r->objects->commit_graph, &pos))\n \t\tfill_commit_graph_info(item, r->objects->commit_graph, pos);\n }\n \n@@ -1091,9 +1096,9 @@ static int write_graph_chunk_data(struct hashfile *f,\n \t\t\t\tedge_value += ctx->new_num_commits_in_base;\n \t\t\telse if (ctx->new_base_graph) {\n \t\t\t\tuint32_t pos;\n-\t\t\t\tif (find_commit_in_graph(parent->item,\n-\t\t\t\t\t\t\t ctx->new_base_graph,\n-\t\t\t\t\t\t\t &pos))\n+\t\t\t\tif (find_commit_pos_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n \t\t\t\t\tedge_value = pos;\n \t\t\t}\n \n@@ -1122,9 +1127,9 @@ static int write_graph_chunk_data(struct hashfile *f,\n \t\t\t\tedge_value += ctx->new_num_commits_in_base;\n \t\t\telse if (ctx->new_base_graph) {\n \t\t\t\tuint32_t pos;\n-\t\t\t\tif (find_commit_in_graph(parent->item,\n-\t\t\t\t\t\t\t ctx->new_base_graph,\n-\t\t\t\t\t\t\t &pos))\n+\t\t\t\tif (find_commit_pos_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n \t\t\t\t\tedge_value = pos;\n \t\t\t}\n \n@@ -1235,9 +1240,9 @@ static int write_graph_chunk_extra_edges(struct hashfile *f,\n \t\t\t\tedge_value += ctx->new_num_commits_in_base;\n \t\t\telse if (ctx->new_base_graph) {\n \t\t\t\tuint32_t pos;\n-\t\t\t\tif (find_commit_in_graph(parent->item,\n-\t\t\t\t\t\t\t ctx->new_base_graph,\n-\t\t\t\t\t\t\t &pos))\n+\t\t\t\tif (find_commit_pos_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n \t\t\t\t\tedge_value = pos;\n \t\t\t}\n \n-- \n2.32.0\n\n"},{"id":"432099","messageId":"900c5a9c60282e0fb57840305cbab013ffa398f1.1628162156.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1628162156.git.ps@pks.im","subject":"[PATCH v4 6/6] revision: avoid hitting packfiles when commits are in commit-graph","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-05T11:25:45Z","receivedAt":"2021-08-05T11:25:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When queueing references in git-rev-list(1), we try to either reuse an\nalready parsed object or alternatively we load the object header from\ndisk in order to determine its type. This is inefficient for commits\nthough in cases where we have a commit graph available: instead of\nhitting the real object on disk to determine its type, we may instead\nsearch the object graph for the object ID. In case it's found, we can\ndirectly fill in the commit object, otherwise we can still hit the disk\nto determine the object's type.\n\nExpose a new function `parse_commit_in_graph_gently()`, which fills in\nan object of unknown type in case we find its object ID in the graph.\nThis provides a big performance win in cases where there is a\ncommit-graph available in the repository in case we load lots of\nreferences. The following has been executed in a real-world repository\nwith about 2.2 million refs:\n\n Benchmark #1: HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n Time (mean ± σ): 4.508 s ± 0.039 s [User: 4.131 s, System: 0.377 s]\n Range (min … max): 4.455 s … 4.576 s 10 runs\n\n Benchmark #2: HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n Time (mean ± σ): 3.072 s ± 0.031 s [User: 2.707 s, System: 0.365 s]\n Range (min … max): 3.040 s … 3.144 s 10 runs\n\n Summary\n 'HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev' ran\n 1.47 ± 0.02 times faster than 'HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev'\n\nSigned-off-by: Patrick Steinhardt \n---\n commit-graph.c | 26 ++++++++++++++++++++++++++\n commit-graph.h | 7 +++++++\n revision.c | 17 +++++++++++++++--\n 3 files changed, 48 insertions(+), 2 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8c4c7262c8..cc7136243d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -891,6 +891,32 @@ static int find_commit_pos_in_graph(struct commit *item, struct commit_graph *g,\n \t}\n }\n \n+int parse_commit_in_graph_gently(struct repository *repo, struct object *object)\n+{\n+\tstruct commit *commit;\n+\tuint32_t pos;\n+\n+\tif (object->parsed) {\n+\t\tif (object->type != OBJ_COMMIT)\n+\t\t\treturn -1;\n+\t\treturn 0;\n+\t}\n+\n+\tif (!repo->objects->commit_graph)\n+\t\treturn -1;\n+\n+\tif (!search_commit_pos_in_graph(&object->oid, repo->objects->commit_graph, &pos))\n+\t\treturn -1;\n+\n+\tcommit = object_as_type(object, OBJ_COMMIT, 1);\n+\tif (!commit)\n+\t\treturn -1;\n+\tif (!fill_commit_in_graph(repo, commit, repo->objects->commit_graph, pos))\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n static int parse_commit_in_graph_one(struct repository *r,\n \t\t\t\t struct commit_graph *g,\n \t\t\t\t struct commit *item)\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 96c24fb577..e2e93b0ee1 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -40,6 +40,13 @@ int open_commit_graph(const char *graph_file, int *fd, struct stat *st);\n */\n int parse_commit_in_graph(struct repository *r, struct commit *item);\n \n+/*\n+ * Given an object of unknown type, try to fill in the object in case it is a\n+ * commit part of the commit-graph. Returns 0 if the object is a parsed commit\n+ * or if it could be filled in via the commit graph, otherwise it returns -1.\n+ */\n+int parse_commit_in_graph_gently(struct repository *repo, struct object *object);\n+\n /*\n * It is possible that we loaded commit contents from the commit buffer,\n * but we also want to ensure the commit-graph content is correctly\ndiff --git a/revision.c b/revision.c\nindex 25f4784fdd..318b43026e 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -362,8 +362,21 @@ static struct object *get_reference(struct rev_info *revs, const char *name,\n \tstruct object *object = lookup_unknown_object(revs->repo, oid);\n \n \tif (object->type == OBJ_NONE) {\n-\t\tint type = oid_object_info(revs->repo, oid, NULL);\n-\t\tif (type < 0 || !object_as_type(object, type, 1)) {\n+\t\t/*\n+\t\t * It's likely that the reference points to a commit, so we\n+\t\t * first try to look it up via the commit-graph. If successful,\n+\t\t * then we know it's a commit and don't have to unpack the\n+\t\t * object header. We still need to assert that the object\n+\t\t * exists, but given that we don't request any info about the\n+\t\t * object this is a lot faster than `oid_object_info()`.\n+\t\t */\n+\t\tif (parse_commit_in_graph_gently(revs->repo, object) < 0) {\n+\t\t\tint type = oid_object_info(revs->repo, oid, NULL);\n+\t\t\tif (type < 0 || !object_as_type(object, type, 1)) {\n+\t\t\t\tobject = NULL;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\t\t} else if (!repo_has_object_file(revs->repo, oid)) {\n \t\t\tobject = NULL;\n \t\t\tgoto out;\n \t\t}\n-- \n2.32.0\n\n"},{"id":"432118","messageId":"xmqqy29flvd7.fsf@gitster.g","threadId":"56018","inReplyTo":"YQvFGuf3Ba/5UNXk@ncase","subject":"Re: [PATCH v3 4/4] revision: avoid hitting packfiles when commits are in commit-graph","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-05T16:16:52Z","receivedAt":"2021-08-05T16:16:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Tue, Aug 03, 2021 at 02:56:49PM -0700, Junio C Hamano wrote:\n>> Patrick Steinhardt writes:\n>> \n>> > I wonder what our stance on this is. I can definitely understand the\n>> > angle that this would be a deal breaker given that we now claim commits\n>> > exist which don't anymore.\n>> \n>> An optimization that produces a wrong result very fast is a useless\n>> optimization that has no place in our codebase. But don't we have\n>> some clue recorded in the commit graph file that tells us with what\n>> packfile the graph is to be used (iow, if the named packfile still\n>> exists there, the objects recorded in the graph file are to be found\n>> there) or something?\n>\n> Unfortunately, no. For bitmaps we have this information given that a\n> bitmap is tied to a specific pack anyway. But for commit-graphs, the\n> story is different given that they don't really care about the packs per\n> se, but only about the commits.\n\n[jc: refreshed Cc: list to limit to those in \"shortlog commit-graph.[ch]\"]\n\nOn this subject, I'd ask those who have worked on the commit-graph\nfor ideas. It would be a glaring flaw _if_ the data structure that\nis designed to be a \"cache of precomputed summary that would help\nruntime performance\" has no way to detect out-of-date cache and/or\nto invalidate when it goes stale, but I somehow doubt that is the\ncase, given the caliber of folks who have worked in it. To me, it\nfeels a lot more likely that we may be missing an existing mechanism\nto do so. It could be that ...\n\n> We can do the following on top though:\n>\n> diff --git a/revision.c b/revision.c\n> index 3527ef3f65..9e62de20ab 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -368,6 +368,8 @@ static struct object *get_reference(struct rev_info *revs, const char *name,\n> \t\t\t\tobject = NULL;\n> \t\t\t\tgoto out;\n> \t\t\t}\n> +\t\t} else if (!repo_has_object_file(revs->repo, oid)) {\n> +\t\t\tdie(\"bad object %s\", name);\n> \t\t}\n> \t}\n>\n> We assert that the object exists, but `repo_has_object_file()` won't try\n> to unpack the object header given that we request no info about the\n> object. And because the object ID has been part of the commit-graph, we\n> know that it's a commit. It's a bit slower compared to the version where\n> we don't assert object existence, but still a lot faster compared to\n> looking up the object type via the ODB.\n\n... the above is the designed way to correctly use the commit-graph\ndata? That is, you find an object in the commit-graph, and you make\nsure the object exists in the object store in some other means\n(because there is no mechanism for commit-graph to prevent a gc from\npruning an object recorded in it away) before you consider you can\nuse the object.\n\nThoughts and help?\n\nThanks.\n"},{"id":"432123","messageId":"xmqqfsvnloju.fsf@gitster.g","threadId":"56018","inReplyTo":"9d7f484907e2bd2492e6676238579e9f0c6ed374.1628162156.git.ps@pks.im","subject":"Re: [PATCH v4 2/6] connected: do not sort input revisions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-05T18:44:05Z","receivedAt":"2021-08-05T18:44:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> +\t} else if (!strcmp(arg, \"--unsorted-input\")) {\n> +\t\tif (revs->no_walk && !revs->unsorted_input)\n> +\t\t\tdie(_(\"--unsorted-input is incompatible with --no-walk and --no-walk=sorted\"));\n> +\t\trevs->unsorted_input = 1;\n\nSo this can be used with --no-walk=unsorted, even though doing\nso would be redundant and meaningless. OK.\n\n> @@ -2651,6 +2655,8 @@ static int handle_revision_pseudo_opt(const char *submodule,\n> \t} else if (!strcmp(arg, \"--not\")) {\n> \t\t*flags ^= UNINTERESTING | BOTTOM;\n> \t} else if (!strcmp(arg, \"--no-walk\")) {\n> +\t\tif (revs->unsorted_input)\n> +\t\t\tdie(_(\"--no-walk is incompatible with --no-walk=unsorted and --unsorted-input\"));\n\nAnd likewise, --no-walk is --no-walk=sorted so we do not allow it to\nbe used with --unsorted-input or --no=walk=unsorted. OK.\n\n> \t\trevs->no_walk = 1;\n> \t} else if (skip_prefix(arg, \"--no-walk=\", &optarg)) {\n> \t\t/*\n> @@ -2658,9 +2664,12 @@ static int handle_revision_pseudo_opt(const char *submodule,\n> \t\t * not allowed, since the argument is optional.\n> \t\t */\n> \t\trevs->no_walk = 1;\n> -\t\tif (!strcmp(optarg, \"sorted\"))\n> +\t\tif (!strcmp(optarg, \"sorted\")) {\n> +\t\t\tif (revs->unsorted_input)\n> +\t\t\t\tdie(_(\"--no-walk=sorted is incompatible with --no-walk=unsorted \"\n> +\t\t\t\t \"and --unsorted-input\"));\n\nOK.\n\n> \t\t\trevs->unsorted_input = 0;\n> -\t\telse if (!strcmp(optarg, \"unsorted\"))\n> +\t\t} else if (!strcmp(optarg, \"unsorted\"))\n> \t\t\trevs->unsorted_input = 1;\n\nThis is --no-walk=unsorted; could it have been given after --no-walk\nor --no-walk=unsorted?\n\nThe application of the incompatibility rules seems a bit uneven. An\nearlier piece of code will reject \"--no-walk=unsorted --no-walk\" given\nin this order (see \"And likewise\" above). But here, this part of\nthe code will happily take \"--no-walk --no-walk=unsorted\".\n\nOf course these details can be fixed with more careful code design,\nbut I wonder if it may be result in the code and behaviour that is\nfar simpler to explain (and probably implement) if we declare that\n\n * --no-walk is not a synonym to --no-walk=sorted; it just flips\n .no_walk member on.\n\n * --no-walk=sorted and --no-walk=unsorted flip .no_walk member on,\n and then flips .unsorted_input member off or on, respectively.\n\nand define that the usual last-one-wins rule would apply?\n\nThanks.\n"},{"id":"432124","messageId":"xmqqa6lvloec.fsf@gitster.g","threadId":"56018","inReplyTo":"67232910acf4a248654060ad51a17af37acba0fb.1628162156.git.ps@pks.im","subject":"Re: [PATCH v4 1/6] revision: separate walk and unsorted flags","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-05T18:47:23Z","receivedAt":"2021-08-05T18:47:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> The `--no-walk` flag supports two modes: either it sorts the revisions\n> given as input input or it doesn't. This is reflected in a single\n> `no_walk` flag, which reflects one of the three states \"walk\", \"don't\n> walk but without sorting\" and \"don't walk but with sorting\".\n>\n> Split up the flag into two separate bits, one indicating whether we\n> should walk or not and one indicating whether the input should be sorted\n> or not. This will allow us to more easily introduce a new flag\n> `--unsorted-input`, which only impacts the sorting bit.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> builtin/log.c | 2 +-\n> builtin/revert.c | 3 ++-\n> revision.c | 9 +++++----\n> revision.h | 7 ++-----\n> 4 files changed, 10 insertions(+), 11 deletions(-)\n\nEvery line of changes in the patch is very satisfying ;-)\n"},{"id":"432152","messageId":"YQzQGVQdh4t0uZ3N@ncase","threadId":"56018","inReplyTo":"xmqqfsvnloju.fsf@gitster.g","subject":"Re: [PATCH v4 2/6] connected: do not sort input revisions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-06T06:00:57Z","receivedAt":"2021-08-06T06:20:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Aug 05, 2021 at 11:44:05AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> > \t\t\trevs->unsorted_input = 0;\n> > -\t\telse if (!strcmp(optarg, \"unsorted\"))\n> > +\t\t} else if (!strcmp(optarg, \"unsorted\"))\n> > \t\t\trevs->unsorted_input = 1;\n> \n> This is --no-walk=unsorted; could it have been given after --no-walk\n> or --no-walk=unsorted?\n> \n> The application of the incompatibility rules seems a bit uneven. An\n> earlier piece of code will reject \"--no-walk=unsorted --no-walk\" given\n> in this order (see \"And likewise\" above). But here, this part of\n> the code will happily take \"--no-walk --no-walk=unsorted\".\n> \n> Of course these details can be fixed with more careful code design,\n> but I wonder if it may be result in the code and behaviour that is\n> far simpler to explain (and probably implement) if we declare that\n> \n> * --no-walk is not a synonym to --no-walk=sorted; it just flips\n> .no_walk member on.\n> \n> * --no-walk=sorted and --no-walk=unsorted flip .no_walk member on,\n> and then flips .unsorted_input member off or on, respectively.\n> \n> and define that the usual last-one-wins rule would apply?\n> \n> Thanks.\n\nWouldn't that effectively change semantics though? If the user passes\n`git rev-list --no-walk=unsorted --no-walk`, then the result is a sorted\nrevwalk right now. One may argue that most likely, nobody is doing that,\nbut you never really know.\n\nAn easier approach which keeps existing semantics is to just make\n`--no-walk` and `--unsorted-input` mutually exclusive:\n\n - If the `unsorted_input` bit is set and `no_walk` isn't, and we\n observe any `--no-walk` option, then we bail.\n\n - Likewise, if the `no_walk` bit is set, then we bail when we see\n `--unsorted-input` regardless of the value of `unsorted_input`.\n This would keep current semantics of `--no-walk`, but prohobit\n using it together with the new option.\n\nPatrick\n"},{"id":"432177","messageId":"YQ1EZAylyI4pAC0E@ncase","threadId":"56018","inReplyTo":"YQkHUZKZ6gkFkaD0@ncase","subject":"Re: [PATCH v3 3/4] revision: avoid loading object headers multiple times","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-06T14:17:08Z","receivedAt":"2021-08-06T14:17:20Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Aug 03, 2021 at 11:07:29AM +0200, Patrick Steinhardt wrote:\n> On Mon, Aug 02, 2021 at 12:40:56PM -0700, Junio C Hamano wrote:\n> > Patrick Steinhardt writes:\n> > \n> > > When loading references, we try to optimize loading of commits by using\n> > > the commit graph. To do so, we first need to determine whether the\n> > > object actually is a commit or not, which is why we always execute\n> > > `oid_object_info()` first. Like this, we'll unpack the object header of\n> > > each object first.\n> > >\n> > > This pattern can be quite inefficient in case many references point to\n> > > the same commit: if the object didn't end up in the cached objects, then\n> > > we'll repeatedly unpack the same object header, even if we've already\n> > > seen the object before.\n> > > ...\n> > > Assuming that in almost all repositories, most references will point to\n> > > either a tag or a commit, we'd have a modest increase in memory\n> > > consumption of about 12.5% here.\n> > \n> > I wonder if we can also say almost all repositories, the majority of\n> > refs point at the same object. If that holds, this would certainly\n> > be a win, but otherwise, it is not so clear.\n> \n> I doubt that's the case in general. I rather assume that it's typically\n> going to be a smallish subset that points to the same commit, but for\n> these cases we at least avoid doing the lookup multiple times. As I\n> said, it's definitely a tradeoff between memory and performance: in the\n> worst case (all references point to different blobs) we allocate 33%\n> more memory without having any speedups. A more realistic scenario would\n> probably be something like a trunk-based development repo, where there's\n> a single branch only and the rest is tags. There we'd allocate 11% more\n> memory without any speedups. In general, it's going to be various shades\n> of gray, where we allocate something from 0% to 11% more memory while\n> getting some modest speedups in some cases.\n> \n> So if we only inspect this commit as a standalone it's definitely\n> debatable whether we'd want to take it or not. But one important thing\n> is that it's a prerequisite for patch 4/4: in order to not parse commits\n> in case they're part of the commit-graph, we need to first obtain an\n> object such that we can fill it in via the graph. So we have to call\n> `lookup_unknown_object()` anyway. Might be sensible to document this as\n> part of the commit message.\n\nScratch that: we can just rewrite this to use `lookup_object()` to check\nwhether we already know the object, and then we can rewrite the\n`parse_commit_in_graph_gently()` to be `lookup_commit_in_graph()`, which\ndoes a `lookup_commit()` to create the object in case the OID was found\nin the graph. That's also got nicer calling semantics..\n\nIt's negligibly slower (3.021s instead of 2.983s), but it doesn't need\nme arguing about the performance/memory tradeoff. Which is a good thing\nI guess: there will always be that one repo that's completely different\nand where such assumptions don't hold.\n\nPatrick\n"},{"id":"432186","messageId":"xmqqwnoyikkd.fsf@gitster.g","threadId":"56018","inReplyTo":"YQzQGVQdh4t0uZ3N@ncase","subject":"Re: [PATCH v4 2/6] connected: do not sort input revisions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-08-06T16:50:42Z","receivedAt":"2021-08-06T16:50:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> Wouldn't that effectively change semantics though? If the user passes\n> `git rev-list --no-walk=unsorted --no-walk`, then the result is a sorted\n> revwalk right now. One may argue that most likely, nobody is doing that,\n> but you never really know.\n\nTrue.\n\n> An easier approach which keeps existing semantics is to just make\n> `--no-walk` and `--unsorted-input` mutually exclusive:\n>\n> - If the `unsorted_input` bit is set and `no_walk` isn't, and we\n> observe any `--no-walk` option, then we bail.\n>\n> - Likewise, if the `no_walk` bit is set, then we bail when we see\n> `--unsorted-input` regardless of the value of `unsorted_input`.\n> This would keep current semantics of `--no-walk`, but prohobit\n> using it together with the new option.\n\nTrue again. As I said, I do prefer going in the \"start tight to\nforbid combination\" route. But as I pointed out, the coverage by\nthe posted patch seemed to have gap(s).\n\nThanks.\n\n"},{"id":"432292","messageId":"cover.1628495467.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1624858240.git.ps@pks.im","subject":"[PATCH v5 0/5] Speed up connectivity checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-09T08:00:39Z","receivedAt":"2021-08-09T08:00:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the fifth version of my series to speed up connectivity checks\nin the context of repos with many refs. While the original goal has been\nto speed up connectivity checks only, the series is now optimizing\ngit-rev-list(1) in general to be able to more efficiently load\nreferences. Like this, `--not --all` is a lot faster in the context of\nmany refs, but other usecases benefit, too.\n\nChanges compared to v4:\n\n - I've changed the interface to load commits via the commit-graph.\n Instead of the previous version where you'd had to pass in a\n `struct object`, which forced us to use `lookup_unknown_object()`,\n one now only passes in an object ID. If the object ID is found in\n the commit graph and if the corresponding object exists in the\n ODB, then we return the parsed commit object.\n\n This also avoids a previous pitfal: we'd have parsed the commit\n via the graph and thus had allocated the object even if the\n corresponding object didn't exist. While we knew to handle this in\n `get_reference()` by asserting object existence, any other caller\n which executes `lookup_commit()` would get the parsed commit and\n assume that it exists. This now cannot happen anymore given that\n we only create the commit object in case we know the ID exists in\n the ODB.\n\n - With this change, I could now drop the patch which avoided loading\n objects multiple times: we don't need `lookup_unknown_object()`\n anymore and thus don't thave the memory/perf tradeoff. And with\n the new interface to load commits via the graph, the deduplication\n only resulted in a ~1% speedup.\n\nPatrick\n\nPatrick Steinhardt (5):\n revision: separate walk and unsorted flags\n connected: do not sort input revisions\n revision: stop retrieving reference twice\n commit-graph: split out function to search commit position\n revision: avoid hitting packfiles when commits are in commit-graph\n\n Documentation/rev-list-options.txt | 8 ++-\n builtin/log.c | 2 +-\n builtin/revert.c | 3 +-\n commit-graph.c | 79 ++++++++++++++++++++----------\n commit-graph.h | 8 +++\n connected.c | 1 +\n revision.c | 42 +++++++++-------\n revision.h | 7 +--\n t/t6000-rev-list-misc.sh | 38 ++++++++++++++\n 9 files changed, 138 insertions(+), 50 deletions(-)\n\nRange-diff against v4:\n1: 67232910ac = 1: 67232910ac revision: separate walk and unsorted flags\n2: 9d7f484907 = 2: 9d7f484907 connected: do not sort input revisions\n3: d8e63d0943 = 3: d8e63d0943 revision: stop retrieving reference twice\n4: ba8df5cad0 < -: ---------- revision: avoid loading object headers multiple times\n5: e33cd51ebf = 4: 549d85e5c2 commit-graph: split out function to search commit position\n6: 900c5a9c60 ! 5: 4b893d943f revision: avoid hitting packfiles when commits are in commit-graph\n @@ Metadata\n ## Commit message ##\n revision: avoid hitting packfiles when commits are in commit-graph\n \n - When queueing references in git-rev-list(1), we try to either reuse an\n - already parsed object or alternatively we load the object header from\n - disk in order to determine its type. This is inefficient for commits\n - though in cases where we have a commit graph available: instead of\n - hitting the real object on disk to determine its type, we may instead\n - search the object graph for the object ID. In case it's found, we can\n - directly fill in the commit object, otherwise we can still hit the disk\n - to determine the object's type.\n + When queueing references in git-rev-list(1), we try to optimize parsing\n + of commits via the commit-graph. To do so, we first look up the object's\n + type, and if it is a commit we call `repo_parse_commit()` instead of\n + `parse_object()`. This is quite inefficient though given that we're\n + always uncompressing the object header in order to determine the type.\n + Instead, we can opportunistically search the commit-graph for the object\n + ID: in case it's found, we know it's a commit and can directly fill in\n + the commit object without having to uncompress the object header.\n \n - Expose a new function `parse_commit_in_graph_gently()`, which fills in\n - an object of unknown type in case we find its object ID in the graph.\n - This provides a big performance win in cases where there is a\n - commit-graph available in the repository in case we load lots of\n - references. The following has been executed in a real-world repository\n - with about 2.2 million refs:\n + Expose a new function `lookup_commit_in_graph()`, which tries to find a\n + commit in the commit-graph by ID, and convert `get_reference()` to use\n + this function. This provides a big performance win in cases where we\n + load references in a repository with lots of references pointing to\n + commits. The following has been executed in a real-world repository with\n + about 2.2 million refs:\n \n Benchmark #1: HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n - Time (mean ± σ): 4.508 s ± 0.039 s [User: 4.131 s, System: 0.377 s]\n - Range (min … max): 4.455 s … 4.576 s 10 runs\n + Time (mean ± σ): 4.458 s ± 0.044 s [User: 4.115 s, System: 0.342 s]\n + Range (min … max): 4.409 s … 4.534 s 10 runs\n \n Benchmark #2: HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n - Time (mean ± σ): 3.072 s ± 0.031 s [User: 2.707 s, System: 0.365 s]\n - Range (min … max): 3.040 s … 3.144 s 10 runs\n + Time (mean ± σ): 3.089 s ± 0.015 s [User: 2.768 s, System: 0.321 s]\n + Range (min … max): 3.061 s … 3.105 s 10 runs\n \n Summary\n 'HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev' ran\n - 1.47 ± 0.02 times faster than 'HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev'\n + 1.44 ± 0.02 times faster than 'HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev'\n \n Signed-off-by: Patrick Steinhardt \n \n @@ commit-graph.c: static int find_commit_pos_in_graph(struct commit *item, struct\n \t}\n }\n \n -+int parse_commit_in_graph_gently(struct repository *repo, struct object *object)\n ++struct commit *lookup_commit_in_graph(struct repository *repo, const struct object_id *id)\n +{\n +\tstruct commit *commit;\n +\tuint32_t pos;\n +\n -+\tif (object->parsed) {\n -+\t\tif (object->type != OBJ_COMMIT)\n -+\t\t\treturn -1;\n -+\t\treturn 0;\n -+\t}\n -+\n +\tif (!repo->objects->commit_graph)\n -+\t\treturn -1;\n ++\t\treturn NULL;\n ++\tif (!search_commit_pos_in_graph(id, repo->objects->commit_graph, &pos))\n ++\t\treturn NULL;\n ++\tif (!repo_has_object_file(repo, id))\n ++\t\treturn NULL;\n +\n -+\tif (!search_commit_pos_in_graph(&object->oid, repo->objects->commit_graph, &pos))\n -+\t\treturn -1;\n -+\n -+\tcommit = object_as_type(object, OBJ_COMMIT, 1);\n ++\tcommit = lookup_commit(repo, id);\n +\tif (!commit)\n -+\t\treturn -1;\n -+\tif (!fill_commit_in_graph(repo, commit, repo->objects->commit_graph, pos))\n -+\t\treturn -1;\n ++\t\treturn NULL;\n ++\tif (commit->object.parsed)\n ++\t\treturn commit;\n +\n -+\treturn 0;\n ++\tif (!fill_commit_in_graph(repo, commit, repo->objects->commit_graph, pos))\n ++\t\treturn NULL;\n ++\n ++\treturn commit;\n +}\n +\n static int parse_commit_in_graph_one(struct repository *r,\n @@ commit-graph.h: int open_commit_graph(const char *graph_file, int *fd, struct st\n int parse_commit_in_graph(struct repository *r, struct commit *item);\n \n +/*\n -+ * Given an object of unknown type, try to fill in the object in case it is a\n -+ * commit part of the commit-graph. Returns 0 if the object is a parsed commit\n -+ * or if it could be filled in via the commit graph, otherwise it returns -1.\n ++ * Look up the given commit ID in the commit-graph. This will only return a\n ++ * commit if the ID exists both in the graph and in the object database such\n ++ * that we don't return commits whose object has been pruned. Otherwise, this\n ++ * function returns `NULL`.\n + */\n -+int parse_commit_in_graph_gently(struct repository *repo, struct object *object);\n ++struct commit *lookup_commit_in_graph(struct repository *repo, const struct object_id *id);\n +\n /*\n * It is possible that we loaded commit contents from the commit buffer,\n @@ commit-graph.h: int open_commit_graph(const char *graph_file, int *fd, struct st\n \n ## revision.c ##\n @@ revision.c: static struct object *get_reference(struct rev_info *revs, const char *name,\n - \tstruct object *object = lookup_unknown_object(revs->repo, oid);\n + \t\t\t\t unsigned int flags)\n + {\n + \tstruct object *object;\n ++\tstruct commit *commit;\n \n - \tif (object->type == OBJ_NONE) {\n --\t\tint type = oid_object_info(revs->repo, oid, NULL);\n --\t\tif (type < 0 || !object_as_type(object, type, 1)) {\n -+\t\t/*\n -+\t\t * It's likely that the reference points to a commit, so we\n -+\t\t * first try to look it up via the commit-graph. If successful,\n -+\t\t * then we know it's a commit and don't have to unpack the\n -+\t\t * object header. We still need to assert that the object\n -+\t\t * exists, but given that we don't request any info about the\n -+\t\t * object this is a lot faster than `oid_object_info()`.\n -+\t\t */\n -+\t\tif (parse_commit_in_graph_gently(revs->repo, object) < 0) {\n -+\t\t\tint type = oid_object_info(revs->repo, oid, NULL);\n -+\t\t\tif (type < 0 || !object_as_type(object, type, 1)) {\n -+\t\t\t\tobject = NULL;\n -+\t\t\t\tgoto out;\n -+\t\t\t}\n -+\t\t} else if (!repo_has_object_file(revs->repo, oid)) {\n - \t\t\tobject = NULL;\n - \t\t\tgoto out;\n - \t\t}\n + \t/*\n +-\t * If the repository has commit graphs, repo_parse_commit() avoids\n +-\t * reading the object buffer, so use it whenever possible.\n ++\t * If the repository has commit graphs, we try to opportunistically\n ++\t * look up the object ID in those graphs. Like this, we can avoid\n ++\t * parsing commit data from disk.\n + \t */\n +-\tif (oid_object_info(revs->repo, oid, NULL) == OBJ_COMMIT) {\n +-\t\tstruct commit *c = lookup_commit(revs->repo, oid);\n +-\t\tif (!repo_parse_commit(revs->repo, c))\n +-\t\t\tobject = (struct object *) c;\n +-\t\telse\n +-\t\t\tobject = NULL;\n +-\t} else {\n ++\tcommit = lookup_commit_in_graph(revs->repo, oid);\n ++\tif (commit)\n ++\t\tobject = &commit->object;\n ++\telse\n + \t\tobject = parse_object(revs->repo, oid);\n +-\t}\n + \n + \tif (!object) {\n + \t\tif (revs->ignore_missing)\n-- \n2.32.0\n\n"},{"id":"432293","messageId":"YRDg+U0IJ3rcst9l@ncase","threadId":"56018","inReplyTo":"cover.1628495467.git.ps@pks.im","subject":"Re: [PATCH v5 0/5] Speed up connectivity checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-09T08:02:01Z","receivedAt":"2021-08-09T08:02:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"You can ignore this version. I realized I forgot to address some\nfeedback while I was sending mails out. I'll resend in some minutes.\n\nPatrick\n\nOn Mon, Aug 09, 2021 at 10:00:39AM +0200, Patrick Steinhardt wrote:\n> Hi,\n> \n> this is the fifth version of my series to speed up connectivity checks\n> in the context of repos with many refs. While the original goal has been\n> to speed up connectivity checks only, the series is now optimizing\n> git-rev-list(1) in general to be able to more efficiently load\n> references. Like this, `--not --all` is a lot faster in the context of\n> many refs, but other usecases benefit, too.\n> \n> Changes compared to v4:\n> \n> - I've changed the interface to load commits via the commit-graph.\n> Instead of the previous version where you'd had to pass in a\n> `struct object`, which forced us to use `lookup_unknown_object()`,\n> one now only passes in an object ID. If the object ID is found in\n> the commit graph and if the corresponding object exists in the\n> ODB, then we return the parsed commit object.\n> \n> This also avoids a previous pitfal: we'd have parsed the commit\n> via the graph and thus had allocated the object even if the\n> corresponding object didn't exist. While we knew to handle this in\n> `get_reference()` by asserting object existence, any other caller\n> which executes `lookup_commit()` would get the parsed commit and\n> assume that it exists. This now cannot happen anymore given that\n> we only create the commit object in case we know the ID exists in\n> the ODB.\n> \n> - With this change, I could now drop the patch which avoided loading\n> objects multiple times: we don't need `lookup_unknown_object()`\n> anymore and thus don't thave the memory/perf tradeoff. And with\n> the new interface to load commits via the graph, the deduplication\n> only resulted in a ~1% speedup.\n> \n> Patrick\n> \n> Patrick Steinhardt (5):\n> revision: separate walk and unsorted flags\n> connected: do not sort input revisions\n> revision: stop retrieving reference twice\n> commit-graph: split out function to search commit position\n> revision: avoid hitting packfiles when commits are in commit-graph\n> \n> Documentation/rev-list-options.txt | 8 ++-\n> builtin/log.c | 2 +-\n> builtin/revert.c | 3 +-\n> commit-graph.c | 79 ++++++++++++++++++++----------\n> commit-graph.h | 8 +++\n> connected.c | 1 +\n> revision.c | 42 +++++++++-------\n> revision.h | 7 +--\n> t/t6000-rev-list-misc.sh | 38 ++++++++++++++\n> 9 files changed, 138 insertions(+), 50 deletions(-)\n> \n> Range-diff against v4:\n> 1: 67232910ac = 1: 67232910ac revision: separate walk and unsorted flags\n> 2: 9d7f484907 = 2: 9d7f484907 connected: do not sort input revisions\n> 3: d8e63d0943 = 3: d8e63d0943 revision: stop retrieving reference twice\n> 4: ba8df5cad0 < -: ---------- revision: avoid loading object headers multiple times\n> 5: e33cd51ebf = 4: 549d85e5c2 commit-graph: split out function to search commit position\n> 6: 900c5a9c60 ! 5: 4b893d943f revision: avoid hitting packfiles when commits are in commit-graph\n> @@ Metadata\n> ## Commit message ##\n> revision: avoid hitting packfiles when commits are in commit-graph\n> \n> - When queueing references in git-rev-list(1), we try to either reuse an\n> - already parsed object or alternatively we load the object header from\n> - disk in order to determine its type. This is inefficient for commits\n> - though in cases where we have a commit graph available: instead of\n> - hitting the real object on disk to determine its type, we may instead\n> - search the object graph for the object ID. In case it's found, we can\n> - directly fill in the commit object, otherwise we can still hit the disk\n> - to determine the object's type.\n> + When queueing references in git-rev-list(1), we try to optimize parsing\n> + of commits via the commit-graph. To do so, we first look up the object's\n> + type, and if it is a commit we call `repo_parse_commit()` instead of\n> + `parse_object()`. This is quite inefficient though given that we're\n> + always uncompressing the object header in order to determine the type.\n> + Instead, we can opportunistically search the commit-graph for the object\n> + ID: in case it's found, we know it's a commit and can directly fill in\n> + the commit object without having to uncompress the object header.\n> \n> - Expose a new function `parse_commit_in_graph_gently()`, which fills in\n> - an object of unknown type in case we find its object ID in the graph.\n> - This provides a big performance win in cases where there is a\n> - commit-graph available in the repository in case we load lots of\n> - references. The following has been executed in a real-world repository\n> - with about 2.2 million refs:\n> + Expose a new function `lookup_commit_in_graph()`, which tries to find a\n> + commit in the commit-graph by ID, and convert `get_reference()` to use\n> + this function. This provides a big performance win in cases where we\n> + load references in a repository with lots of references pointing to\n> + commits. The following has been executed in a real-world repository with\n> + about 2.2 million refs:\n> \n> Benchmark #1: HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n> - Time (mean ± σ): 4.508 s ± 0.039 s [User: 4.131 s, System: 0.377 s]\n> - Range (min … max): 4.455 s … 4.576 s 10 runs\n> + Time (mean ± σ): 4.458 s ± 0.044 s [User: 4.115 s, System: 0.342 s]\n> + Range (min … max): 4.409 s … 4.534 s 10 runs\n> \n> Benchmark #2: HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n> - Time (mean ± σ): 3.072 s ± 0.031 s [User: 2.707 s, System: 0.365 s]\n> - Range (min … max): 3.040 s … 3.144 s 10 runs\n> + Time (mean ± σ): 3.089 s ± 0.015 s [User: 2.768 s, System: 0.321 s]\n> + Range (min … max): 3.061 s … 3.105 s 10 runs\n> \n> Summary\n> 'HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev' ran\n> - 1.47 ± 0.02 times faster than 'HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev'\n> + 1.44 ± 0.02 times faster than 'HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev'\n> \n> Signed-off-by: Patrick Steinhardt \n> \n> @@ commit-graph.c: static int find_commit_pos_in_graph(struct commit *item, struct\n> \t}\n> }\n> \n> -+int parse_commit_in_graph_gently(struct repository *repo, struct object *object)\n> ++struct commit *lookup_commit_in_graph(struct repository *repo, const struct object_id *id)\n> +{\n> +\tstruct commit *commit;\n> +\tuint32_t pos;\n> +\n> -+\tif (object->parsed) {\n> -+\t\tif (object->type != OBJ_COMMIT)\n> -+\t\t\treturn -1;\n> -+\t\treturn 0;\n> -+\t}\n> -+\n> +\tif (!repo->objects->commit_graph)\n> -+\t\treturn -1;\n> ++\t\treturn NULL;\n> ++\tif (!search_commit_pos_in_graph(id, repo->objects->commit_graph, &pos))\n> ++\t\treturn NULL;\n> ++\tif (!repo_has_object_file(repo, id))\n> ++\t\treturn NULL;\n> +\n> -+\tif (!search_commit_pos_in_graph(&object->oid, repo->objects->commit_graph, &pos))\n> -+\t\treturn -1;\n> -+\n> -+\tcommit = object_as_type(object, OBJ_COMMIT, 1);\n> ++\tcommit = lookup_commit(repo, id);\n> +\tif (!commit)\n> -+\t\treturn -1;\n> -+\tif (!fill_commit_in_graph(repo, commit, repo->objects->commit_graph, pos))\n> -+\t\treturn -1;\n> ++\t\treturn NULL;\n> ++\tif (commit->object.parsed)\n> ++\t\treturn commit;\n> +\n> -+\treturn 0;\n> ++\tif (!fill_commit_in_graph(repo, commit, repo->objects->commit_graph, pos))\n> ++\t\treturn NULL;\n> ++\n> ++\treturn commit;\n> +}\n> +\n> static int parse_commit_in_graph_one(struct repository *r,\n> @@ commit-graph.h: int open_commit_graph(const char *graph_file, int *fd, struct st\n> int parse_commit_in_graph(struct repository *r, struct commit *item);\n> \n> +/*\n> -+ * Given an object of unknown type, try to fill in the object in case it is a\n> -+ * commit part of the commit-graph. Returns 0 if the object is a parsed commit\n> -+ * or if it could be filled in via the commit graph, otherwise it returns -1.\n> ++ * Look up the given commit ID in the commit-graph. This will only return a\n> ++ * commit if the ID exists both in the graph and in the object database such\n> ++ * that we don't return commits whose object has been pruned. Otherwise, this\n> ++ * function returns `NULL`.\n> + */\n> -+int parse_commit_in_graph_gently(struct repository *repo, struct object *object);\n> ++struct commit *lookup_commit_in_graph(struct repository *repo, const struct object_id *id);\n> +\n> /*\n> * It is possible that we loaded commit contents from the commit buffer,\n> @@ commit-graph.h: int open_commit_graph(const char *graph_file, int *fd, struct st\n> \n> ## revision.c ##\n> @@ revision.c: static struct object *get_reference(struct rev_info *revs, const char *name,\n> - \tstruct object *object = lookup_unknown_object(revs->repo, oid);\n> + \t\t\t\t unsigned int flags)\n> + {\n> + \tstruct object *object;\n> ++\tstruct commit *commit;\n> \n> - \tif (object->type == OBJ_NONE) {\n> --\t\tint type = oid_object_info(revs->repo, oid, NULL);\n> --\t\tif (type < 0 || !object_as_type(object, type, 1)) {\n> -+\t\t/*\n> -+\t\t * It's likely that the reference points to a commit, so we\n> -+\t\t * first try to look it up via the commit-graph. If successful,\n> -+\t\t * then we know it's a commit and don't have to unpack the\n> -+\t\t * object header. We still need to assert that the object\n> -+\t\t * exists, but given that we don't request any info about the\n> -+\t\t * object this is a lot faster than `oid_object_info()`.\n> -+\t\t */\n> -+\t\tif (parse_commit_in_graph_gently(revs->repo, object) < 0) {\n> -+\t\t\tint type = oid_object_info(revs->repo, oid, NULL);\n> -+\t\t\tif (type < 0 || !object_as_type(object, type, 1)) {\n> -+\t\t\t\tobject = NULL;\n> -+\t\t\t\tgoto out;\n> -+\t\t\t}\n> -+\t\t} else if (!repo_has_object_file(revs->repo, oid)) {\n> - \t\t\tobject = NULL;\n> - \t\t\tgoto out;\n> - \t\t}\n> + \t/*\n> +-\t * If the repository has commit graphs, repo_parse_commit() avoids\n> +-\t * reading the object buffer, so use it whenever possible.\n> ++\t * If the repository has commit graphs, we try to opportunistically\n> ++\t * look up the object ID in those graphs. Like this, we can avoid\n> ++\t * parsing commit data from disk.\n> + \t */\n> +-\tif (oid_object_info(revs->repo, oid, NULL) == OBJ_COMMIT) {\n> +-\t\tstruct commit *c = lookup_commit(revs->repo, oid);\n> +-\t\tif (!repo_parse_commit(revs->repo, c))\n> +-\t\t\tobject = (struct object *) c;\n> +-\t\telse\n> +-\t\t\tobject = NULL;\n> +-\t} else {\n> ++\tcommit = lookup_commit_in_graph(revs->repo, oid);\n> ++\tif (commit)\n> ++\t\tobject = &commit->object;\n> ++\telse\n> + \t\tobject = parse_object(revs->repo, oid);\n> +-\t}\n> + \n> + \tif (!object) {\n> + \t\tif (revs->ignore_missing)\n> -- \n> 2.32.0\n> \n\n\n"},{"id":"432295","messageId":"cover.1628496538.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1624858240.git.ps@pks.im","subject":"[PATCH v5 0/5] Speed up connectivity checks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-09T08:11:42Z","receivedAt":"2021-08-09T08:11:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the fifth version of my series to speed up connectivity checks\nin the context of repos with many refs. While the original goal has been\nto speed up connectivity checks only, the series is now optimizing\ngit-rev-list(1) in general to be able to more efficiently load\nreferences. Like this, `--not --all` is a lot faster in the context of\nmany refs, but other usecases benefit, too.\n\nChanges compared to v4:\n\n - I've changed the interface to load commits via the commit-graph.\n Instead of the previous version where you'd had to pass in a\n `struct object`, which forced us to use `lookup_unknown_object()`,\n one now only passes in an object ID. If the object ID is found in\n the commit graph and if the corresponding object exists in the\n ODB, then we return the parsed commit object.\n\n This also avoids a previous pitfal: we'd have parsed the commit\n via the graph and thus had allocated the object even if the\n corresponding object didn't exist. While we knew to handle this in\n `get_reference()` by asserting object existence, any other caller\n which executes `lookup_commit()` would get the parsed commit and\n assume that it exists. This now cannot happen anymore given that\n we only create the commit object in case we know the ID exists in\n the ODB.\n\n - With this change, I could now drop the patch which avoided loading\n objects multiple times: we don't need `lookup_unknown_object()`\n anymore and thus don't thave the memory/perf tradeoff. And with\n the new interface to load commits via the graph, the deduplication\n only resulted in a ~1% speedup.\n\n - `--unsorted-input` and `--no-walk` are now mutually exclusive\n regardless of whether `--no-walk` is sorted or unsorted,\n simplifying the logic.\n\nPatrick\n\nPatrick Steinhardt (5):\n revision: separate walk and unsorted flags\n connected: do not sort input revisions\n revision: stop retrieving reference twice\n commit-graph: split out function to search commit position\n revision: avoid hitting packfiles when commits are in commit-graph\n\n Documentation/rev-list-options.txt | 8 ++-\n builtin/log.c | 2 +-\n builtin/revert.c | 3 +-\n commit-graph.c | 79 ++++++++++++++++++++----------\n commit-graph.h | 8 +++\n connected.c | 1 +\n revision.c | 38 ++++++++------\n revision.h | 7 +--\n t/t6000-rev-list-misc.sh | 31 ++++++++++++\n 9 files changed, 129 insertions(+), 48 deletions(-)\n\nRange-diff against v4:\n1: 67232910ac = 1: 67232910ac revision: separate walk and unsorted flags\n2: 9d7f484907 ! 2: d3f498a563 connected: do not sort input revisions\n @@ revision.c: static int handle_revision_opt(struct rev_info *revs, int argc, cons\n \t\trevs->sort_order = REV_SORT_BY_AUTHOR_DATE;\n \t\trevs->topo_order = 1;\n +\t} else if (!strcmp(arg, \"--unsorted-input\")) {\n -+\t\tif (revs->no_walk && !revs->unsorted_input)\n -+\t\t\tdie(_(\"--unsorted-input is incompatible with --no-walk and --no-walk=sorted\"));\n ++\t\tif (revs->no_walk)\n ++\t\t\tdie(_(\"--unsorted-input is incompatible with --no-walk\"));\n +\t\trevs->unsorted_input = 1;\n \t} else if (!strcmp(arg, \"--early-output\")) {\n \t\trevs->early_output = 100;\n @@ revision.c: static int handle_revision_pseudo_opt(const char *submodule,\n \t} else if (!strcmp(arg, \"--not\")) {\n \t\t*flags ^= UNINTERESTING | BOTTOM;\n \t} else if (!strcmp(arg, \"--no-walk\")) {\n -+\t\tif (revs->unsorted_input)\n -+\t\t\tdie(_(\"--no-walk is incompatible with --no-walk=unsorted and --unsorted-input\"));\n ++\t\tif (!revs->no_walk && revs->unsorted_input)\n ++\t\t\tdie(_(\"--no-walk is incompatible with --unsorted-input\"));\n \t\trevs->no_walk = 1;\n \t} else if (skip_prefix(arg, \"--no-walk=\", &optarg)) {\n ++\t\tif (!revs->no_walk && revs->unsorted_input)\n ++\t\t\tdie(_(\"--no-walk is incompatible with --unsorted-input\"));\n ++\n \t\t/*\n -@@ revision.c: static int handle_revision_pseudo_opt(const char *submodule,\n + \t\t * Detached form (\"--no-walk X\" as opposed to \"--no-walk=X\")\n \t\t * not allowed, since the argument is optional.\n - \t\t */\n - \t\trevs->no_walk = 1;\n --\t\tif (!strcmp(optarg, \"sorted\"))\n -+\t\tif (!strcmp(optarg, \"sorted\")) {\n -+\t\t\tif (revs->unsorted_input)\n -+\t\t\t\tdie(_(\"--no-walk=sorted is incompatible with --no-walk=unsorted \"\n -+\t\t\t\t \"and --unsorted-input\"));\n - \t\t\trevs->unsorted_input = 0;\n --\t\telse if (!strcmp(optarg, \"unsorted\"))\n -+\t\t} else if (!strcmp(optarg, \"unsorted\"))\n - \t\t\trevs->unsorted_input = 1;\n - \t\telse\n - \t\t\treturn error(\"invalid argument to --no-walk\");\n \n ## t/t6000-rev-list-misc.sh ##\n @@ t/t6000-rev-list-misc.sh: test_expect_success 'rev-list --count --objects' '\n @@ t/t6000-rev-list-misc.sh: test_expect_success 'rev-list --count --objects' '\n +\ttest_cmp first.sorted second.sorted\n +'\n +\n -+test_expect_success 'rev-list --unsorted-input compatible with --no-walk=unsorted' '\n -+\tgit rev-list --unsorted-input --no-walk=unsorted HEAD HEAD~ >actual &&\n -+\tgit rev-parse HEAD >expect &&\n -+\tgit rev-parse HEAD~ >>expect &&\n -+\ttest_cmp expect actual\n -+'\n -+\n -+test_expect_success 'rev-list --unsorted-input incompatible with --no-walk=sorted' '\n ++test_expect_success 'rev-list --unsorted-input incompatible with --no-walk' '\n +\tcat >expect <<-EOF &&\n -+\t\tfatal: --no-walk is incompatible with --no-walk=unsorted and --unsorted-input\n ++\t\tfatal: --no-walk is incompatible with --unsorted-input\n +\tEOF\n +\ttest_must_fail git rev-list --unsorted-input --no-walk HEAD 2>error &&\n +\ttest_cmp expect error &&\n -+\n -+\tcat >expect <<-EOF &&\n -+\t\tfatal: --no-walk=sorted is incompatible with --no-walk=unsorted and --unsorted-input\n -+\tEOF\n +\ttest_must_fail git rev-list --unsorted-input --no-walk=sorted HEAD 2>error &&\n +\ttest_cmp expect error &&\n ++\ttest_must_fail git rev-list --unsorted-input --no-walk=unsorted HEAD 2>error &&\n ++\ttest_cmp expect error &&\n +\n +\tcat >expect <<-EOF &&\n -+\t\tfatal: --unsorted-input is incompatible with --no-walk and --no-walk=sorted\n ++\t\tfatal: --unsorted-input is incompatible with --no-walk\n +\tEOF\n +\ttest_must_fail git rev-list --no-walk --unsorted-input HEAD 2>error &&\n +\ttest_cmp expect error &&\n +\ttest_must_fail git rev-list --no-walk=sorted --unsorted-input HEAD 2>error &&\n ++\ttest_cmp expect error &&\n ++\ttest_must_fail git rev-list --no-walk=unsorted --unsorted-input HEAD 2>error &&\n +\ttest_cmp expect error\n +'\n +\n3: d8e63d0943 = 3: c9a630927b revision: stop retrieving reference twice\n4: ba8df5cad0 < -: ---------- revision: avoid loading object headers multiple times\n5: e33cd51ebf = 4: bc89325fdf commit-graph: split out function to search commit position\n6: 900c5a9c60 ! 5: fdb1fa9d57 revision: avoid hitting packfiles when commits are in commit-graph\n @@ Metadata\n ## Commit message ##\n revision: avoid hitting packfiles when commits are in commit-graph\n \n - When queueing references in git-rev-list(1), we try to either reuse an\n - already parsed object or alternatively we load the object header from\n - disk in order to determine its type. This is inefficient for commits\n - though in cases where we have a commit graph available: instead of\n - hitting the real object on disk to determine its type, we may instead\n - search the object graph for the object ID. In case it's found, we can\n - directly fill in the commit object, otherwise we can still hit the disk\n - to determine the object's type.\n + When queueing references in git-rev-list(1), we try to optimize parsing\n + of commits via the commit-graph. To do so, we first look up the object's\n + type, and if it is a commit we call `repo_parse_commit()` instead of\n + `parse_object()`. This is quite inefficient though given that we're\n + always uncompressing the object header in order to determine the type.\n + Instead, we can opportunistically search the commit-graph for the object\n + ID: in case it's found, we know it's a commit and can directly fill in\n + the commit object without having to uncompress the object header.\n \n - Expose a new function `parse_commit_in_graph_gently()`, which fills in\n - an object of unknown type in case we find its object ID in the graph.\n - This provides a big performance win in cases where there is a\n - commit-graph available in the repository in case we load lots of\n - references. The following has been executed in a real-world repository\n - with about 2.2 million refs:\n + Expose a new function `lookup_commit_in_graph()`, which tries to find a\n + commit in the commit-graph by ID, and convert `get_reference()` to use\n + this function. This provides a big performance win in cases where we\n + load references in a repository with lots of references pointing to\n + commits. The following has been executed in a real-world repository with\n + about 2.2 million refs:\n \n Benchmark #1: HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n - Time (mean ± σ): 4.508 s ± 0.039 s [User: 4.131 s, System: 0.377 s]\n - Range (min … max): 4.455 s … 4.576 s 10 runs\n + Time (mean ± σ): 4.458 s ± 0.044 s [User: 4.115 s, System: 0.342 s]\n + Range (min … max): 4.409 s … 4.534 s 10 runs\n \n Benchmark #2: HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n - Time (mean ± σ): 3.072 s ± 0.031 s [User: 2.707 s, System: 0.365 s]\n - Range (min … max): 3.040 s … 3.144 s 10 runs\n + Time (mean ± σ): 3.089 s ± 0.015 s [User: 2.768 s, System: 0.321 s]\n + Range (min … max): 3.061 s … 3.105 s 10 runs\n \n Summary\n 'HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev' ran\n - 1.47 ± 0.02 times faster than 'HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev'\n + 1.44 ± 0.02 times faster than 'HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev'\n \n Signed-off-by: Patrick Steinhardt \n \n @@ commit-graph.c: static int find_commit_pos_in_graph(struct commit *item, struct\n \t}\n }\n \n -+int parse_commit_in_graph_gently(struct repository *repo, struct object *object)\n ++struct commit *lookup_commit_in_graph(struct repository *repo, const struct object_id *id)\n +{\n +\tstruct commit *commit;\n +\tuint32_t pos;\n +\n -+\tif (object->parsed) {\n -+\t\tif (object->type != OBJ_COMMIT)\n -+\t\t\treturn -1;\n -+\t\treturn 0;\n -+\t}\n -+\n +\tif (!repo->objects->commit_graph)\n -+\t\treturn -1;\n ++\t\treturn NULL;\n ++\tif (!search_commit_pos_in_graph(id, repo->objects->commit_graph, &pos))\n ++\t\treturn NULL;\n ++\tif (!repo_has_object_file(repo, id))\n ++\t\treturn NULL;\n +\n -+\tif (!search_commit_pos_in_graph(&object->oid, repo->objects->commit_graph, &pos))\n -+\t\treturn -1;\n -+\n -+\tcommit = object_as_type(object, OBJ_COMMIT, 1);\n ++\tcommit = lookup_commit(repo, id);\n +\tif (!commit)\n -+\t\treturn -1;\n -+\tif (!fill_commit_in_graph(repo, commit, repo->objects->commit_graph, pos))\n -+\t\treturn -1;\n ++\t\treturn NULL;\n ++\tif (commit->object.parsed)\n ++\t\treturn commit;\n +\n -+\treturn 0;\n ++\tif (!fill_commit_in_graph(repo, commit, repo->objects->commit_graph, pos))\n ++\t\treturn NULL;\n ++\n ++\treturn commit;\n +}\n +\n static int parse_commit_in_graph_one(struct repository *r,\n @@ commit-graph.h: int open_commit_graph(const char *graph_file, int *fd, struct st\n int parse_commit_in_graph(struct repository *r, struct commit *item);\n \n +/*\n -+ * Given an object of unknown type, try to fill in the object in case it is a\n -+ * commit part of the commit-graph. Returns 0 if the object is a parsed commit\n -+ * or if it could be filled in via the commit graph, otherwise it returns -1.\n ++ * Look up the given commit ID in the commit-graph. This will only return a\n ++ * commit if the ID exists both in the graph and in the object database such\n ++ * that we don't return commits whose object has been pruned. Otherwise, this\n ++ * function returns `NULL`.\n + */\n -+int parse_commit_in_graph_gently(struct repository *repo, struct object *object);\n ++struct commit *lookup_commit_in_graph(struct repository *repo, const struct object_id *id);\n +\n /*\n * It is possible that we loaded commit contents from the commit buffer,\n @@ commit-graph.h: int open_commit_graph(const char *graph_file, int *fd, struct st\n \n ## revision.c ##\n @@ revision.c: static struct object *get_reference(struct rev_info *revs, const char *name,\n - \tstruct object *object = lookup_unknown_object(revs->repo, oid);\n + \t\t\t\t unsigned int flags)\n + {\n + \tstruct object *object;\n ++\tstruct commit *commit;\n \n - \tif (object->type == OBJ_NONE) {\n --\t\tint type = oid_object_info(revs->repo, oid, NULL);\n --\t\tif (type < 0 || !object_as_type(object, type, 1)) {\n -+\t\t/*\n -+\t\t * It's likely that the reference points to a commit, so we\n -+\t\t * first try to look it up via the commit-graph. If successful,\n -+\t\t * then we know it's a commit and don't have to unpack the\n -+\t\t * object header. We still need to assert that the object\n -+\t\t * exists, but given that we don't request any info about the\n -+\t\t * object this is a lot faster than `oid_object_info()`.\n -+\t\t */\n -+\t\tif (parse_commit_in_graph_gently(revs->repo, object) < 0) {\n -+\t\t\tint type = oid_object_info(revs->repo, oid, NULL);\n -+\t\t\tif (type < 0 || !object_as_type(object, type, 1)) {\n -+\t\t\t\tobject = NULL;\n -+\t\t\t\tgoto out;\n -+\t\t\t}\n -+\t\t} else if (!repo_has_object_file(revs->repo, oid)) {\n - \t\t\tobject = NULL;\n - \t\t\tgoto out;\n - \t\t}\n + \t/*\n +-\t * If the repository has commit graphs, repo_parse_commit() avoids\n +-\t * reading the object buffer, so use it whenever possible.\n ++\t * If the repository has commit graphs, we try to opportunistically\n ++\t * look up the object ID in those graphs. Like this, we can avoid\n ++\t * parsing commit data from disk.\n + \t */\n +-\tif (oid_object_info(revs->repo, oid, NULL) == OBJ_COMMIT) {\n +-\t\tstruct commit *c = lookup_commit(revs->repo, oid);\n +-\t\tif (!repo_parse_commit(revs->repo, c))\n +-\t\t\tobject = (struct object *) c;\n +-\t\telse\n +-\t\t\tobject = NULL;\n +-\t} else {\n ++\tcommit = lookup_commit_in_graph(revs->repo, oid);\n ++\tif (commit)\n ++\t\tobject = &commit->object;\n ++\telse\n + \t\tobject = parse_object(revs->repo, oid);\n +-\t}\n + \n + \tif (!object) {\n + \t\tif (revs->ignore_missing)\n-- \n2.32.0\n\n"},{"id":"432296","messageId":"67232910acf4a248654060ad51a17af37acba0fb.1628496539.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1628496538.git.ps@pks.im","subject":"[PATCH v5 1/5] revision: separate walk and unsorted flags","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-09T08:11:46Z","receivedAt":"2021-08-09T08:11:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The `--no-walk` flag supports two modes: either it sorts the revisions\ngiven as input input or it doesn't. This is reflected in a single\n`no_walk` flag, which reflects one of the three states \"walk\", \"don't\nwalk but without sorting\" and \"don't walk but with sorting\".\n\nSplit up the flag into two separate bits, one indicating whether we\nshould walk or not and one indicating whether the input should be sorted\nor not. This will allow us to more easily introduce a new flag\n`--unsorted-input`, which only impacts the sorting bit.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/log.c | 2 +-\n builtin/revert.c | 3 ++-\n revision.c | 9 +++++----\n revision.h | 7 ++-----\n 4 files changed, 10 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/log.c b/builtin/log.c\nindex 3d7717ba5c..f75d87e8d7 100644\n--- a/builtin/log.c\n+++ b/builtin/log.c\n@@ -637,7 +637,7 @@ int cmd_show(int argc, const char **argv, const char *prefix)\n \trepo_init_revisions(the_repository, &rev, prefix);\n \trev.diff = 1;\n \trev.always_show_header = 1;\n-\trev.no_walk = REVISION_WALK_NO_WALK_SORTED;\n+\trev.no_walk = 1;\n \trev.diffopt.stat_width = -1; \t/* Scale to real terminal size */\n \n \tmemset(&opt, 0, sizeof(opt));\ndiff --git a/builtin/revert.c b/builtin/revert.c\nindex 237f2f18d4..2e13660e4b 100644\n--- a/builtin/revert.c\n+++ b/builtin/revert.c\n@@ -191,7 +191,8 @@ static int run_sequencer(int argc, const char **argv, struct replay_opts *opts)\n \t\tstruct setup_revision_opt s_r_opt;\n \t\topts->revs = xmalloc(sizeof(*opts->revs));\n \t\trepo_init_revisions(the_repository, opts->revs, NULL);\n-\t\topts->revs->no_walk = REVISION_WALK_NO_WALK_UNSORTED;\n+\t\topts->revs->no_walk = 1;\n+\t\topts->revs->unsorted_input = 1;\n \t\tif (argc < 2)\n \t\t\tusage_with_options(usage_str, options);\n \t\tif (!strcmp(argv[1], \"-\"))\ndiff --git a/revision.c b/revision.c\nindex cddd0542a6..86bbcd10d2 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -2651,16 +2651,17 @@ static int handle_revision_pseudo_opt(const char *submodule,\n \t} else if (!strcmp(arg, \"--not\")) {\n \t\t*flags ^= UNINTERESTING | BOTTOM;\n \t} else if (!strcmp(arg, \"--no-walk\")) {\n-\t\trevs->no_walk = REVISION_WALK_NO_WALK_SORTED;\n+\t\trevs->no_walk = 1;\n \t} else if (skip_prefix(arg, \"--no-walk=\", &optarg)) {\n \t\t/*\n \t\t * Detached form (\"--no-walk X\" as opposed to \"--no-walk=X\")\n \t\t * not allowed, since the argument is optional.\n \t\t */\n+\t\trevs->no_walk = 1;\n \t\tif (!strcmp(optarg, \"sorted\"))\n-\t\t\trevs->no_walk = REVISION_WALK_NO_WALK_SORTED;\n+\t\t\trevs->unsorted_input = 0;\n \t\telse if (!strcmp(optarg, \"unsorted\"))\n-\t\t\trevs->no_walk = REVISION_WALK_NO_WALK_UNSORTED;\n+\t\t\trevs->unsorted_input = 1;\n \t\telse\n \t\t\treturn error(\"invalid argument to --no-walk\");\n \t} else if (!strcmp(arg, \"--do-walk\")) {\n@@ -3584,7 +3585,7 @@ int prepare_revision_walk(struct rev_info *revs)\n \n \tif (!revs->reflog_info)\n \t\tprepare_to_use_bloom_filter(revs);\n-\tif (revs->no_walk != REVISION_WALK_NO_WALK_UNSORTED)\n+\tif (!revs->unsorted_input)\n \t\tcommit_list_sort_by_date(&revs->commits);\n \tif (revs->no_walk)\n \t\treturn 0;\ndiff --git a/revision.h b/revision.h\nindex fbb068da9f..0c65a760ee 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -79,10 +79,6 @@ struct rev_cmdline_info {\n \t} *rev;\n };\n \n-#define REVISION_WALK_WALK 0\n-#define REVISION_WALK_NO_WALK_SORTED 1\n-#define REVISION_WALK_NO_WALK_UNSORTED 2\n-\n struct oidset;\n struct topo_walk_info;\n \n@@ -129,7 +125,8 @@ struct rev_info {\n \t/* Traversal flags */\n \tunsigned int\tdense:1,\n \t\t\tprune:1,\n-\t\t\tno_walk:2,\n+\t\t\tno_walk:1,\n+\t\t\tunsorted_input:1,\n \t\t\tremove_empty_trees:1,\n \t\t\tsimplify_history:1,\n \t\t\tshow_pulls:1,\n-- \n2.32.0\n\n"},{"id":"432297","messageId":"d3f498a5633d88403381aefca58c8b11e1194831.1628496539.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1628496538.git.ps@pks.im","subject":"[PATCH v5 2/5] connected: do not sort input revisions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-09T08:11:50Z","receivedAt":"2021-08-09T08:11:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In order to compute whether objects reachable from a set of tips are all\nconnected, we do a revision walk with these tips as positive references\nand `--not --all`. `--not --all` will cause the revision walk to load\nall preexisting references as uninteresting, which can be very expensive\nin repositories with many references.\n\nBenchmarking the git-rev-list(1) command highlights that by far the most\nexpensive single phase is initial sorting of the input revisions: after\nall references have been loaded, we first sort commits by author date.\nIn a real-world repository with about 2.2 million references, it makes\nup about 40% of the total runtime of git-rev-list(1).\n\nUltimately, the connectivity check shouldn't really bother about the\norder of input revisions at all. We only care whether we can actually\nwalk all objects until we hit the cut-off point. So sorting the input is\na complete waste of time.\n\nIntroduce a new \"--unsorted-input\" flag to git-rev-list(1) which will\ncause it to not sort the commits and adjust the connectivity check to\nalways pass the flag. This results in the following speedups, executed\nin a clone of gitlab-org/gitlab [1]:\n\n Benchmark #1: git rev-list --objects --quiet --not --all --not $(cat newrev)\n Time (mean ± σ): 7.639 s ± 0.065 s [User: 7.304 s, System: 0.335 s]\n Range (min … max): 7.543 s … 7.742 s 10 runs\n\n Benchmark #2: git rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n Time (mean ± σ): 4.995 s ± 0.044 s [User: 4.657 s, System: 0.337 s]\n Range (min … max): 4.909 s … 5.048 s 10 runs\n\n Summary\n 'git rev-list --unsorted-input --objects --quiet --not --all --not $(cat newrev)' ran\n 1.53 ± 0.02 times faster than 'git rev-list --objects --quiet --not --all --not $newrev'\n\n[1]: https://gitlab.com/gitlab-org/gitlab.git. Note that not all refs\n are visible to clients.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/rev-list-options.txt | 8 +++++++-\n connected.c | 1 +\n revision.c | 9 +++++++++\n t/t6000-rev-list-misc.sh | 31 ++++++++++++++++++++++++++++++\n 4 files changed, 48 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 24569b06d1..b7bd27e171 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -968,6 +968,11 @@ list of the missing objects. Object IDs are prefixed with a ``?'' character.\n \tobjects.\n endif::git-rev-list[]\n \n+--unsorted-input::\n+\tShow commits in the order they were given on the command line instead\n+\tof sorting them in reverse chronological order by commit time. Cannot\n+\tbe combined with `--no-walk` or `--no-walk=sorted`.\n+\n --no-walk[=(sorted|unsorted)]::\n \tOnly show the given commits, but do not traverse their ancestors.\n \tThis has no effect if a range is specified. If the argument\n@@ -975,7 +980,8 @@ endif::git-rev-list[]\n \tgiven on the command line. Otherwise (if `sorted` or no argument\n \twas given), the commits are shown in reverse chronological order\n \tby commit time.\n-\tCannot be combined with `--graph`.\n+\tCannot be combined with `--graph`. Cannot be combined with\n+\t`--unsorted-input` if `sorted` or no argument was given.\n \n --do-walk::\n \tOverrides a previous `--no-walk`.\ndiff --git a/connected.c b/connected.c\nindex b18299fdf0..b5f9523a5f 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -106,6 +106,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \tif (opt->progress)\n \t\tstrvec_pushf(&rev_list.args, \"--progress=%s\",\n \t\t\t _(\"Checking connectivity\"));\n+\tstrvec_push(&rev_list.args, \"--unsorted-input\");\n \n \trev_list.git_cmd = 1;\n \trev_list.env = opt->env;\ndiff --git a/revision.c b/revision.c\nindex 86bbcd10d2..47541407d2 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -2256,6 +2256,10 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg\n \t} else if (!strcmp(arg, \"--author-date-order\")) {\n \t\trevs->sort_order = REV_SORT_BY_AUTHOR_DATE;\n \t\trevs->topo_order = 1;\n+\t} else if (!strcmp(arg, \"--unsorted-input\")) {\n+\t\tif (revs->no_walk)\n+\t\t\tdie(_(\"--unsorted-input is incompatible with --no-walk\"));\n+\t\trevs->unsorted_input = 1;\n \t} else if (!strcmp(arg, \"--early-output\")) {\n \t\trevs->early_output = 100;\n \t\trevs->topo_order = 1;\n@@ -2651,8 +2655,13 @@ static int handle_revision_pseudo_opt(const char *submodule,\n \t} else if (!strcmp(arg, \"--not\")) {\n \t\t*flags ^= UNINTERESTING | BOTTOM;\n \t} else if (!strcmp(arg, \"--no-walk\")) {\n+\t\tif (!revs->no_walk && revs->unsorted_input)\n+\t\t\tdie(_(\"--no-walk is incompatible with --unsorted-input\"));\n \t\trevs->no_walk = 1;\n \t} else if (skip_prefix(arg, \"--no-walk=\", &optarg)) {\n+\t\tif (!revs->no_walk && revs->unsorted_input)\n+\t\t\tdie(_(\"--no-walk is incompatible with --unsorted-input\"));\n+\n \t\t/*\n \t\t * Detached form (\"--no-walk X\" as opposed to \"--no-walk=X\")\n \t\t * not allowed, since the argument is optional.\ndiff --git a/t/t6000-rev-list-misc.sh b/t/t6000-rev-list-misc.sh\nindex 12def7bcbf..ef849e5bc8 100755\n--- a/t/t6000-rev-list-misc.sh\n+++ b/t/t6000-rev-list-misc.sh\n@@ -169,4 +169,35 @@ test_expect_success 'rev-list --count --objects' '\n \ttest_line_count = $count actual\n '\n \n+test_expect_success 'rev-list --unsorted-input results in different sorting' '\n+\tgit rev-list --unsorted-input HEAD HEAD~ >first &&\n+\tgit rev-list --unsorted-input HEAD~ HEAD >second &&\n+\t! test_cmp first second &&\n+\tsort first >first.sorted &&\n+\tsort second >second.sorted &&\n+\ttest_cmp first.sorted second.sorted\n+'\n+\n+test_expect_success 'rev-list --unsorted-input incompatible with --no-walk' '\n+\tcat >expect <<-EOF &&\n+\t\tfatal: --no-walk is incompatible with --unsorted-input\n+\tEOF\n+\ttest_must_fail git rev-list --unsorted-input --no-walk HEAD 2>error &&\n+\ttest_cmp expect error &&\n+\ttest_must_fail git rev-list --unsorted-input --no-walk=sorted HEAD 2>error &&\n+\ttest_cmp expect error &&\n+\ttest_must_fail git rev-list --unsorted-input --no-walk=unsorted HEAD 2>error &&\n+\ttest_cmp expect error &&\n+\n+\tcat >expect <<-EOF &&\n+\t\tfatal: --unsorted-input is incompatible with --no-walk\n+\tEOF\n+\ttest_must_fail git rev-list --no-walk --unsorted-input HEAD 2>error &&\n+\ttest_cmp expect error &&\n+\ttest_must_fail git rev-list --no-walk=sorted --unsorted-input HEAD 2>error &&\n+\ttest_cmp expect error &&\n+\ttest_must_fail git rev-list --no-walk=unsorted --unsorted-input HEAD 2>error &&\n+\ttest_cmp expect error\n+'\n+\n test_done\n-- \n2.32.0\n\n"},{"id":"432298","messageId":"c9a630927b9386bce8766940e5cabd8debe6e0a4.1628496539.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1628496538.git.ps@pks.im","subject":"[PATCH v5 3/5] revision: stop retrieving reference twice","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-09T08:11:54Z","receivedAt":"2021-08-09T08:12:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When queueing up references for the revision walk, `handle_one_ref()`\nwill resolve the reference's object ID via `get_reference()` and then\nqueue the ID as pending object via `add_pending_oid()`. But given that\n`add_pending_oid()` is only a thin wrapper around `add_pending_object()`\nwhich fist calls `get_reference()`, we effectively resolve the reference\ntwice and thus duplicate some of the work.\n\nFix the issue by instead calling `add_pending_object()` directly, which\ntakes the already-resolved object as input. In a repository with lots of\nrefs, this translates into a near 10% speedup:\n\n Benchmark #1: HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n Time (mean ± σ): 5.015 s ± 0.038 s [User: 4.698 s, System: 0.316 s]\n Range (min … max): 4.970 s … 5.089 s 10 runs\n\n Benchmark #2: HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n Time (mean ± σ): 4.606 s ± 0.029 s [User: 4.260 s, System: 0.345 s]\n Range (min … max): 4.565 s … 4.657 s 10 runs\n\n Summary\n 'HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev' ran\n 1.09 ± 0.01 times faster than 'HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev'\n\nSigned-off-by: Patrick Steinhardt \n---\n revision.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/revision.c b/revision.c\nindex 47541407d2..80a59896b9 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1534,7 +1534,7 @@ static int handle_one_ref(const char *path, const struct object_id *oid,\n \n \tobject = get_reference(cb->all_revs, path, oid, cb->all_flags);\n \tadd_rev_cmdline(cb->all_revs, object, path, REV_CMD_REF, cb->all_flags);\n-\tadd_pending_oid(cb->all_revs, path, oid, cb->all_flags);\n+\tadd_pending_object(cb->all_revs, object, path);\n \treturn 0;\n }\n \n-- \n2.32.0\n\n"},{"id":"432299","messageId":"bc89325fdf170bde142745a10602207c0952d22a.1628496539.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1628496538.git.ps@pks.im","subject":"[PATCH v5 4/5] commit-graph: split out function to search commit position","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-09T08:11:59Z","receivedAt":"2021-08-09T08:12:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The function `find_commit_in_graph()` assumes that the caller has passed\nan object which was already determined to be a commit given that it will\naccess the commit's graph position, which is stored in a commit slab. In\na subsequent patch, we want to search for an object ID though without\nknowing whether it is a commit or not, which is not currently possible.\n\nSplit out the logic to search the commit graph for a given object ID to\nprepare for this change. This commit also renames the function to\n`find_commit_pos_in_graph()`, which more accurately reflects what this\nfunction does. Furthermore, in order to allow for the searched object ID\nto be const, we need to adjust `bsearch_graph()`'s signature to accept a\nconstant object ID as input, too.\n\nSigned-off-by: Patrick Steinhardt \n---\n commit-graph.c | 55 +++++++++++++++++++++++++++-----------------------\n 1 file changed, 30 insertions(+), 25 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 3860a0d847..8c4c7262c8 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -723,7 +723,7 @@ void close_commit_graph(struct raw_object_store *o)\n \to->commit_graph = NULL;\n }\n \n-static int bsearch_graph(struct commit_graph *g, struct object_id *oid, uint32_t *pos)\n+static int bsearch_graph(struct commit_graph *g, const struct object_id *oid, uint32_t *pos)\n {\n \treturn bsearch_hash(oid->hash, g->chunk_oid_fanout,\n \t\t\t g->chunk_oid_lookup, g->hash_len, pos);\n@@ -864,25 +864,30 @@ static int fill_commit_in_graph(struct repository *r,\n \treturn 1;\n }\n \n-static int find_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t *pos)\n+static int search_commit_pos_in_graph(const struct object_id *id, struct commit_graph *g, uint32_t *pos)\n+{\n+\tstruct commit_graph *cur_g = g;\n+\tuint32_t lex_index;\n+\n+\twhile (cur_g && !bsearch_graph(cur_g, id, &lex_index))\n+\t\tcur_g = cur_g->base_graph;\n+\n+\tif (cur_g) {\n+\t\t*pos = lex_index + cur_g->num_commits_in_base;\n+\t\treturn 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static int find_commit_pos_in_graph(struct commit *item, struct commit_graph *g, uint32_t *pos)\n {\n \tuint32_t graph_pos = commit_graph_position(item);\n \tif (graph_pos != COMMIT_NOT_FROM_GRAPH) {\n \t\t*pos = graph_pos;\n \t\treturn 1;\n \t} else {\n-\t\tstruct commit_graph *cur_g = g;\n-\t\tuint32_t lex_index;\n-\n-\t\twhile (cur_g && !bsearch_graph(cur_g, &(item->object.oid), &lex_index))\n-\t\t\tcur_g = cur_g->base_graph;\n-\n-\t\tif (cur_g) {\n-\t\t\t*pos = lex_index + cur_g->num_commits_in_base;\n-\t\t\treturn 1;\n-\t\t}\n-\n-\t\treturn 0;\n+\t\treturn search_commit_pos_in_graph(&item->object.oid, g, pos);\n \t}\n }\n \n@@ -895,7 +900,7 @@ static int parse_commit_in_graph_one(struct repository *r,\n \tif (item->object.parsed)\n \t\treturn 1;\n \n-\tif (find_commit_in_graph(item, g, &pos))\n+\tif (find_commit_pos_in_graph(item, g, &pos))\n \t\treturn fill_commit_in_graph(r, item, g, pos);\n \n \treturn 0;\n@@ -921,7 +926,7 @@ void load_commit_graph_info(struct repository *r, struct commit *item)\n \tuint32_t pos;\n \tif (!prepare_commit_graph(r))\n \t\treturn;\n-\tif (find_commit_in_graph(item, r->objects->commit_graph, &pos))\n+\tif (find_commit_pos_in_graph(item, r->objects->commit_graph, &pos))\n \t\tfill_commit_graph_info(item, r->objects->commit_graph, pos);\n }\n \n@@ -1091,9 +1096,9 @@ static int write_graph_chunk_data(struct hashfile *f,\n \t\t\t\tedge_value += ctx->new_num_commits_in_base;\n \t\t\telse if (ctx->new_base_graph) {\n \t\t\t\tuint32_t pos;\n-\t\t\t\tif (find_commit_in_graph(parent->item,\n-\t\t\t\t\t\t\t ctx->new_base_graph,\n-\t\t\t\t\t\t\t &pos))\n+\t\t\t\tif (find_commit_pos_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n \t\t\t\t\tedge_value = pos;\n \t\t\t}\n \n@@ -1122,9 +1127,9 @@ static int write_graph_chunk_data(struct hashfile *f,\n \t\t\t\tedge_value += ctx->new_num_commits_in_base;\n \t\t\telse if (ctx->new_base_graph) {\n \t\t\t\tuint32_t pos;\n-\t\t\t\tif (find_commit_in_graph(parent->item,\n-\t\t\t\t\t\t\t ctx->new_base_graph,\n-\t\t\t\t\t\t\t &pos))\n+\t\t\t\tif (find_commit_pos_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n \t\t\t\t\tedge_value = pos;\n \t\t\t}\n \n@@ -1235,9 +1240,9 @@ static int write_graph_chunk_extra_edges(struct hashfile *f,\n \t\t\t\tedge_value += ctx->new_num_commits_in_base;\n \t\t\telse if (ctx->new_base_graph) {\n \t\t\t\tuint32_t pos;\n-\t\t\t\tif (find_commit_in_graph(parent->item,\n-\t\t\t\t\t\t\t ctx->new_base_graph,\n-\t\t\t\t\t\t\t &pos))\n+\t\t\t\tif (find_commit_pos_in_graph(parent->item,\n+\t\t\t\t\t\t\t ctx->new_base_graph,\n+\t\t\t\t\t\t\t &pos))\n \t\t\t\t\tedge_value = pos;\n \t\t\t}\n \n-- \n2.32.0\n\n"},{"id":"432300","messageId":"fdb1fa9d574a27500df7f16a165997a41a8cc8e8.1628496539.git.ps@pks.im","threadId":"56018","inReplyTo":"cover.1628496538.git.ps@pks.im","subject":"[PATCH v5 5/5] revision: avoid hitting packfiles when commits are in commit-graph","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2021-08-09T08:12:03Z","receivedAt":"2021-08-09T08:12:08Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When queueing references in git-rev-list(1), we try to optimize parsing\nof commits via the commit-graph. To do so, we first look up the object's\ntype, and if it is a commit we call `repo_parse_commit()` instead of\n`parse_object()`. This is quite inefficient though given that we're\nalways uncompressing the object header in order to determine the type.\nInstead, we can opportunistically search the commit-graph for the object\nID: in case it's found, we know it's a commit and can directly fill in\nthe commit object without having to uncompress the object header.\n\nExpose a new function `lookup_commit_in_graph()`, which tries to find a\ncommit in the commit-graph by ID, and convert `get_reference()` to use\nthis function. This provides a big performance win in cases where we\nload references in a repository with lots of references pointing to\ncommits. The following has been executed in a real-world repository with\nabout 2.2 million refs:\n\n Benchmark #1: HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n Time (mean ± σ): 4.458 s ± 0.044 s [User: 4.115 s, System: 0.342 s]\n Range (min … max): 4.409 s … 4.534 s 10 runs\n\n Benchmark #2: HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev\n Time (mean ± σ): 3.089 s ± 0.015 s [User: 2.768 s, System: 0.321 s]\n Range (min … max): 3.061 s … 3.105 s 10 runs\n\n Summary\n 'HEAD: rev-list --unsorted-input --objects --quiet --not --all --not $newrev' ran\n 1.44 ± 0.02 times faster than 'HEAD~: rev-list --unsorted-input --objects --quiet --not --all --not $newrev'\n\nSigned-off-by: Patrick Steinhardt \n---\n commit-graph.c | 24 ++++++++++++++++++++++++\n commit-graph.h | 8 ++++++++\n revision.c | 18 ++++++++----------\n 3 files changed, 40 insertions(+), 10 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8c4c7262c8..00614acd65 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -891,6 +891,30 @@ static int find_commit_pos_in_graph(struct commit *item, struct commit_graph *g,\n \t}\n }\n \n+struct commit *lookup_commit_in_graph(struct repository *repo, const struct object_id *id)\n+{\n+\tstruct commit *commit;\n+\tuint32_t pos;\n+\n+\tif (!repo->objects->commit_graph)\n+\t\treturn NULL;\n+\tif (!search_commit_pos_in_graph(id, repo->objects->commit_graph, &pos))\n+\t\treturn NULL;\n+\tif (!repo_has_object_file(repo, id))\n+\t\treturn NULL;\n+\n+\tcommit = lookup_commit(repo, id);\n+\tif (!commit)\n+\t\treturn NULL;\n+\tif (commit->object.parsed)\n+\t\treturn commit;\n+\n+\tif (!fill_commit_in_graph(repo, commit, repo->objects->commit_graph, pos))\n+\t\treturn NULL;\n+\n+\treturn commit;\n+}\n+\n static int parse_commit_in_graph_one(struct repository *r,\n \t\t\t\t struct commit_graph *g,\n \t\t\t\t struct commit *item)\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 96c24fb577..04a94e1830 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -40,6 +40,14 @@ int open_commit_graph(const char *graph_file, int *fd, struct stat *st);\n */\n int parse_commit_in_graph(struct repository *r, struct commit *item);\n \n+/*\n+ * Look up the given commit ID in the commit-graph. This will only return a\n+ * commit if the ID exists both in the graph and in the object database such\n+ * that we don't return commits whose object has been pruned. Otherwise, this\n+ * function returns `NULL`.\n+ */\n+struct commit *lookup_commit_in_graph(struct repository *repo, const struct object_id *id);\n+\n /*\n * It is possible that we loaded commit contents from the commit buffer,\n * but we also want to ensure the commit-graph content is correctly\ndiff --git a/revision.c b/revision.c\nindex 80a59896b9..0dabb5a0bc 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -360,20 +360,18 @@ static struct object *get_reference(struct rev_info *revs, const char *name,\n \t\t\t\t unsigned int flags)\n {\n \tstruct object *object;\n+\tstruct commit *commit;\n \n \t/*\n-\t * If the repository has commit graphs, repo_parse_commit() avoids\n-\t * reading the object buffer, so use it whenever possible.\n+\t * If the repository has commit graphs, we try to opportunistically\n+\t * look up the object ID in those graphs. Like this, we can avoid\n+\t * parsing commit data from disk.\n \t */\n-\tif (oid_object_info(revs->repo, oid, NULL) == OBJ_COMMIT) {\n-\t\tstruct commit *c = lookup_commit(revs->repo, oid);\n-\t\tif (!repo_parse_commit(revs->repo, c))\n-\t\t\tobject = (struct object *) c;\n-\t\telse\n-\t\t\tobject = NULL;\n-\t} else {\n+\tcommit = lookup_commit_in_graph(revs->repo, oid);\n+\tif (commit)\n+\t\tobject = &commit->object;\n+\telse\n \t\tobject = parse_object(revs->repo, oid);\n-\t}\n \n \tif (!object) {\n \t\tif (revs->ignore_missing)\n-- \n2.32.0\n\n"}]}