{"thread":{"id":"47724","subject":"[PATCH v2 00/14] Serialized Git Commit Graph","startedAt":"2018-01-30T21:40:10Z","lastAt":"2018-04-07T22:37:28Z","messageCount":146,"participants":["Derrick Stolee","Stefan Beller","Igor Djordjevic","Jonathan Tan","SZEDER Gábor","Junio C Hamano","Jeff King","Jakub Narebski"],"isPatch":true,"patchVersion":2,"patchTotal":14},"messages":[{"id":"337828","messageId":"1517348383-112294-1-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":null,"subject":"[PATCH v2 00/14] Serialized Git Commit Graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-30T21:39:29Z","receivedAt":"2018-01-30T21:40:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Thanks to everyone who gave comments on v1. I tried my best to respond to\nall of the feedback, but may have missed some while I was doing several\nrenames, including:\n\n* builtin/graph.c -> builtin/commit-graph.c\n* packed-graph.[c|h] -> commit-graph.[c|h]\n* t/t5319-graph.sh -> t/t5318-commit-graph.sh\n\nBecause of these renames (and several type/function renames) the diff\nis too large to conveniently share here.\n\nSome issues that came up and are addressed:\n\n* Use instead of when referring to the graph-.graph\n filenames and the contents of graph-head.\n* 32-bit timestamps will not cause undefined behavior.\n* timestamp_t is unsigned, so they are never negative.\n* The config setting \"core.commitgraph\" now only controls consuming the\n graph during normal operations and will not block the commit-graph\n plumbing command.\n* The --stdin-commits is better about sanitizing the input for strings\n that do not parse to OIDs or are OIDs for non-commit objects.\n\nOne unresolved comment that I would like consensus on is the use of\nglobals to store the config setting and the graph state. I'm currently\nusing the pattern from packed_git instead of putting these values in\nthe_repository. However, we want to eventually remove globals like\npacked_git. Should I deviate from the pattern _now_ in order to keep\nthe problem from growing, or should I keep to the known pattern?\n\nFinally, I tried to clean up my incorrect style as I was recreating\nthese commits. Feel free to be merciless in style feedback now that the\narchitecture is more stable.\n\nThanks,\n-Stolee\n\n-- >8 --\n\nAs promised [1], this patch contains a way to serialize the commit graph.\nThe current implementation defines a new file format to store the graph\nstructure (parent relationships) and basic commit metadata (commit date,\nroot tree OID) in order to prevent parsing raw commits while performing\nbasic graph walks. For example, we do not need to parse the full commit\nwhen performing these walks:\n\n* 'git log --topo-order -1000' walks all reachable commits to avoid\n incorrect topological orders, but only needs the commit message for\n the top 1000 commits.\n\n* 'git merge-base ' may walk many commits to find the correct\n boundary between the commits reachable from A and those reachable\n from B. No commit messages are needed.\n\n* 'git branch -vv' checks ahead/behind status for all local branches\n compared to their upstream remote branches. This is essentially as\n hard as computing merge bases for each.\n\nThe current patch speeds up these calculations by injecting a check in\nparse_commit_gently() to check if there is a graph file and using that\nto provide the required metadata to the struct commit.\n\nThe file format has room to store generation numbers, which will be\nprovided as a patch after this framework is merged. Generation numbers\nare referenced by the design document but not implemented in order to\nmake the current patch focus on the graph construction process. Once\nthat is stable, it will be easier to add generation numbers and make\ngraph walks aware of generation numbers one-by-one.\n\nHere are some performance results for a copy of the Linux repository\nwhere 'master' has 704,766 reachable commits and is behind 'origin/master'\nby 19,610 commits.\n\n| Command | Before | After | Rel % |\n|----------------------------------|--------|--------|-------|\n| log --oneline --topo-order -1000 | 5.9s | 0.7s | -88% |\n| branch -vv | 0.42s | 0.27s | -35% |\n| rev-list --all | 6.4s | 1.0s | -84% |\n| rev-list --all --objects | 32.6s | 27.6s | -15% |\n\nTo test this yourself, run the following on your repo:\n\n git config core.commitgraph true\n git show-ref -s | git graph --write --update-head --stdin-commits\n\nThe second command writes a commit graph file containing every commit\nreachable from your refs. Now, all git commands that walk commits will\ncheck your graph first before consulting the ODB. You can run your own\nperformance comparisions by toggling the 'core.commitgraph' setting.\n\n[1] https://public-inbox.org/git/d154319e-bb9e-b300-7c37-27b1dcd2a2ce@jeffhostetler.com/\n Re: What's cooking in git.git (Jan 2018, #03; Tue, 23)\n\n[2] https://github.com/derrickstolee/git/pull/2\n A GitHub pull request containing the latest version of this patch.\n\nDerrick Stolee (14):\n commit-graph: add format document\n graph: add commit graph design document\n commit-graph: create git-commit-graph builtin\n commit-graph: implement construct_commit_graph()\n commit-graph: implement git-commit-graph --write\n commit-graph: implement git-commit-graph --read\n commit-graph: implement git-commit-graph --update-head\n commit-graph: implement git-commit-graph --clear\n commit-graph: teach git-commit-graph --delete-expired\n commit-graph: add core.commitgraph setting\n commit: integrate commit graph with commit parsing\n commit-graph: read only from specific pack-indexes\n commit-graph: close under reachability\n commit-graph: build graph from starting commits\n\n .gitignore | 1 +\n Documentation/config.txt | 3 +\n Documentation/git-commit-graph.txt | 100 +++\n Documentation/technical/commit-graph-format.txt | 89 +++\n Documentation/technical/commit-graph.txt | 189 ++++++\n Makefile | 2 +\n alloc.c | 1 +\n builtin.h | 1 +\n builtin/commit-graph.c | 229 +++++++\n cache.h | 1 +\n command-list.txt | 1 +\n commit-graph.c | 841 ++++++++++++++++++++++++\n commit-graph.h | 69 ++\n commit.c | 10 +-\n commit.h | 4 +\n config.c | 5 +\n environment.c | 1 +\n git.c | 1 +\n log-tree.c | 3 +-\n packfile.c | 4 +-\n packfile.h | 2 +\n t/t5318-commit-graph.sh | 272 ++++++++\n 22 files changed, 1824 insertions(+), 5 deletions(-)\n create mode 100644 Documentation/git-commit-graph.txt\n create mode 100644 Documentation/technical/commit-graph-format.txt\n create mode 100644 Documentation/technical/commit-graph.txt\n create mode 100644 builtin/commit-graph.c\n create mode 100644 commit-graph.c\n create mode 100644 commit-graph.h\n create mode 100755 t/t5318-commit-graph.sh\n\n-- \n2.16.0\n\n"},{"id":"337829","messageId":"1517348383-112294-2-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1517348383-112294-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v2 01/14] commit-graph: add format document","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-30T21:39:30Z","receivedAt":"2018-01-30T21:40:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Add document specifying the binary format for commit graphs. This\nformat allows for:\n\n* New versions.\n* New hash functions and hash lengths.\n* Optional extensions.\n\nBasic header information is followed by a binary table of contents\ninto \"chunks\" that include:\n\n* An ordered list of commit object IDs.\n* A 256-entry fanout into that list of OIDs.\n* A list of metadata for the commits.\n* A list of \"large edges\" to enable octopus merges.\n\nThe format automatically includes two parent positions for every\ncommit. This favors speed over space, since using only one position\nper commit would cause an extra level of indirection for every merge\ncommit. (Octopus merges suffer from this indirection, but they are\nvery rare.)\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph-format.txt | 89 +++++++++++++++++++++++++\n 1 file changed, 89 insertions(+)\n create mode 100644 Documentation/technical/commit-graph-format.txt\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nnew file mode 100644\nindex 0000000000..8a987c7aa9\n--- /dev/null\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -0,0 +1,89 @@\n+Git commit graph format\n+=======================\n+\n+The Git commit graph stores a list of commit OIDs and some associated\n+metadata, including:\n+\n+- The generation number of the commit. Commits with no parents have\n+ generation number 1; commits with parents have generation number\n+ one more than the maximum generation number of its parents. We\n+ reserve zero as special, and can be used to mark a generation\n+ number invalid or as \"not computed\".\n+\n+- The root tree OID.\n+\n+- The commit date.\n+\n+- The parents of the commit, stored using positional references within\n+ the graph file.\n+\n+== graph-*.graph files have the following format:\n+\n+In order to allow extensions that add extra data to the graph, we organize\n+the body into \"chunks\" and provide a binary lookup table at the beginning\n+of the body. The header includes certain values, such as number of chunks,\n+hash lengths and types.\n+\n+All 4-byte numbers are in network order.\n+\n+HEADER:\n+\n+ 4-byte signature:\n+ The signature is: {'C', 'G', 'P', 'H'}\n+\n+ 1-byte version number:\n+ Currently, the only valid version is 1.\n+\n+ 1-byte Object Id Version (1 = SHA-1)\n+\n+ 1-byte Object Id Length (H)\n+\n+ 1-byte number (C) of \"chunks\"\n+\n+CHUNK LOOKUP:\n+\n+ (C + 1) * 12 bytes listing the table of contents for the chunks:\n+ First 4 bytes describe chunk id. Value 0 is a terminating label.\n+ Other 8 bytes provide offset in current file for chunk to start.\n+ (Chunks are ordered contiguously in the file, so you can infer\n+ the length using the next chunk position if necessary.)\n+\n+ The remaining data in the body is described one chunk at a time, and\n+ these chunks may be given in any order. Chunks are required unless\n+ otherwise specified.\n+\n+CHUNK DATA:\n+\n+ OID Fanout (ID: {'O', 'I', 'D', 'F'}) (256 * 4 bytes)\n+ The ith entry, F[i], stores the number of OIDs with first\n+ byte at most i. Thus F[255] stores the total\n+ number of commits (N).\n+\n+ OID Lookup (ID: {'O', 'I', 'D', 'L'}) (N * H bytes)\n+ The OIDs for all commits in the graph, sorted in ascending order.\n+\n+ Commit Data (ID: {'C', 'G', 'E', 'T' }) (N * (H + 16) bytes)\n+ * The first H bytes are for the OID of the root tree.\n+ * The next 8 bytes are for the int-ids of the first two parents\n+ of the ith commit. Stores value 0xffffffff if no parent in that\n+ position. If there are more than two parents, the second value\n+ has its most-significant bit on and the other bits store an array\n+ position into the Large Edge List chunk.\n+ * The next 8 bytes store the generation number of the commit and\n+ the commit time in seconds since EPOCH. The generation number\n+ uses the higher 30 bits of the first 4 bytes, while the commit\n+ time uses the 32 bits of the second 4 bytes, along with the lowest\n+ 2 bits of the lowest byte, storing the 33rd and 34th bit of the\n+ commit time.\n+\n+ Large Edge List (ID: {'E', 'D', 'G', 'E'})\n+ This list of 4-byte values store the second through nth parents for\n+ all octopus merges. The second parent value in the commit data is a\n+ negative number pointing into this list. Then iterate through this\n+ list starting at that position until reaching a value with the most-\n+ significant bit on. The other bits correspond to the int-id of the\n+ last parent. This chunk should always be present, but may be empty.\n+\n+TRAILER:\n+\n+\tH-byte HASH-checksum of all of the above.\n-- \n2.16.0.15.g9c3cf44.dirty\n\n"},{"id":"337830","messageId":"1517348383-112294-4-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1517348383-112294-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v2 03/14] commit-graph: create git-commit-graph builtin","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-30T21:39:32Z","receivedAt":"2018-01-30T21:40:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach git the 'commit-graph' builtin that will be used for writing and\nreading packed graph files. The current implementation is mostly\nempty, except for a '--pack-dir' option.\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-commit-graph.txt | 7 +++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/commit-graph.c | 33 +++++++++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n 7 files changed, 45 insertions(+)\n create mode 100644 Documentation/git-commit-graph.txt\n create mode 100644 builtin/commit-graph.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 833ef3b0b7..e82f90184d 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -34,6 +34,7 @@\n /git-clone\n /git-column\n /git-commit\n+/git-commit-graph\n /git-commit-tree\n /git-config\n /git-count-objects\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nnew file mode 100644\nindex 0000000000..c8ea548dfb\n--- /dev/null\n+++ b/Documentation/git-commit-graph.txt\n@@ -0,0 +1,7 @@\n+git-commit-graph(1)\n+============\n+\n+NAME\n+----\n+git-commit-graph - Write and verify Git commit graphs (.graph files)\n+\ndiff --git a/Makefile b/Makefile\nindex 1a9b23b679..aee5d3f7b9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -965,6 +965,7 @@ BUILTIN_OBJS += builtin/for-each-ref.o\n BUILTIN_OBJS += builtin/fsck.o\n BUILTIN_OBJS += builtin/gc.o\n BUILTIN_OBJS += builtin/get-tar-commit-id.o\n+BUILTIN_OBJS += builtin/commit-graph.o\n BUILTIN_OBJS += builtin/grep.o\n BUILTIN_OBJS += builtin/hash-object.o\n BUILTIN_OBJS += builtin/help.o\ndiff --git a/builtin.h b/builtin.h\nindex 42378f3aa4..079855b6d4 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -149,6 +149,7 @@ extern int cmd_clone(int argc, const char **argv, const char *prefix);\n extern int cmd_clean(int argc, const char **argv, const char *prefix);\n extern int cmd_column(int argc, const char **argv, const char *prefix);\n extern int cmd_commit(int argc, const char **argv, const char *prefix);\n+extern int cmd_commit_graph(int argc, const char **argv, const char *prefix);\n extern int cmd_commit_tree(int argc, const char **argv, const char *prefix);\n extern int cmd_config(int argc, const char **argv, const char *prefix);\n extern int cmd_count_objects(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nnew file mode 100644\nindex 0000000000..2104550d25\n--- /dev/null\n+++ b/builtin/commit-graph.c\n@@ -0,0 +1,33 @@\n+#include \"builtin.h\"\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"dir.h\"\n+#include \"git-compat-util.h\"\n+#include \"lockfile.h\"\n+#include \"packfile.h\"\n+#include \"parse-options.h\"\n+\n+static char const * const builtin_commit_graph_usage[] = {\n+\tN_(\"git commit-graph [--pack-dir ]\"),\n+\tNULL\n+};\n+\n+static struct opts_commit_graph {\n+\tconst char *pack_dir;\n+} opts;\n+\n+int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic struct option builtin_commit_graph_options[] = {\n+\t\t{ OPTION_STRING, 'p', \"pack-dir\", &opts.pack_dir,\n+\t\t\tN_(\"dir\"),\n+\t\t\tN_(\"The pack directory to store the graph\") },\n+\t\tOPT_END(),\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_commit_graph_usage,\n+\t\t\t\t builtin_commit_graph_options);\n+\n+\treturn 0;\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex a1fad28fd8..835c5890be 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -34,6 +34,7 @@ git-clean mainporcelain\n git-clone mainporcelain init\n git-column purehelpers\n git-commit mainporcelain history\n+git-commit-graph plumbingmanipulators\n git-commit-tree plumbingmanipulators\n git-config ancillarymanipulators\n git-count-objects ancillaryinterrogators\ndiff --git a/git.c b/git.c\nindex c870b9719c..c7b5adae7b 100644\n--- a/git.c\n+++ b/git.c\n@@ -388,6 +388,7 @@ static struct cmd_struct commands[] = {\n \t{ \"clone\", cmd_clone },\n \t{ \"column\", cmd_column, RUN_SETUP_GENTLY },\n \t{ \"commit\", cmd_commit, RUN_SETUP | NEED_WORK_TREE },\n+\t{ \"commit-graph\", cmd_commit_graph, RUN_SETUP },\n \t{ \"commit-tree\", cmd_commit_tree, RUN_SETUP },\n \t{ \"config\", cmd_config, RUN_SETUP_GENTLY },\n \t{ \"count-objects\", cmd_count_objects, RUN_SETUP },\n-- \n2.16.0.15.g9c3cf44.dirty\n\n"},{"id":"337831","messageId":"1517348383-112294-7-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1517348383-112294-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v2 06/14] commit-graph: implement git-commit-graph --read","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-30T21:39:35Z","receivedAt":"2018-01-30T21:40:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach git-commit-graph to read commit graph files and summarize their contents.\n\nUse the --read option to verify the contents of a commit graph file in the\ntests.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 7 ++\n builtin/commit-graph.c | 55 +++++++++++++++\n commit-graph.c | 138 ++++++++++++++++++++++++++++++++++++-\n commit-graph.h | 25 +++++++\n t/t5318-commit-graph.sh | 28 ++++++--\n 5 files changed, 247 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 3f3790d9a8..09aeaf6c82 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -10,6 +10,7 @@ SYNOPSIS\n --------\n [verse]\n 'git commit-graph' --write [--pack-dir ]\n+'git commit-graph' --read [--pack-dir ]\n \n EXAMPLES\n --------\n@@ -20,6 +21,12 @@ EXAMPLES\n $ git commit-graph --write\n ------------------------------------------------\n \n+* Read basic information from a graph file.\n++\n+------------------------------------------------\n+$ git commit-graph --read --graph-hash=\n+------------------------------------------------\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 7affd512f1..218740b1f8 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -10,15 +10,58 @@\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--pack-dir ]\"),\n+\tN_(\"git commit-graph --read [--graph-hash=]\"),\n \tN_(\"git commit-graph --write [--pack-dir ]\"),\n \tNULL\n };\n \n static struct opts_commit_graph {\n \tconst char *pack_dir;\n+\tint read;\n+\tconst char *graph_hash;\n \tint write;\n } opts;\n \n+static int graph_read(void)\n+{\n+\tstruct object_id graph_hash;\n+\tstruct commit_graph *graph = 0;\n+\tconst char *graph_file;\n+\n+\tif (opts.graph_hash && strlen(opts.graph_hash) == GIT_MAX_HEXSZ)\n+\t\tget_oid_hex(opts.graph_hash, &graph_hash);\n+\telse\n+\t\tdie(\"no graph hash specified\");\n+\n+\tgraph_file = get_commit_graph_filename_hash(opts.pack_dir, &graph_hash);\n+\tgraph = load_commit_graph_one(graph_file, opts.pack_dir);\n+\n+\tif (!graph)\n+\t\tdie(\"graph file %s does not exist\", graph_file);\n+\n+\tprintf(\"header: %08x %02x %02x %02x %02x\\n\",\n+\t\tntohl(graph->hdr->graph_signature),\n+\t\tgraph->hdr->graph_version,\n+\t\tgraph->hdr->hash_version,\n+\t\tgraph->hdr->hash_len,\n+\t\tgraph->hdr->num_chunks);\n+\tprintf(\"num_commits: %u\\n\", graph->num_commits);\n+\tprintf(\"chunks:\");\n+\n+\tif (graph->chunk_oid_fanout)\n+\t\tprintf(\" oid_fanout\");\n+\tif (graph->chunk_oid_lookup)\n+\t\tprintf(\" oid_lookup\");\n+\tif (graph->chunk_commit_data)\n+\t\tprintf(\" commit_metadata\");\n+\tif (graph->chunk_large_edges)\n+\t\tprintf(\" large_edges\");\n+\tprintf(\"\\n\");\n+\n+\tprintf(\"pack_dir: %s\\n\", graph->pack_dir);\n+\treturn 0;\n+}\n+\n static int graph_write(void)\n {\n \tstruct object_id *graph_hash = construct_commit_graph(opts.pack_dir);\n@@ -36,8 +79,14 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\t{ OPTION_STRING, 'p', \"pack-dir\", &opts.pack_dir,\n \t\t\tN_(\"dir\"),\n \t\t\tN_(\"The pack directory to store the graph\") },\n+\t\tOPT_BOOL('r', \"read\", &opts.read,\n+\t\t\tN_(\"read graph file\")),\n \t\tOPT_BOOL('w', \"write\", &opts.write,\n \t\t\tN_(\"write commit graph file\")),\n+\t\t{ OPTION_STRING, 'H', \"graph-hash\", &opts.graph_hash,\n+\t\t\tN_(\"hash\"),\n+\t\t\tN_(\"A hash for a specific graph file in the pack-dir.\"),\n+\t\t\tPARSE_OPT_OPTARG, NULL, (intptr_t) \"\" },\n \t\tOPT_END(),\n \t};\n \n@@ -49,6 +98,10 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\t\t builtin_commit_graph_options,\n \t\t\t builtin_commit_graph_usage, 0);\n \n+\tif (opts.write + opts.read > 1)\n+\t\tusage_with_options(builtin_commit_graph_usage,\n+\t\t\t\t builtin_commit_graph_options);\n+\n \tif (!opts.pack_dir) {\n \t\tstruct strbuf path = STRBUF_INIT;\n \t\tstrbuf_addstr(&path, get_object_directory());\n@@ -56,6 +109,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\topts.pack_dir = strbuf_detach(&path, NULL);\n \t}\n \n+\tif (opts.read)\n+\t\treturn graph_read();\n \tif (opts.write)\n \t\treturn graph_write();\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex db2b7390c7..622a650259 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -48,6 +48,142 @@ char* get_commit_graph_filename_hash(const char *pack_dir,\n \treturn strbuf_detach(&head_path, &len);\n }\n \n+static struct commit_graph *alloc_commit_graph(int extra)\n+{\n+\tstruct commit_graph *g = xmalloc(st_add(sizeof(*g), extra));\n+\tmemset(g, 0, sizeof(*g));\n+\tg->graph_fd = -1;\n+\n+\treturn g;\n+}\n+\n+int close_commit_graph(struct commit_graph *g)\n+{\n+\tif (g->graph_fd < 0)\n+\t\treturn 0;\n+\n+\tmunmap((void *)g->data, g->data_len);\n+\tg->data = 0;\n+\n+\tclose(g->graph_fd);\n+\tg->graph_fd = -1;\n+\n+\treturn 1;\n+}\n+\n+static void free_commit_graph(struct commit_graph **g)\n+{\n+\tif (!g || !*g)\n+\t\treturn;\n+\n+\tclose_commit_graph(*g);\n+\n+\tfree(*g);\n+\t*g = NULL;\n+}\n+\n+struct commit_graph *load_commit_graph_one(const char *graph_file, const char *pack_dir)\n+{\n+\tvoid *graph_map;\n+\tconst unsigned char *data;\n+\tstruct commit_graph_header *hdr;\n+\tsize_t graph_size;\n+\tstruct stat st;\n+\tuint32_t i;\n+\tstruct commit_graph *graph;\n+\tint fd = git_open(graph_file);\n+\tuint64_t last_chunk_offset;\n+\tuint32_t last_chunk_id;\n+\n+\tif (fd < 0)\n+\t\treturn 0;\n+\tif (fstat(fd, &st)) {\n+\t\tclose(fd);\n+\t\treturn 0;\n+\t}\n+\tgraph_size = xsize_t(st.st_size);\n+\n+\tif (graph_size < GRAPH_MIN_SIZE) {\n+\t\tclose(fd);\n+\t\tdie(\"graph file %s is too small\", graph_file);\n+\t}\n+\tgraph_map = xmmap(NULL, graph_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tdata = (const unsigned char *)graph_map;\n+\n+\thdr = graph_map;\n+\tif (ntohl(hdr->graph_signature) != GRAPH_SIGNATURE) {\n+\t\tuint32_t signature = ntohl(hdr->graph_signature);\n+\t\tmunmap(graph_map, graph_size);\n+\t\tclose(fd);\n+\t\tdie(\"graph signature %X does not match signature %X\",\n+\t\t\tsignature, GRAPH_SIGNATURE);\n+\t}\n+\tif (hdr->graph_version != GRAPH_VERSION) {\n+\t\tunsigned char version = hdr->graph_version;\n+\t\tmunmap(graph_map, graph_size);\n+\t\tclose(fd);\n+\t\tdie(\"graph version %X does not match version %X\",\n+\t\t\tversion, GRAPH_VERSION);\n+\t}\n+\n+\tgraph = alloc_commit_graph(strlen(pack_dir) + 1);\n+\n+\tgraph->hdr = hdr;\n+\tgraph->graph_fd = fd;\n+\tgraph->data = graph_map;\n+\tgraph->data_len = graph_size;\n+\n+\tlast_chunk_id = 0;\n+\tlast_chunk_offset = (uint64_t)sizeof(*hdr);\n+\tfor (i = 0; i < hdr->num_chunks; i++) {\n+\t\tuint32_t chunk_id = ntohl(*(uint32_t*)(data + sizeof(*hdr) + 12 * i));\n+\t\tuint64_t chunk_offset1 = ntohl(*(uint32_t*)(data + sizeof(*hdr) + 12 * i + 4));\n+\t\tuint32_t chunk_offset2 = ntohl(*(uint32_t*)(data + sizeof(*hdr) + 12 * i + 8));\n+\t\tuint64_t chunk_offset = (chunk_offset1 << 32) | chunk_offset2;\n+\n+\t\tif (chunk_offset > graph_size - GIT_MAX_RAWSZ)\n+\t\t\tdie(\"improper chunk offset %08x%08x\", (uint32_t)(chunk_offset >> 32),\n+\t\t\t (uint32_t)chunk_offset);\n+\n+\t\tswitch (chunk_id) {\n+\t\t\tcase GRAPH_CHUNKID_OIDFANOUT:\n+\t\t\t\tgraph->chunk_oid_fanout = data + chunk_offset;\n+\t\t\t\tbreak;\n+\n+\t\t\tcase GRAPH_CHUNKID_OIDLOOKUP:\n+\t\t\t\tgraph->chunk_oid_lookup = data + chunk_offset;\n+\t\t\t\tbreak;\n+\n+\t\t\tcase GRAPH_CHUNKID_DATA:\n+\t\t\t\tgraph->chunk_commit_data = data + chunk_offset;\n+\t\t\t\tbreak;\n+\n+\t\t\tcase GRAPH_CHUNKID_LARGEEDGES:\n+\t\t\t\tgraph->chunk_large_edges = data + chunk_offset;\n+\t\t\t\tbreak;\n+\n+\t\t\tcase 0:\n+\t\t\t\tbreak;\n+\n+\t\t\tdefault:\n+\t\t\t\tfree_commit_graph(&graph);\n+\t\t\t\tdie(\"unrecognized graph chunk id: %08x\", chunk_id);\n+\t\t}\n+\n+\t\tif (last_chunk_id == GRAPH_CHUNKID_OIDLOOKUP)\n+\t\t{\n+\t\t\tgraph->num_commits = (chunk_offset - last_chunk_offset)\n+\t\t\t\t\t / hdr->hash_len;\n+\t\t}\n+\n+\t\tlast_chunk_id = chunk_id;\n+\t\tlast_chunk_offset = chunk_offset;\n+\t}\n+\n+\tstrcpy(graph->pack_dir, pack_dir);\n+\treturn graph;\n+}\n+\n static void write_graph_chunk_fanout(struct sha1file *f,\n \t\t\t\t struct commit **commits,\n \t\t\t\t int nr_commits)\n@@ -361,7 +497,7 @@ struct object_id *construct_commit_graph(const char *pack_dir)\n \tsha1close(f, final_hash, CSUM_CLOSE | CSUM_FSYNC);\n \n \tf_hash = (struct object_id *)malloc(sizeof(struct object_id));\n-\tmemcpy(f_hash->hash, final_hash, GIT_MAX_RAWSZ);\n+\thashcpy(f_hash->hash, final_hash);\n \tfname = get_commit_graph_filename_hash(pack_dir, f_hash);\n \n \tif (rename(graph_name, fname))\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 7b3469a7df..e046ae575c 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -15,6 +15,31 @@ struct commit_graph_header {\n \tunsigned char num_chunks;\n };\n \n+extern struct commit_graph {\n+\tint graph_fd;\n+\n+\tconst unsigned char *data;\n+\tsize_t data_len;\n+\n+\tconst struct commit_graph_header *hdr;\n+\n+\tstruct object_id oid;\n+\n+\tuint32_t num_commits;\n+\n+\tconst unsigned char *chunk_oid_fanout;\n+\tconst unsigned char *chunk_oid_lookup;\n+\tconst unsigned char *chunk_commit_data;\n+\tconst unsigned char *chunk_large_edges;\n+\n+\t/* something like \".git/objects/pack\" */\n+\tchar pack_dir[FLEX_ARRAY]; /* more */\n+} *commit_graph;\n+\n+extern int close_commit_graph(struct commit_graph *g);\n+\n+extern struct commit_graph *load_commit_graph_one(const char *graph_file, const char *pack_dir);\n+\n extern struct object_id *construct_commit_graph(const char *pack_dir);\n \n #endif\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 6bcd1cc264..da565624e3 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -25,11 +25,23 @@ test_expect_success 'create commits and repack' \\\n done &&\n git repack'\n \n+_graph_read_expect() {\n+ cat >expect <<- EOF\n+header: 43475048 01 01 14 04\n+num_commits: $1\n+chunks: oid_fanout oid_lookup commit_metadata large_edges\n+pack_dir: $2\n+EOF\n+}\n+\n test_expect_success 'write graph' \\\n 'graph1=$(git commit-graph --write) &&\n- test_path_is_file ${packdir}/graph-${graph1}.graph'\n+ test_path_is_file ${packdir}/graph-${graph1}.graph &&\n+ git commit-graph --read --graph-hash=${graph1} >output &&\n+ _graph_read_expect \"5\" \"${packdir}\" &&\n+ cmp expect output'\n \n-t_expect_success 'Add more commits' \\\n+test_expect_success 'Add more commits' \\\n 'git reset --hard commits/3 &&\n for i in $(test_seq 6 10)\n do\n@@ -79,7 +91,10 @@ t_expect_success 'Add more commits' \\\n \n test_expect_success 'write graph with merges' \\\n 'graph2=$(git commit-graph --write) &&\n- test_path_is_file ${packdir}/graph-${graph2}.graph'\n+ test_path_is_file ${packdir}/graph-${graph2}.graph &&\n+ git commit-graph --read --graph-hash=${graph2} >output &&\n+ _graph_read_expect \"18\" \"${packdir}\" &&\n+ cmp expect output'\n \n test_expect_success 'setup bare repo' \\\n 'cd .. &&\n@@ -87,10 +102,13 @@ test_expect_success 'setup bare repo' \\\n cd bare &&\n git config core.graph true &&\n git config pack.threads 1 &&\n- baredir=\"objects/pack\"'\n+ baredir=\"./objects/pack\"'\n \n test_expect_success 'write graph in bare repo' \\\n 'graphbare=$(git commit-graph --write) &&\n- test_path_is_file ${baredir}/graph-${graphbare}.graph'\n+ test_path_is_file ${baredir}/graph-${graphbare}.graph &&\n+ git commit-graph --read --graph-hash=${graphbare} >output &&\n+ _graph_read_expect \"18\" \"${baredir}\" &&\n+ cmp expect output'\n \n test_done\n-- \n2.16.0.15.g9c3cf44.dirty\n\n"},{"id":"337832","messageId":"1517348383-112294-9-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1517348383-112294-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v2 08/14] commit-graph: implement git-commit-graph --clear","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-30T21:39:37Z","receivedAt":"2018-01-30T21:40:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach Git to delete the current 'graph_head' file and the commit graph\nit references. This is a good safety valve if somehow the file is\ncorrupted and needs to be recalculated. Since the commit graph is a\nsummary of contents already in the ODB, it can be regenerated.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 16 ++++++++++++++--\n builtin/commit-graph.c | 32 +++++++++++++++++++++++++++++++-\n t/t5318-commit-graph.sh | 7 ++++++-\n 3 files changed, 51 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 99ced16ddc..33d6567f11 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -11,6 +11,7 @@ SYNOPSIS\n [verse]\n 'git commit-graph' --write [--pack-dir ]\n 'git commit-graph' --read [--pack-dir ]\n+'git commit-graph' --clear [--pack-dir ]\n \n OPTIONS\n -------\n@@ -18,16 +19,21 @@ OPTIONS\n \tUse given directory for the location of packfiles, graph-head,\n \tand graph files.\n \n+--clear::\n+\tDelete the graph-head file and the graph file it references.\n+\t(Cannot be combined with --read or --write.)\n+\n --read::\n \tRead a graph file given by the graph-head file and output basic\n-\tdetails about the graph file. (Cannot be combined with --write.)\n+\tdetails about the graph file. (Cannot be combined with --clear\n+\tor --write.)\n \n --graph-id::\n \tWhen used with --read, consider the graph file graph-.graph.\n \n --write::\n \tWrite a new graph file to the pack directory. (Cannot be combined\n-\twith --read.)\n+\twith --clear or --read.)\n \n --update-head::\n \tWhen used with --write, update the graph-head file to point to\n@@ -61,6 +67,12 @@ $ git commit-graph --write --update-head\n $ git commit-graph --read --graph-hash=\n ------------------------------------------------\n \n+* Delete /graph-head and the file it references.\n++\n+------------------------------------------------\n+$ git commit-graph --clear --pack-dir=\n+------------------------------------------------\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex d73cbc907d..4970dec133 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -10,6 +10,7 @@\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--pack-dir ]\"),\n+\tN_(\"git commit-graph --clear [--pack-dir ]\"),\n \tN_(\"git commit-graph --read [--graph-hash=]\"),\n \tN_(\"git commit-graph --write [--pack-dir ] [--update-head]\"),\n \tNULL\n@@ -17,6 +18,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \n static struct opts_commit_graph {\n \tconst char *pack_dir;\n+\tint clear;\n \tint read;\n \tconst char *graph_hash;\n \tint write;\n@@ -25,6 +27,30 @@ static struct opts_commit_graph {\n \tstruct object_id old_graph_hash;\n } opts;\n \n+static int graph_clear(void)\n+{\n+\tstruct strbuf head_path = STRBUF_INIT;\n+\tchar *old_path;\n+\n+\tif (!opts.has_existing)\n+\t\treturn 0;\n+\n+\tstrbuf_addstr(&head_path, opts.pack_dir);\n+\tstrbuf_addstr(&head_path, \"/\");\n+\tstrbuf_addstr(&head_path, \"graph-head\");\n+\tif (remove_path(head_path.buf))\n+\t\tdie(\"failed to remove path %s\", head_path.buf);\n+\tstrbuf_release(&head_path);\n+\n+\told_path = get_commit_graph_filename_hash(opts.pack_dir,\n+\t\t\t\t\t\t &opts.old_graph_hash);\n+\tif (remove_path(old_path))\n+\t\tdie(\"failed to remove path %s\", old_path);\n+\tfree(old_path);\n+\n+\treturn 0;\n+}\n+\n static int graph_read(void)\n {\n \tstruct object_id graph_hash;\n@@ -105,6 +131,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\t{ OPTION_STRING, 'p', \"pack-dir\", &opts.pack_dir,\n \t\t\tN_(\"dir\"),\n \t\t\tN_(\"The pack directory to store the graph\") },\n+\t\tOPT_BOOL('c', \"clear\", &opts.clear,\n+\t\t\tN_(\"clear graph file and graph-head\")),\n \t\tOPT_BOOL('r', \"read\", &opts.read,\n \t\t\tN_(\"read graph file\")),\n \t\tOPT_BOOL('w', \"write\", &opts.write,\n@@ -126,7 +154,7 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\t\t builtin_commit_graph_options,\n \t\t\t builtin_commit_graph_usage, 0);\n \n-\tif (opts.write + opts.read > 1)\n+\tif (opts.write + opts.read + opts.clear > 1)\n \t\tusage_with_options(builtin_commit_graph_usage,\n \t\t\t\t builtin_commit_graph_options);\n \n@@ -139,6 +167,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \n \topts.has_existing = !!get_graph_head_hash(opts.pack_dir, &opts.old_graph_hash);\n \n+\tif (opts.clear)\n+\t\treturn graph_clear();\n \tif (opts.read)\n \t\treturn graph_read();\n \tif (opts.write)\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex d1a23bcdaf..6e3b62b754 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -101,6 +101,11 @@ test_expect_success 'write graph with merges' \\\n _graph_read_expect \"18\" \"${packdir}\" &&\n cmp expect output'\n \n+test_expect_success 'clear graph' \\\n+ 'git commit-graph --clear &&\n+ test_path_is_missing ${packdir}/graph-${graph2}.graph &&\n+ test_path_is_missing ${packdir}/graph-head'\n+\n test_expect_success 'setup bare repo' \\\n 'cd .. &&\n git clone --bare full bare &&\n@@ -110,7 +115,7 @@ test_expect_success 'setup bare repo' \\\n baredir=\"./objects/pack\"'\n \n test_expect_success 'write graph in bare repo' \\\n- 'graphbare=$(git commit-graph --write) &&\n+ 'graphbare=$(git commit-graph --write --update-head) &&\n test_path_is_file ${baredir}/graph-${graphbare}.graph &&\n test_path_is_file ${baredir}/graph-head &&\n echo ${graphbare} >expect &&\n-- \n2.16.0.15.g9c3cf44.dirty\n\n"},{"id":"337833","messageId":"1517348383-112294-12-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1517348383-112294-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v2 11/14] commit: integrate commit graph with commit parsing","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-30T21:39:40Z","receivedAt":"2018-01-30T21:40:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach Git to inspect a commit graph file to supply the contents of a\nstruct commit when calling parse_commit_gently(). This implementation\nsatisfies all post-conditions on the struct commit, including loading\nparents, the root tree, and the commit date. The only loosely-expected\ncondition is that the commit buffer is loaded into the cache. This\nwas checked in log-tree.c:show_log(), but the \"return;\" on failure\nproduced unexpected results (i.e. the message line was never terminated).\nThe new behavior of loading the buffer when needed prevents the\nunexpected behavior.\n\nIf core.commitgraph is false, then do not check graph files.\n\nIn test script t5319-commit-graph.sh, add output-matching conditions on\nread-only graph operations.\n\nBy loading commits from the graph instead of parsing commit buffers, we\nsave a lot of time on long commit walks. Here are some performance\nresults for a copy of the Linux repository where 'master' has 704,766\nreachable commits and is behind 'origin/master' by 19,610 commits.\n\n| Command | Before | After | Rel % |\n|----------------------------------|--------|--------|-------|\n| log --oneline --topo-order -1000 | 5.9s | 0.7s | -88% |\n| branch -vv | 0.42s | 0.27s | -35% |\n| rev-list --all | 6.4s | 1.0s | -84% |\n| rev-list --all --objects | 32.6s | 27.6s | -15% |\n\nSigned-off-by: Derrick Stolee \n---\n alloc.c | 1 +\n commit-graph.c | 237 ++++++++++++++++++++++++++++++++++++++++++++++++\n commit-graph.h | 20 +++-\n commit.c | 10 +-\n commit.h | 4 +\n log-tree.c | 3 +-\n t/t5318-commit-graph.sh | 47 ++++++++++\n 7 files changed, 318 insertions(+), 4 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 12afadfacd..cf4f8b61e1 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -93,6 +93,7 @@ void *alloc_commit_node(void)\n \tstruct commit *c = alloc_node(&commit_state, sizeof(struct commit));\n \tc->object.type = OBJ_COMMIT;\n \tc->index = alloc_commit_index();\n+\tc->graph_pos = COMMIT_NOT_FROM_GRAPH;\n \treturn c;\n }\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 764e016ddb..fc816533c6 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -35,6 +35,9 @@\n #define GRAPH_MIN_SIZE (GRAPH_CHUNKLOOKUP_SIZE + GRAPH_FANOUT_SIZE + \\\n \t\t\tGRAPH_OID_LEN + sizeof(struct commit_graph_header))\n \n+/* global storage */\n+struct commit_graph *commit_graph = 0;\n+\n struct object_id *get_graph_head_hash(const char *pack_dir, struct object_id *hash)\n {\n \tstruct strbuf head_filename = STRBUF_INIT;\n@@ -209,6 +212,220 @@ struct commit_graph *load_commit_graph_one(const char *graph_file, const char *p\n \treturn graph;\n }\n \n+static void prepare_commit_graph_one(const char *obj_dir)\n+{\n+\tchar *graph_file;\n+\tstruct object_id oid;\n+\tstruct strbuf pack_dir = STRBUF_INIT;\n+\tstrbuf_addstr(&pack_dir, obj_dir);\n+\tstrbuf_add(&pack_dir, \"/pack\", 5);\n+\n+\tif (!get_graph_head_hash(pack_dir.buf, &oid))\n+\t\treturn;\n+\n+\tgraph_file = get_commit_graph_filename_hash(pack_dir.buf, &oid);\n+\n+\tcommit_graph = load_commit_graph_one(graph_file, pack_dir.buf);\n+\tstrbuf_release(&pack_dir);\n+}\n+\n+static int prepare_commit_graph_run_once = 0;\n+void prepare_commit_graph(void)\n+{\n+\tstruct alternate_object_database *alt;\n+\tchar *obj_dir;\n+\n+\tif (prepare_commit_graph_run_once)\n+\t\treturn;\n+\tprepare_commit_graph_run_once = 1;\n+\n+\tobj_dir = get_object_directory();\n+\tprepare_commit_graph_one(obj_dir);\n+\tprepare_alt_odb();\n+\tfor (alt = alt_odb_list; !commit_graph && alt; alt = alt->next)\n+\t\tprepare_commit_graph_one(alt->path);\n+}\n+\n+static int bsearch_graph(struct commit_graph *g, struct object_id *oid, uint32_t *pos)\n+{\n+\tuint32_t last, first = 0;\n+\n+\tif (oid->hash[0])\n+\t\tfirst = ntohl(*(uint32_t*)(g->chunk_oid_fanout + 4 * (oid->hash[0] - 1)));\n+\tlast = ntohl(*(uint32_t*)(g->chunk_oid_fanout + 4 * oid->hash[0]));\n+\n+\twhile (first < last) {\n+\t\tuint32_t mid = first + (last - first) / 2;\n+\t\tconst unsigned char *current;\n+\t\tint cmp;\n+\n+\t\tcurrent = g->chunk_oid_lookup + g->hdr->hash_len * mid;\n+\t\tcmp = hashcmp(oid->hash, current);\n+\t\tif (!cmp) {\n+\t\t\t*pos = mid;\n+\t\t\treturn 1;\n+\t\t}\n+\t\tif (cmp > 0) {\n+\t\t\tfirst = mid + 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tlast = mid;\n+\t}\n+\n+\t*pos = first;\n+\treturn 0;\n+}\n+\n+struct object_id *get_nth_commit_oid(struct commit_graph *g,\n+\t\t\t\t uint32_t n,\n+\t\t\t\t struct object_id *oid)\n+{\n+\thashcpy(oid->hash, g->chunk_oid_lookup + g->hdr->hash_len * n);\n+\treturn oid;\n+}\n+\n+static int full_parse_commit(struct commit *item, struct commit_graph *g,\n+\t\t\t uint32_t pos, const unsigned char *commit_data)\n+{\n+\tstruct object_id oid;\n+\tstruct commit *new_parent;\n+\tuint32_t new_parent_pos;\n+\tuint32_t *parent_data_ptr;\n+\tuint64_t date_low, date_high;\n+\tstruct commit_list **pptr;\n+\n+\titem->object.parsed = 1;\n+\titem->graph_pos = pos;\n+\n+\thashcpy(oid.hash, commit_data);\n+\titem->tree = lookup_tree(&oid);\n+\n+\tdate_high = ntohl(*(uint32_t*)(commit_data + g->hdr->hash_len + 8)) & 0x3;\n+\tdate_low = ntohl(*(uint32_t*)(commit_data + g->hdr->hash_len + 12));\n+\titem->date = (timestamp_t)((date_high << 32) | date_low);\n+\n+\tpptr = &item->parents;\n+\n+\tnew_parent_pos = ntohl(*(uint32_t*)(commit_data + g->hdr->hash_len));\n+\tif (new_parent_pos == GRAPH_PARENT_NONE)\n+\t\treturn 1;\n+\tget_nth_commit_oid(g, new_parent_pos, &oid);\n+\tnew_parent = lookup_commit(&oid);\n+\tif (new_parent) {\n+\t\tnew_parent->graph_pos = new_parent_pos;\n+\t\tpptr = &commit_list_insert(new_parent, pptr)->next;\n+\t} else {\n+\t\tdie(\"could not find commit %s\", oid_to_hex(&oid));\n+\t}\n+\n+\tnew_parent_pos = ntohl(*(uint32_t*)(commit_data + g->hdr->hash_len + 4));\n+\tif (new_parent_pos == GRAPH_PARENT_NONE)\n+\t\treturn 1;\n+\tif (!(new_parent_pos & GRAPH_LARGE_EDGES_NEEDED)) {\n+\t\tget_nth_commit_oid(g, new_parent_pos, &oid);\n+\t\tnew_parent = lookup_commit(&oid);\n+\t\tif (new_parent) {\n+\t\t\tnew_parent->graph_pos = new_parent_pos;\n+\t\t\tpptr = &commit_list_insert(new_parent, pptr)->next;\n+\t\t} else\n+\t\t\tdie(\"could not find commit %s\", oid_to_hex(&oid));\n+\t\treturn 1;\n+\t}\n+\n+\tparent_data_ptr = (uint32_t*)(g->chunk_large_edges + 4 * (new_parent_pos ^ GRAPH_LARGE_EDGES_NEEDED));\n+\tdo {\n+\t\tnew_parent_pos = ntohl(*parent_data_ptr);\n+\n+\t\tget_nth_commit_oid(g, new_parent_pos & GRAPH_EDGE_LAST_MASK, &oid);\n+\t\tnew_parent = lookup_commit(&oid);\n+\t\tif (new_parent) {\n+\t\t\tnew_parent->graph_pos = new_parent_pos & GRAPH_EDGE_LAST_MASK;\n+\t\t\tpptr = &commit_list_insert(new_parent, pptr)->next;\n+\t\t} else\n+\t\t\tdie(\"could not find commit %s\", oid_to_hex(&oid));\n+\t\tparent_data_ptr++;\n+\t} while (!(new_parent_pos & GRAPH_LAST_EDGE));\n+\n+\treturn 1;\n+}\n+\n+/**\n+ * Fill 'item' to contain all information that would be parsed by parse_commit_buffer.\n+ */\n+static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t pos)\n+{\n+\tuint32_t new_parent_pos;\n+\tuint32_t *parent_data_ptr;\n+\tconst unsigned char *commit_data = g->chunk_commit_data + (g->hdr->hash_len + 16) * pos;\n+\n+\tnew_parent_pos = ntohl(*(uint32_t*)(commit_data + g->hdr->hash_len));\n+\n+\tif (new_parent_pos == GRAPH_PARENT_MISSING)\n+\t\treturn 0;\n+\n+\tif (new_parent_pos == GRAPH_PARENT_NONE)\n+\t\treturn full_parse_commit(item, g, pos, commit_data);\n+\n+\tnew_parent_pos = ntohl(*(uint32_t*)(commit_data + g->hdr->hash_len + 4));\n+\n+\tif (new_parent_pos == GRAPH_PARENT_MISSING)\n+\t\treturn 0;\n+\tif (!(new_parent_pos & GRAPH_LARGE_EDGES_NEEDED))\n+\t\treturn full_parse_commit(item, g, pos, commit_data);\n+\n+\tnew_parent_pos = new_parent_pos ^ GRAPH_LARGE_EDGES_NEEDED;\n+\n+\tif (new_parent_pos == GRAPH_PARENT_MISSING)\n+\t\treturn 0;\n+\n+\tparent_data_ptr = (uint32_t*)(g->chunk_large_edges + 4 * new_parent_pos);\n+\tdo {\n+\t\tnew_parent_pos = ntohl(*parent_data_ptr);\n+\n+\t\tif ((new_parent_pos & GRAPH_EDGE_LAST_MASK) == GRAPH_PARENT_MISSING)\n+\t\t\treturn 0;\n+\n+\t\tparent_data_ptr++;\n+\t} while (!(new_parent_pos & GRAPH_LAST_EDGE));\n+\n+\treturn full_parse_commit(item, g, pos, commit_data);\n+}\n+\n+/**\n+ * Given a commit struct, try to fill the commit struct info, including:\n+ * 1. tree object\n+ * 2. date\n+ * 3. parents.\n+ *\n+ * Returns 1 if and only if the commit was found in the commit graph.\n+ *\n+ * See parse_commit_buffer() for the fallback after this call.\n+ */\n+int parse_commit_in_graph(struct commit *item)\n+{\n+\tif (!core_commitgraph)\n+\t\treturn 0;\n+\tif (item->object.parsed)\n+\t\treturn 1;\n+\n+\tprepare_commit_graph();\n+\tif (commit_graph) {\n+\t\tuint32_t pos;\n+\t\tint found;\n+\t\tif (item->graph_pos != COMMIT_NOT_FROM_GRAPH) {\n+\t\t\tpos = item->graph_pos;\n+\t\t\tfound = 1;\n+\t\t} else {\n+\t\t\tfound = bsearch_graph(commit_graph, &(item->object.oid), &pos);\n+\t\t}\n+\n+\t\tif (found)\n+\t\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n+\t}\n+\n+\treturn 0;\n+}\n+\n static void write_graph_chunk_fanout(struct sha1file *f,\n \t\t\t\t struct commit **commits,\n \t\t\t\t int nr_commits)\n@@ -439,9 +656,24 @@ struct object_id *construct_commit_graph(const char *pack_dir)\n \tchar *fname;\n \tstruct commit_list *parent;\n \n+\tprepare_commit_graph();\n+\n \toids.num = 0;\n \toids.size = 1024;\n+\n+\tif (commit_graph && oids.size < commit_graph->num_commits)\n+\t\toids.size = commit_graph->num_commits;\n+\n \tALLOC_ARRAY(oids.list, oids.size);\n+\n+\tif (commit_graph) {\n+\t\tfor (i = 0; i < commit_graph->num_commits; i++) {\n+\t\t\toids.list[i] = malloc(sizeof(struct object_id));\n+\t\t\tget_nth_commit_oid(commit_graph, i, oids.list[i]);\n+\t\t}\n+\t\toids.num = commit_graph->num_commits;\n+\t}\n+\n \tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n \tQSORT(oids.list, oids.num, commit_compare);\n \n@@ -525,6 +757,11 @@ struct object_id *construct_commit_graph(const char *pack_dir)\n \thashcpy(f_hash->hash, final_hash);\n \tfname = get_commit_graph_filename_hash(pack_dir, f_hash);\n \n+\tif (commit_graph) {\n+\t\tclose_commit_graph(commit_graph);\n+\t\tFREE_AND_NULL(commit_graph);\n+\t}\n+\n \tif (rename(graph_name, fname))\n \t\tdie(\"failed to rename %s to %s\", graph_name, fname);\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex 43eb0aec84..05ddbbe165 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -4,6 +4,18 @@\n #include \"git-compat-util.h\"\n #include \"commit.h\"\n \n+/**\n+ * Given a commit struct, try to fill the commit struct info, including:\n+ * 1. tree object\n+ * 2. date\n+ * 3. parents.\n+ *\n+ * Returns 1 if and only if the commit was found in the packed graph.\n+ *\n+ * See parse_commit_buffer() for the fallback after this call.\n+ */\n+extern int parse_commit_in_graph(struct commit *item);\n+\n extern struct object_id *get_graph_head_hash(const char *pack_dir,\n \t\t\t\t\t struct object_id *hash);\n extern char* get_commit_graph_filename_hash(const char *pack_dir,\n@@ -40,7 +52,13 @@ extern struct commit_graph {\n \n extern int close_commit_graph(struct commit_graph *g);\n \n-extern struct commit_graph *load_commit_graph_one(const char *graph_file, const char *pack_dir);\n+extern struct commit_graph *load_commit_graph_one(const char *graph_file,\n+\t\t\t\t\t\t const char *pack_dir);\n+extern void prepare_commit_graph(void);\n+\n+extern struct object_id *get_nth_commit_oid(struct commit_graph *g,\n+\t\t\t\t\t uint32_t n,\n+\t\t\t\t\t struct object_id *oid);\n \n extern struct object_id *construct_commit_graph(const char *pack_dir);\n \ndiff --git a/commit.c b/commit.c\nindex cab8d4455b..4437798e84 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -12,6 +12,7 @@\n #include \"prio-queue.h\"\n #include \"sha1-lookup.h\"\n #include \"wt-status.h\"\n+#include \"commit-graph.h\"\n \n static struct commit_extra_header *read_commit_extra_header_lines(const char *buf, size_t len, const char **);\n \n@@ -374,7 +375,7 @@ int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long s\n \treturn 0;\n }\n \n-int parse_commit_gently(struct commit *item, int quiet_on_missing)\n+int parse_commit_internal(struct commit *item, int quiet_on_missing, int check_packed)\n {\n \tenum object_type type;\n \tvoid *buffer;\n@@ -385,6 +386,8 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \t\treturn -1;\n \tif (item->object.parsed)\n \t\treturn 0;\n+\tif (check_packed && parse_commit_in_graph(item))\n+\t\treturn 0;\n \tbuffer = read_sha1_file(item->object.oid.hash, &type, &size);\n \tif (!buffer)\n \t\treturn quiet_on_missing ? -1 :\n@@ -404,6 +407,11 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \treturn ret;\n }\n \n+int parse_commit_gently(struct commit *item, int quiet_on_missing)\n+{\n+\treturn parse_commit_internal(item, quiet_on_missing, 1);\n+}\n+\n void parse_commit_or_die(struct commit *item)\n {\n \tif (parse_commit(item))\ndiff --git a/commit.h b/commit.h\nindex 8c68ca1a5a..fc8880d187 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -9,6 +9,8 @@\n #include \"string-list.h\"\n #include \"pretty.h\"\n \n+#define COMMIT_NOT_FROM_GRAPH 0xFFFFFFFF\n+\n struct commit_list {\n \tstruct commit *item;\n \tstruct commit_list *next;\n@@ -21,6 +23,7 @@ struct commit {\n \ttimestamp_t date;\n \tstruct commit_list *parents;\n \tstruct tree *tree;\n+\tuint32_t graph_pos;\n };\n \n extern int save_commit_buffer;\n@@ -60,6 +63,7 @@ struct commit *lookup_commit_reference_by_name(const char *name);\n struct commit *lookup_commit_or_die(const struct object_id *oid, const char *ref_name);\n \n int parse_commit_buffer(struct commit *item, const void *buffer, unsigned long size);\n+extern int parse_commit_internal(struct commit *item, int quiet_on_missing, int check_packed);\n int parse_commit_gently(struct commit *item, int quiet_on_missing);\n static inline int parse_commit(struct commit *item)\n {\ndiff --git a/log-tree.c b/log-tree.c\nindex fca29d4799..156aed4541 100644\n--- a/log-tree.c\n+++ b/log-tree.c\n@@ -659,8 +659,7 @@ void show_log(struct rev_info *opt)\n \t\tshow_mergetag(opt, commit);\n \t}\n \n-\tif (!get_cached_commit_buffer(commit, NULL))\n-\t\treturn;\n+\tget_commit_buffer(commit, NULL);\n \n \tif (opt->show_notes) {\n \t\tint raw;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex b56a6d4217..93b0d4f51b 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -26,6 +26,24 @@ test_expect_success 'create commits and repack' \\\n done &&\n git repack'\n \n+_graph_git_two_modes() {\n+ git -c core.commitgraph=true $1 >output\n+ git -c core.commitgraph=false $1 >expect\n+ cmp output expect\n+}\n+\n+_graph_git_behavior() {\n+ BRANCH=$1\n+ COMPARE=$2\n+ test_expect_success 'check normal git operations' \\\n+ '_graph_git_two_modes \"log --oneline ${BRANCH}\" &&\n+ _graph_git_two_modes \"log --topo-order ${BRANCH}\" &&\n+ _graph_git_two_modes \"branch -vv\" &&\n+ _graph_git_two_modes \"merge-base -a ${BRANCH} ${COMPARE}\"'\n+}\n+\n+_graph_git_behavior \"commits/3\" \"commits/4\"\n+\n _graph_read_expect() {\n cat >expect <<- EOF\n header: 43475048 01 01 14 04\n@@ -43,6 +61,8 @@ test_expect_success 'write graph' \\\n _graph_read_expect \"5\" \"${packdir}\" &&\n cmp expect output'\n \n+_graph_git_behavior \"commits/3\" \"commits/4\"\n+\n test_expect_success 'Add more commits' \\\n 'git reset --hard commits/3 &&\n for i in $(test_seq 6 10)\n@@ -91,6 +111,10 @@ test_expect_success 'Add more commits' \\\n # |\n # 1\n \n+_graph_git_behavior \"merge/1\" \"merge/2\"\n+_graph_git_behavior \"merge/1\" \"merge/3\"\n+_graph_git_behavior \"merge/2\" \"merge/3\"\n+\n test_expect_success 'write graph with merges' \\\n 'graph2=$(git commit-graph --write --update-head) &&\n test_path_is_file ${packdir}/graph-${graph2}.graph &&\n@@ -101,6 +125,10 @@ test_expect_success 'write graph with merges' \\\n _graph_read_expect \"18\" \"${packdir}\" &&\n cmp expect output'\n \n+_graph_git_behavior merge/1 merge/2\n+_graph_git_behavior merge/1 merge/3\n+_graph_git_behavior merge/2 merge/3\n+\n test_expect_success 'Add more commits' \\\n 'for i in $(test_seq 16 20)\n do\n@@ -141,6 +169,10 @@ test_expect_success 'Add more commits' \\\n # |\n # 1\n \n+# Test behavior while in mixed mode\n+_graph_git_behavior commits/20 merge/1\n+_graph_git_behavior commits/20 merge/2\n+\n test_expect_success 'write graph with merges' \\\n 'graph3=$(git commit-graph --write --update-head --delete-expired) &&\n test_path_is_file ${packdir}/graph-${graph3}.graph &&\n@@ -153,6 +185,9 @@ test_expect_success 'write graph with merges' \\\n _graph_read_expect \"23\" \"${packdir}\" &&\n cmp expect output'\n \n+_graph_git_behavior commits/20 merge/1\n+_graph_git_behavior commits/20 merge/2\n+\n test_expect_success 'write graph with nothing new' \\\n 'graph4=$(git commit-graph --write --update-head --delete-expired) &&\n test_path_is_file ${packdir}/graph-${graph4}.graph &&\n@@ -164,12 +199,18 @@ test_expect_success 'write graph with nothing new' \\\n _graph_read_expect \"23\" \"${packdir}\" &&\n cmp expect output'\n \n+_graph_git_behavior commits/20 merge/1\n+_graph_git_behavior commits/20 merge/2\n+\n test_expect_success 'clear graph' \\\n 'git commit-graph --clear &&\n test_path_is_missing ${packdir}/graph-${graph2}.graph &&\n test_path_is_file ${packdir}/graph-${graph1}.graph &&\n test_path_is_missing ${packdir}/graph-head'\n \n+_graph_git_behavior commits/20 merge/1\n+_graph_git_behavior commits/20 merge/2\n+\n test_expect_success 'setup bare repo' \\\n 'cd .. &&\n git clone --bare full bare &&\n@@ -178,6 +219,9 @@ test_expect_success 'setup bare repo' \\\n git config pack.threads 1 &&\n baredir=\"./objects/pack\"'\n \n+_graph_git_behavior commits/20 merge/1\n+_graph_git_behavior commits/20 merge/2\n+\n test_expect_success 'write graph in bare repo' \\\n 'graphbare=$(git commit-graph --write --update-head) &&\n test_path_is_file ${baredir}/graph-${graphbare}.graph &&\n@@ -188,4 +232,7 @@ test_expect_success 'write graph in bare repo' \\\n _graph_read_expect \"23\" \"${baredir}\" &&\n cmp expect output'\n \n+_graph_git_behavior commits/20 merge/1\n+_graph_git_behavior commits/20 merge/2\n+\n test_done\n-- \n2.16.0.15.g9c3cf44.dirty\n\n"},{"id":"337834","messageId":"1517348383-112294-14-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1517348383-112294-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v2 13/14] commit-graph: close under reachability","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-30T21:39:42Z","receivedAt":"2018-01-30T21:40:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach construct_commit_graph() to walk all parents from the commits\ndiscovered in packfiles. This prevents gaps given by loose objects or\npreviously-missed packfiles.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 26 ++++++++++++++++++++++++++\n t/t5318-commit-graph.sh | 14 ++++++++++++++\n 2 files changed, 40 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex e5a1d9ee8b..cfa0415a21 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -5,6 +5,7 @@\n #include \"packfile.h\"\n #include \"commit.h\"\n #include \"object.h\"\n+#include \"revision.h\"\n #include \"commit-graph.h\"\n \n #define GRAPH_SIGNATURE 0x43475048 /* \"CGPH\" */\n@@ -638,6 +639,29 @@ static int if_packed_commit_add_to_list(const struct object_id *oid,\n \treturn 0;\n }\n \n+static void close_reachable(struct packed_oid_list *oids)\n+{\n+\tint i;\n+\tstruct rev_info revs;\n+\tstruct commit *commit;\n+\tinit_revisions(&revs, NULL);\n+\n+\tfor (i = 0; i < oids->num; i++) {\n+\t\tcommit = lookup_commit(oids->list[i]);\n+\t\tif (commit && !parse_commit(commit))\n+\t\t\trevs.commits = commit_list_insert(commit, &revs.commits);\n+\t}\n+\n+\tif (prepare_revision_walk(&revs))\n+\t\tdie(_(\"revision walk setup failed\"));\n+\n+\twhile ((commit = get_revision(&revs)) != NULL) {\n+\t\tALLOC_GROW(oids->list, oids->num + 1, oids->size);\n+\t\toids->list[oids->num] = &(commit->object.oid);\n+\t\t(oids->num)++;\n+\t}\n+}\n+\n struct object_id *construct_commit_graph(const char *pack_dir,\n \t\t\t\t\t char **pack_indexes,\n \t\t\t\t\t int nr_packs)\n@@ -696,6 +720,8 @@ struct object_id *construct_commit_graph(const char *pack_dir,\n \t} else {\n \t\tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n \t}\n+\n+\tclose_reachable(&oids);\n \tQSORT(oids.list, oids.num, commit_compare);\n \n \tcount_distinct = 1;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex b9a73f398c..2001b0b5b5 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -213,6 +213,20 @@ test_expect_success 'clear graph' \\\n _graph_git_behavior commits/20 merge/1\n _graph_git_behavior commits/20 merge/2\n \n+test_expect_success 'build graph from latest pack with closure' \\\n+ 'graph5=$(cat new-idx | git commit-graph --write --update-head --stdin-packs) &&\n+ test_path_is_file ${packdir}/graph-${graph5}.graph &&\n+ test_path_is_file ${packdir}/graph-${graph1}.graph &&\n+ test_path_is_file ${packdir}/graph-head &&\n+ echo ${graph5} >expect &&\n+ cmp -n 40 expect ${packdir}/graph-head &&\n+ git commit-graph --read --graph-hash=${graph5} >output &&\n+ _graph_read_expect \"21\" \"${packdir}\" &&\n+ cmp expect output'\n+\n+_graph_git_behavior commits/20 merge/1\n+_graph_git_behavior commits/20 merge/2\n+\n test_expect_success 'setup bare repo' \\\n 'cd .. &&\n git clone --bare full bare &&\n-- \n2.16.0.15.g9c3cf44.dirty\n\n"},{"id":"337835","messageId":"1517348383-112294-10-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1517348383-112294-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v2 09/14] commit-graph: teach git-commit-graph --delete-expired","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-30T21:39:38Z","receivedAt":"2018-01-30T21:40:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach git-commit-graph to delete the graph previously referenced by 'graph_head'\nwhen writing a new graph file and updating 'graph_head'. This prevents\ndata creep by storing a list of useless graphs. Be careful to not delete\nthe graph if the file did not change.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 8 +++--\n builtin/commit-graph.c | 16 ++++++++-\n t/t5318-commit-graph.sh | 66 +++++++++++++++++++++++++++++++++++++-\n 3 files changed, 86 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 33d6567f11..7b376e9212 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -39,6 +39,10 @@ OPTIONS\n \tWhen used with --write, update the graph-head file to point to\n \tthe written graph file.\n \n+--delete-expired::\n+\tWhen used with --write and --update-head, delete the graph file\n+\tpreviously referenced by graph-head.\n+\n EXAMPLES\n --------\n \n@@ -55,10 +59,10 @@ $ git commit-graph --write\n ------------------------------------------------\n \n * Write a graph file for the packed commits in your local .git folder,\n-* and update graph-head.\n+* update graph-head, and delete the old graph-.graph file.\n +\n ------------------------------------------------\n-$ git commit-graph --write --update-head\n+$ git commit-graph --write --update-head --delete-expired\n ------------------------------------------------\n \n * Read basic information from a graph file.\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 4970dec133..766f09e6fc 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -12,7 +12,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--pack-dir ]\"),\n \tN_(\"git commit-graph --clear [--pack-dir ]\"),\n \tN_(\"git commit-graph --read [--graph-hash=]\"),\n-\tN_(\"git commit-graph --write [--pack-dir ] [--update-head]\"),\n+\tN_(\"git commit-graph --write [--pack-dir ] [--update-head] [--delete-expired]\"),\n \tNULL\n };\n \n@@ -23,6 +23,7 @@ static struct opts_commit_graph {\n \tconst char *graph_hash;\n \tint write;\n \tint update_head;\n+\tint delete_expired;\n \tint has_existing;\n \tstruct object_id old_graph_hash;\n } opts;\n@@ -121,6 +122,17 @@ static int graph_write(void)\n \tif (graph_hash)\n \t\tprintf(\"%s\\n\", oid_to_hex(graph_hash));\n \n+\n+\tif (opts.delete_expired && opts.update_head && opts.has_existing &&\n+\t oidcmp(graph_hash, &opts.old_graph_hash)) {\n+\t\tchar *old_path = get_commit_graph_filename_hash(opts.pack_dir,\n+\t\t\t\t\t\t\t\t&opts.old_graph_hash);\n+\t\tif (remove_path(old_path))\n+\t\t\tdie(\"failed to remove path %s\", old_path);\n+\n+\t\tfree(old_path);\n+\t}\n+\n \tfree(graph_hash);\n \treturn 0;\n }\n@@ -139,6 +151,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"write commit graph file\")),\n \t\tOPT_BOOL('u', \"update-head\", &opts.update_head,\n \t\t\tN_(\"update graph-head to written graph file\")),\n+\t\tOPT_BOOL('d', \"delete-expired\", &opts.delete_expired,\n+\t\t\tN_(\"delete expired head graph file\")),\n \t\t{ OPTION_STRING, 'H', \"graph-hash\", &opts.graph_hash,\n \t\t\tN_(\"hash\"),\n \t\t\tN_(\"A hash for a specific graph file in the pack-dir.\"),\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 6e3b62b754..b56a6d4217 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -101,9 +101,73 @@ test_expect_success 'write graph with merges' \\\n _graph_read_expect \"18\" \"${packdir}\" &&\n cmp expect output'\n \n+test_expect_success 'Add more commits' \\\n+ 'for i in $(test_seq 16 20)\n+ do\n+ echo $i >$i.txt &&\n+ git add $i.txt &&\n+ git commit -m \"commit $i\" &&\n+ git branch commits/$i\n+ done &&\n+ git repack'\n+\n+# Current graph structure:\n+#\n+# 20\n+# |\n+# 19\n+# |\n+# 18\n+# |\n+# 17\n+# |\n+# 16\n+# |\n+# M3\n+# / |\\_____\n+# / 10 15\n+# / | |\n+# / 9 M2 14\n+# | |/ \\ |\n+# | 8 M1 | 13\n+# | |/ | \\_|\n+# 5 7 | 12\n+# | | \\__|\n+# 4 6 11\n+# |____/______/\n+# 3\n+# |\n+# 2\n+# |\n+# 1\n+\n+test_expect_success 'write graph with merges' \\\n+ 'graph3=$(git commit-graph --write --update-head --delete-expired) &&\n+ test_path_is_file ${packdir}/graph-${graph3}.graph &&\n+ test_path_is_missing ${packdir}/graph-${graph2}.graph &&\n+ test_path_is_file ${packdir}/graph-${graph1}.graph &&\n+ test_path_is_file ${packdir}/graph-head &&\n+ echo ${graph3} >expect &&\n+ cmp -n 40 expect ${packdir}/graph-head &&\n+ git commit-graph --read --graph-hash=${graph3} >output &&\n+ _graph_read_expect \"23\" \"${packdir}\" &&\n+ cmp expect output'\n+\n+test_expect_success 'write graph with nothing new' \\\n+ 'graph4=$(git commit-graph --write --update-head --delete-expired) &&\n+ test_path_is_file ${packdir}/graph-${graph4}.graph &&\n+ test_path_is_file ${packdir}/graph-${graph1}.graph &&\n+ test_path_is_file ${packdir}/graph-head &&\n+ echo ${graph4} >expect &&\n+ cmp -n 40 expect ${packdir}/graph-head &&\n+ git commit-graph --read --graph-hash=${graph4} >output &&\n+ _graph_read_expect \"23\" \"${packdir}\" &&\n+ cmp expect output'\n+\n test_expect_success 'clear graph' \\\n 'git commit-graph --clear &&\n test_path_is_missing ${packdir}/graph-${graph2}.graph &&\n+ test_path_is_file ${packdir}/graph-${graph1}.graph &&\n test_path_is_missing ${packdir}/graph-head'\n \n test_expect_success 'setup bare repo' \\\n@@ -121,7 +185,7 @@ test_expect_success 'write graph in bare repo' \\\n echo ${graphbare} >expect &&\n cmp -n 40 expect ${baredir}/graph-head &&\n git commit-graph --read --graph-hash=${graphbare} >output &&\n- _graph_read_expect \"18\" \"${baredir}\" &&\n+ _graph_read_expect \"23\" \"${baredir}\" &&\n cmp expect output'\n \n test_done\n-- \n2.16.0.15.g9c3cf44.dirty\n\n"},{"id":"337836","messageId":"1517348383-112294-13-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1517348383-112294-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v2 12/14] commit-graph: read only from specific pack-indexes","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-30T21:39:41Z","receivedAt":"2018-01-30T21:40:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach git-commit-graph to inspect the objects only in a certain list\nof pack-indexes within the given pack directory. This allows updating\nthe commit graph iteratively, since we add all commits stored in a\nprevious commit graph.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 13 +++++++++++++\n builtin/commit-graph.c | 25 ++++++++++++++++++++++---\n commit-graph.c | 25 +++++++++++++++++++++++--\n commit-graph.h | 4 +++-\n packfile.c | 4 ++--\n packfile.h | 2 ++\n t/t5318-commit-graph.sh | 6 ++++--\n 7 files changed, 69 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 7b376e9212..d0571cd896 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -43,6 +43,11 @@ OPTIONS\n \tWhen used with --write and --update-head, delete the graph file\n \tpreviously referenced by graph-head.\n \n+--stdin-packs::\n+\tWhen used with --write, generate the new graph by walking objects\n+\tonly in the specified packfiles and any commits in the\n+\texisting graph-head.\n+\n EXAMPLES\n --------\n \n@@ -65,6 +70,14 @@ $ git commit-graph --write\n $ git commit-graph --write --update-head --delete-expired\n ------------------------------------------------\n \n+* Write a graph file, extending the current graph file using commits\n+* in , update graph-head, and delete the old graph-.graph\n+* file.\n++\n+------------------------------------------------\n+$ echo | git commit-graph --write --update-head --delete-expired --stdin-packs\n+------------------------------------------------\n+\n * Read basic information from a graph file.\n +\n ------------------------------------------------\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 766f09e6fc..80a409e784 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -12,7 +12,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--pack-dir ]\"),\n \tN_(\"git commit-graph --clear [--pack-dir ]\"),\n \tN_(\"git commit-graph --read [--graph-hash=]\"),\n-\tN_(\"git commit-graph --write [--pack-dir ] [--update-head] [--delete-expired]\"),\n+\tN_(\"git commit-graph --write [--pack-dir ] [--update-head] [--delete-expired] [--stdin-packs]\"),\n \tNULL\n };\n \n@@ -24,6 +24,7 @@ static struct opts_commit_graph {\n \tint write;\n \tint update_head;\n \tint delete_expired;\n+\tint stdin_packs;\n \tint has_existing;\n \tstruct object_id old_graph_hash;\n } opts;\n@@ -114,7 +115,24 @@ static void update_head_file(const char *pack_dir, const struct object_id *graph\n \n static int graph_write(void)\n {\n-\tstruct object_id *graph_hash = construct_commit_graph(opts.pack_dir);\n+\tstruct object_id *graph_hash;\n+\tchar **pack_indexes = NULL;\n+\tint num_packs = 0;\n+\tint size_packs = 0;\n+\n+\tif (opts.stdin_packs) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tsize_packs = 128;\n+\t\tALLOC_ARRAY(pack_indexes, size_packs);\n+\n+\t\twhile (strbuf_getline(&buf, stdin) != EOF) {\n+\t\t\tALLOC_GROW(pack_indexes, num_packs + 1, size_packs);\n+\t\t\tpack_indexes[num_packs++] = buf.buf;\n+\t\t\tstrbuf_detach(&buf, NULL);\n+\t\t}\n+\t}\n+\n+\tgraph_hash = construct_commit_graph(opts.pack_dir, pack_indexes, num_packs);\n \n \tif (opts.update_head)\n \t\tupdate_head_file(opts.pack_dir, graph_hash);\n@@ -122,7 +140,6 @@ static int graph_write(void)\n \tif (graph_hash)\n \t\tprintf(\"%s\\n\", oid_to_hex(graph_hash));\n \n-\n \tif (opts.delete_expired && opts.update_head && opts.has_existing &&\n \t oidcmp(graph_hash, &opts.old_graph_hash)) {\n \t\tchar *old_path = get_commit_graph_filename_hash(opts.pack_dir,\n@@ -153,6 +170,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"update graph-head to written graph file\")),\n \t\tOPT_BOOL('d', \"delete-expired\", &opts.delete_expired,\n \t\t\tN_(\"delete expired head graph file\")),\n+\t\tOPT_BOOL('s', \"stdin-packs\", &opts.stdin_packs,\n+\t\t\tN_(\"only scan packfiles listed by stdin\")),\n \t\t{ OPTION_STRING, 'H', \"graph-hash\", &opts.graph_hash,\n \t\t\tN_(\"hash\"),\n \t\t\tN_(\"A hash for a specific graph file in the pack-dir.\"),\ndiff --git a/commit-graph.c b/commit-graph.c\nindex fc816533c6..e5a1d9ee8b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -638,7 +638,9 @@ static int if_packed_commit_add_to_list(const struct object_id *oid,\n \treturn 0;\n }\n \n-struct object_id *construct_commit_graph(const char *pack_dir)\n+struct object_id *construct_commit_graph(const char *pack_dir,\n+\t\t\t\t\t char **pack_indexes,\n+\t\t\t\t\t int nr_packs)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n@@ -674,7 +676,26 @@ struct object_id *construct_commit_graph(const char *pack_dir)\n \t\toids.num = commit_graph->num_commits;\n \t}\n \n-\tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n+\tif (pack_indexes) {\n+\t\tint pack_dir_len = strlen(pack_dir) + 1;\n+\t\tstruct strbuf packname = STRBUF_INIT;\n+\t\tstrbuf_add(&packname, pack_dir, pack_dir_len - 1);\n+\t\tstrbuf_addch(&packname, '/');\n+\t\tfor (i = 0; i < nr_packs; i++) {\n+\t\t\tstruct packed_git *p;\n+\t\t\tstrbuf_setlen(&packname, pack_dir_len);\n+\t\t\tstrbuf_addstr(&packname, pack_indexes[i]);\n+\t\t\tp = add_packed_git(packname.buf, packname.len, 1);\n+\t\t\tif (!p)\n+\t\t\t\tdie(\"error adding pack %s\", packname.buf);\n+\t\t\tif (open_pack_index(p))\n+\t\t\t\tdie(\"error opening index for %s\", packname.buf);\n+\t\t\tfor_each_object_in_pack(p, if_packed_commit_add_to_list, &oids);\n+\t\t\tclose_pack(p);\n+\t\t}\n+\t} else {\n+\t\tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n+\t}\n \tQSORT(oids.list, oids.num, commit_compare);\n \n \tcount_distinct = 1;\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 05ddbbe165..3ae1eadce0 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -60,6 +60,8 @@ extern struct object_id *get_nth_commit_oid(struct commit_graph *g,\n \t\t\t\t\t uint32_t n,\n \t\t\t\t\t struct object_id *oid);\n \n-extern struct object_id *construct_commit_graph(const char *pack_dir);\n+extern struct object_id *construct_commit_graph(const char *pack_dir,\n+\t\t\t\t\t\tchar **pack_indexes,\n+\t\t\t\t\t\tint nr_packs);\n \n #endif\ndiff --git a/packfile.c b/packfile.c\nindex 4a5fe7ab18..48133bd669 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -299,7 +299,7 @@ void close_pack_index(struct packed_git *p)\n \t}\n }\n \n-static void close_pack(struct packed_git *p)\n+void close_pack(struct packed_git *p)\n {\n \tclose_pack_windows(p);\n \tclose_pack_fd(p);\n@@ -1860,7 +1860,7 @@ int has_pack_index(const unsigned char *sha1)\n \treturn 1;\n }\n \n-static int for_each_object_in_pack(struct packed_git *p, each_packed_object_fn cb, void *data)\n+int for_each_object_in_pack(struct packed_git *p, each_packed_object_fn cb, void *data)\n {\n \tuint32_t i;\n \tint r = 0;\ndiff --git a/packfile.h b/packfile.h\nindex 0cdeb54dcd..cde868feb6 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -61,6 +61,7 @@ extern void close_pack_index(struct packed_git *);\n \n extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n extern void close_pack_windows(struct packed_git *);\n+extern void close_pack(struct packed_git *);\n extern void close_all_packs(void);\n extern void unuse_pack(struct pack_window **);\n extern void clear_delta_base_cache(void);\n@@ -133,6 +134,7 @@ typedef int each_packed_object_fn(const struct object_id *oid,\n \t\t\t\t struct packed_git *pack,\n \t\t\t\t uint32_t pos,\n \t\t\t\t void *data);\n+extern int for_each_object_in_pack(struct packed_git *p, each_packed_object_fn cb, void *data);\n extern int for_each_packed_object(each_packed_object_fn, void *, unsigned flags);\n \n #endif\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 93b0d4f51b..b9a73f398c 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -137,7 +137,9 @@ test_expect_success 'Add more commits' \\\n git commit -m \"commit $i\" &&\n git branch commits/$i\n done &&\n- git repack'\n+ ls ${packdir} | grep idx >existing-idx &&\n+ git repack &&\n+ ls ${packdir} | grep idx | grep -v --file=existing-idx >new-idx'\n \n # Current graph structure:\n #\n@@ -174,7 +176,7 @@ _graph_git_behavior commits/20 merge/1\n _graph_git_behavior commits/20 merge/2\n \n test_expect_success 'write graph with merges' \\\n- 'graph3=$(git commit-graph --write --update-head --delete-expired) &&\n+ 'graph3=$(cat new-idx | git commit-graph --write --update-head --delete-expired --stdin-packs) &&\n test_path_is_file ${packdir}/graph-${graph3}.graph &&\n test_path_is_missing ${packdir}/graph-${graph2}.graph &&\n test_path_is_file ${packdir}/graph-${graph1}.graph &&\n-- \n2.16.0.15.g9c3cf44.dirty\n\n"},{"id":"337837","messageId":"1517348383-112294-15-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1517348383-112294-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v2 14/14] commit-graph: build graph from starting commits","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-30T21:39:43Z","receivedAt":"2018-01-30T21:40:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach git-commit-graph to read commits from stdin when the\n--stdin-commits flag is specified. Commits reachable from these\ncommits are added to the graph. This is a much faster way to construct\nthe graph than inspecting all packed objects, but is restricted to\nknown tips.\n\nFor the Linux repository, 700,000+ commits were added to the graph\nfile starting from 'master' in 7-9 seconds, depending on the number\nof packfiles in the repo (1, 24, or 120).\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 7 ++++++-\n builtin/commit-graph.c | 34 +++++++++++++++++++++++++---------\n commit-graph.c | 26 +++++++++++++++++++++++---\n commit-graph.h | 4 +++-\n t/t5318-commit-graph.sh | 18 ++++++++++++++++++\n 5 files changed, 75 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex d0571cd896..3357c0cf8f 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -46,7 +46,12 @@ OPTIONS\n --stdin-packs::\n \tWhen used with --write, generate the new graph by walking objects\n \tonly in the specified packfiles and any commits in the\n-\texisting graph-head.\n+\texisting graph-head. (Cannot be combined with --stdin-commits.)\n+\n+--stdin-commits::\n+\tWhen used with --write, generate the new graph by walking commits\n+\tstarting at the commits specified in stdin as a list of OIDs in\n+\thex, one OID per line. (Cannot be combined with --stdin-packs.)\n \n EXAMPLES\n --------\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 80a409e784..adc05f0582 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -12,7 +12,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--pack-dir ]\"),\n \tN_(\"git commit-graph --clear [--pack-dir ]\"),\n \tN_(\"git commit-graph --read [--graph-hash=]\"),\n-\tN_(\"git commit-graph --write [--pack-dir ] [--update-head] [--delete-expired] [--stdin-packs]\"),\n+\tN_(\"git commit-graph --write [--pack-dir ] [--update-head] [--delete-expired] [--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -25,6 +25,7 @@ static struct opts_commit_graph {\n \tint update_head;\n \tint delete_expired;\n \tint stdin_packs;\n+\tint stdin_commits;\n \tint has_existing;\n \tstruct object_id old_graph_hash;\n } opts;\n@@ -117,23 +118,36 @@ static int graph_write(void)\n {\n \tstruct object_id *graph_hash;\n \tchar **pack_indexes = NULL;\n+\tchar **commits = NULL;\n \tint num_packs = 0;\n-\tint size_packs = 0;\n+\tint num_commits = 0;\n+\tchar **lines = NULL;\n+\tint num_lines = 0;\n+\tint size_lines = 0;\n \n-\tif (opts.stdin_packs) {\n+\tif (opts.stdin_packs || opts.stdin_commits) {\n \t\tstruct strbuf buf = STRBUF_INIT;\n-\t\tsize_packs = 128;\n-\t\tALLOC_ARRAY(pack_indexes, size_packs);\n+\t\tsize_lines = 128;\n+\t\tALLOC_ARRAY(lines, size_lines);\n \n \t\twhile (strbuf_getline(&buf, stdin) != EOF) {\n-\t\t\tALLOC_GROW(pack_indexes, num_packs + 1, size_packs);\n-\t\t\tpack_indexes[num_packs++] = buf.buf;\n+\t\t\tALLOC_GROW(lines, num_lines + 1, size_lines);\n+\t\t\tlines[num_lines++] = buf.buf;\n \t\t\tstrbuf_detach(&buf, NULL);\n \t\t}\n-\t}\n \n-\tgraph_hash = construct_commit_graph(opts.pack_dir, pack_indexes, num_packs);\n+\t\tif (opts.stdin_packs) {\n+\t\t\tpack_indexes = lines;\n+\t\t\tnum_packs = num_lines;\n+\t\t}\n+\t\tif (opts.stdin_commits) {\n+\t\t\tcommits = lines;\n+\t\t\tnum_commits = num_lines;\n+\t\t}\n+\t}\n \n+\tgraph_hash = construct_commit_graph(opts.pack_dir, pack_indexes, num_packs,\n+\t\t\t\t\t commits, num_commits);\n \tif (opts.update_head)\n \t\tupdate_head_file(opts.pack_dir, graph_hash);\n \n@@ -172,6 +186,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"delete expired head graph file\")),\n \t\tOPT_BOOL('s', \"stdin-packs\", &opts.stdin_packs,\n \t\t\tN_(\"only scan packfiles listed by stdin\")),\n+\t\tOPT_BOOL('C', \"stdin-commits\", &opts.stdin_commits,\n+\t\t\tN_(\"start walk at commits listed by stdin\")),\n \t\t{ OPTION_STRING, 'H', \"graph-hash\", &opts.graph_hash,\n \t\t\tN_(\"hash\"),\n \t\t\tN_(\"A hash for a specific graph file in the pack-dir.\"),\ndiff --git a/commit-graph.c b/commit-graph.c\nindex cfa0415a21..7f31a6c795 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -664,7 +664,9 @@ static void close_reachable(struct packed_oid_list *oids)\n \n struct object_id *construct_commit_graph(const char *pack_dir,\n \t\t\t\t\t char **pack_indexes,\n-\t\t\t\t\t int nr_packs)\n+\t\t\t\t\t int nr_packs,\n+\t\t\t\t\t char **commit_hex,\n+\t\t\t\t\t int nr_commits)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n@@ -717,10 +719,28 @@ struct object_id *construct_commit_graph(const char *pack_dir,\n \t\t\tfor_each_object_in_pack(p, if_packed_commit_add_to_list, &oids);\n \t\t\tclose_pack(p);\n \t\t}\n-\t} else {\n-\t\tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n \t}\n \n+\tif (commit_hex) {\n+\t\tfor (i = 0; i < nr_commits; i++) {\n+\t\t\tconst char *end;\n+\t\t\tALLOC_GROW(oids.list, oids.num + 1, oids.size);\n+\n+\t\t\toids.list[oids.num] = malloc(sizeof(struct object_id));\n+\n+\t\t\tif (parse_oid_hex(commit_hex[i], oids.list[oids.num], &end)) {\n+\t\t\t\tfree(oids.list[oids.num]);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n+\t\t\tif (lookup_commit(oids.list[oids.num]))\n+\t\t\t\toids.num++;\n+\t\t}\n+\t}\n+\n+\tif (!pack_indexes && !commit_hex)\n+\t\tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n+\n \tclose_reachable(&oids);\n \tQSORT(oids.list, oids.num, commit_compare);\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex 3ae1eadce0..619b1f6def 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -62,6 +62,8 @@ extern struct object_id *get_nth_commit_oid(struct commit_graph *g,\n \n extern struct object_id *construct_commit_graph(const char *pack_dir,\n \t\t\t\t\t\tchar **pack_indexes,\n-\t\t\t\t\t\tint nr_packs);\n+\t\t\t\t\t\tint nr_packs,\n+\t\t\t\t\t\tchar **commits,\n+\t\t\t\t\t\tint nr_commits);\n \n #endif\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 2001b0b5b5..0bf27a2e7c 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -227,6 +227,24 @@ test_expect_success 'build graph from latest pack with closure' \\\n _graph_git_behavior commits/20 merge/1\n _graph_git_behavior commits/20 merge/2\n \n+test_expect_success 'build graph from commits with closure' \\\n+ 'git rev-parse commits/20 >commits-in &&\n+ git rev-parse merge/1 >>commits-in &&\n+ git rev-parse merge/2 >>commits-in &&\n+ graph6=$(cat commits-in | git commit-graph --write --update-head --delete-expired --stdin-commits) &&\n+ test_path_is_file ${packdir}/graph-${graph6}.graph &&\n+ test_path_is_missing ${packdir}/graph-${graph5}.graph &&\n+ test_path_is_file ${packdir}/graph-${graph1}.graph &&\n+ test_path_is_file ${packdir}/graph-head &&\n+ echo ${graph6} >expect &&\n+ cmp -n 40 expect ${packdir}/graph-head &&\n+ git commit-graph --read --graph-hash=${graph6} >output &&\n+ _graph_read_expect \"23\" \"${packdir}\" &&\n+ cmp expect output'\n+\n+_graph_git_behavior commits/20 merge/1\n+_graph_git_behavior commits/20 merge/2\n+\n test_expect_success 'setup bare repo' \\\n 'cd .. &&\n git clone --bare full bare &&\n-- \n2.16.0.15.g9c3cf44.dirty\n\n"},{"id":"337838","messageId":"1517348383-112294-11-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1517348383-112294-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v2 10/14] commit-graph: add core.commitgraph setting","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-30T21:39:39Z","receivedAt":"2018-01-30T21:40:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit graph feature is controlled by the new core.commitgraph config\nsetting. This defaults to 0, so the feature is opt-in.\n\nThe intention of core.commitgraph is that a user can always stop checking\nfor or parsing commit graph files if core.commitgraph=0.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 3 +++\n cache.h | 1 +\n config.c | 5 +++++\n environment.c | 1 +\n 4 files changed, 10 insertions(+)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 0e25b2c92b..5b63559a2b 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -898,6 +898,9 @@ core.notesRef::\n This setting defaults to \"refs/notes/commits\", and it can be overridden by\n the `GIT_NOTES_REF` environment variable. See linkgit:git-notes[1].\n \n+core.commitgraph::\n+\tEnable git commit graph feature. Allows reading from .graph files.\n+\n core.sparseCheckout::\n \tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n \tlinkgit:git-read-tree[1] for more information.\ndiff --git a/cache.h b/cache.h\nindex d8b975a571..e50e447a4f 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -825,6 +825,7 @@ extern char *git_replace_ref_base;\n extern int fsync_object_files;\n extern int core_preload_index;\n extern int core_apply_sparse_checkout;\n+extern int core_commitgraph;\n extern int precomposed_unicode;\n extern int protect_hfs;\n extern int protect_ntfs;\ndiff --git a/config.c b/config.c\nindex e617c2018d..99153fcfdb 100644\n--- a/config.c\n+++ b/config.c\n@@ -1223,6 +1223,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.commitgraph\")) {\n+\t\tcore_commitgraph = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.sparsecheckout\")) {\n \t\tcore_apply_sparse_checkout = git_config_bool(var, value);\n \t\treturn 0;\ndiff --git a/environment.c b/environment.c\nindex 63ac38a46f..faa4323cc5 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -61,6 +61,7 @@ enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n char *notes_ref_name;\n int grafts_replace_parents = 1;\n int core_apply_sparse_checkout;\n+int core_commitgraph;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n unsigned long pack_size_limit_cfg;\n-- \n2.16.0.15.g9c3cf44.dirty\n\n"},{"id":"337839","messageId":"1517348383-112294-3-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1517348383-112294-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v2 02/14] graph: add commit graph design document","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-30T21:39:31Z","receivedAt":"2018-01-30T21:40:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Add Documentation/technical/commit-graph.txt with details of the planned\ncommit graph feature, including future plans.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 189 +++++++++++++++++++++++++++++++\n 1 file changed, 189 insertions(+)\n create mode 100644 Documentation/technical/commit-graph.txt\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nnew file mode 100644\nindex 0000000000..cbf88f7264\n--- /dev/null\n+++ b/Documentation/technical/commit-graph.txt\n@@ -0,0 +1,189 @@\n+Git Commit Graph Design Notes\n+=============================\n+\n+Git walks the commit graph for many reasons, including:\n+\n+1. Listing and filtering commit history.\n+2. Computing merge bases.\n+\n+These operations can become slow as the commit count grows. The merge\n+base calculation shows up in many user-facing commands, such as 'merge-base'\n+or 'git show --remerge-diff' and can take minutes to compute depending on\n+history shape.\n+\n+There are two main costs here:\n+\n+1. Decompressing and parsing commits.\n+2. Walking the entire graph to avoid topological order mistakes.\n+\n+The commit graph file is a supplemental data structure that accelerates\n+commit graph walks. If a user downgrades or disables the 'core.commitgraph'\n+config setting, then the existing ODB is sufficient. The file is stored\n+next to packfiles either in the .git/objects/pack directory or in the pack\n+directory of an alternate.\n+\n+The commit graph file stores the commit graph structure along with some\n+extra metadata to speed up graph walks. By listing commit OIDs in lexi-\n+cographic order, we can identify an integer position for each commit and\n+refer to the parents of a commit using those integer positions. We use\n+binary search to find initial commits and then use the integer positions\n+for fast lookups during the walk.\n+\n+A consumer may load the following info for a commit from the graph:\n+\n+1. The commit OID.\n+2. The list of parents, along with their integer position.\n+3. The commit date.\n+4. The root tree OID.\n+5. The generation number (see definition below).\n+\n+Values 1-4 satisfy the requirements of parse_commit_gently().\n+\n+Define the \"generation number\" of a commit recursively as follows:\n+\n+ * A commit with no parents (a root commit) has generation number one.\n+\n+ * A commit with at least one parent has generation number one more than\n+ the largest generation number among its parents.\n+\n+Equivalently, the generation number of a commit A is one more than the\n+length of a longest path from A to a root commit. The recursive definition\n+is easier to use for computation and observing the following property:\n+\n+ If A and B are commits with generation numbers N and M, respectively,\n+ and N <= M, then A cannot reach B. That is, we know without searching\n+ that B is not an ancestor of A because it is further from a root commit\n+ than A.\n+\n+ Conversely, when checking if A is an ancestor of B, then we only need\n+ to walk commits until all commits on the walk boundary have generation\n+ number at most N. If we walk commits using a priority queue seeded by\n+ generation numbers, then we always expand the boundary commit with highest\n+ generation number and can easily detect the stopping condition.\n+\n+This property can be used to significantly reduce the time it takes to\n+walk commits and determine topological relationships. Without generation\n+numbers, the general heuristic is the following:\n+\n+ If A and B are commits with commit time X and Y, respectively, and\n+ X < Y, then A _probably_ cannot reach B.\n+\n+This heuristic is currently used whenever the computation can make\n+mistakes with topological orders (such as \"git log\" with default order),\n+but is not used when the topological order is required (such as merge\n+base calculations, \"git log --graph\").\n+\n+In practice, we expect some commits to be created recently and not stored\n+in the commit graph. We can treat these commits as having \"infinite\"\n+generation number and walk until reaching commits with known generation\n+number.\n+\n+Design Details\n+--------------\n+\n+- A graph file is stored in a file named 'graph-.graph' in the pack\n+ directory. This could be stored in an alternate.\n+\n+- The most-recent graph file hash is stored in a 'graph-head' file for\n+ immediate access and storing backup graphs. This could be stored in an\n+ alternate, and refers to a 'graph-.graph' file in the same pack\n+ directory.\n+\n+- The core.commitgraph config setting must be on to consume graph files.\n+\n+- The file format includes parameters for the object id length and hash\n+ algorithm, so a future change of hash algorithm does not require a change\n+ in format.\n+\n+Current Limitations\n+-------------------\n+\n+- Only one graph file is used at one time. This allows the integer position\n+ to seek into the single graph file. It is possible to extend the model\n+ for multiple graph files, but that is currently not part of the design.\n+\n+- .graph files are managed only by the 'commit-graph' builtin. These are not\n+ updated automatically during clone, fetch, repack, or creating new commits.\n+\n+- There is no '--verify' option for the 'commit-graph' builtin to verify the\n+ contents of the graph file agree with the contents in the ODB.\n+\n+- When rewriting the graph, we do not check for a commit still existing\n+ in the ODB, so garbage collection may remove commits.\n+\n+- Generation numbers are not computed in the current version. The file\n+ format supports storing them, along with a mechanism to upgrade from\n+ a file without generation numbers to one that uses them.\n+\n+Future Work\n+-----------\n+\n+- The file format includes room for precomputed generation numbers. These\n+ are not currently computed, so all generation numbers will be marked as\n+ 0 (or \"uncomputed\"). A later patch will include this calculation.\n+\n+- The commit graph is currently incompatible with commit grafts. This can be\n+ remedied by duplicating or refactoring the current graft logic.\n+\n+- After computing and storing generation numbers, we must make graph\n+ walks aware of generation numbers to gain the performance benefits they\n+ enable. This will mostly be accomplished by swapping a commit-date-ordered\n+ priority queue with one ordered by generation number. The following\n+ operations are important candidates:\n+\n+ - paint_down_to_common()\n+ - 'log --topo-order'\n+\n+- The graph currently only adds commits to a previously existing graph.\n+ When writing a new graph, we could check that the ODB still contains\n+ the commits and choose to remove the commits that are deleted from the\n+ ODB. For performance reasons, this check should remain optional.\n+\n+- Currently, parse_commit_gently() requires filling in the root tree\n+ object for a commit. This passes through lookup_tree() and consequently\n+ lookup_object(). Also, it calls lookup_commit() when loading the parents.\n+ These method calls check the ODB for object existence, even if the\n+ consumer does not need the content. For example, we do not need the\n+ tree contents when computing merge bases. Now that commit parsing is\n+ removed from the computation time, these lookup operations are the\n+ slowest operations keeping graph walks from being fast. Consider\n+ loading these objects without verifying their existence in the ODB and\n+ only loading them fully when consumers need them. Consider a method\n+ such as \"ensure_tree_loaded(commit)\" that fully loads a tree before\n+ using commit->tree.\n+\n+- The current design uses the 'commit-graph' builtin to generate the graph.\n+ When this feature stabilizes enough to recommend to most users, we should\n+ add automatic graph writes to common operations that create many commits.\n+ For example, one coulde compute a graph on 'clone', 'fetch', or 'repack'\n+ commands.\n+\n+- A server could provide a commit graph file as part of the network protocol\n+ to avoid extra calculations by clients.\n+\n+Related Links\n+-------------\n+[0] https://bugs.chromium.org/p/git/issues/detail?id=8\n+ Chromium work item for: Serialized Commit Graph\n+\n+[1] https://public-inbox.org/git/20110713070517.GC18566@sigill.intra.peff.net/\n+ An abandoned patch that introduced generation numbers.\n+\n+[2] https://public-inbox.org/git/20170908033403.q7e6dj7benasrjes@sigill.intra.peff.net/\n+ Discussion about generation numbers on commits and how they interact\n+ with fsck.\n+\n+[3] https://public-inbox.org/git/20170907094718.b6kuzp2uhvkmwcso@sigill.intra.peff.net/t/#m7a2ea7b355aeda962e6b86404bcbadc648abfbba\n+ More discussion about generation numbers and not storing them inside\n+ commit objects. A valuable quote:\n+\n+ \"I think we should be moving more in the direction of keeping\n+ repo-local caches for optimizations. Reachability bitmaps have been\n+ a big performance win. I think we should be doing the same with our\n+ properties of commits. Not just generation numbers, but making it\n+ cheap to access the graph structure without zlib-inflating whole\n+ commit objects (i.e., packv4 or something like the \"metapacks\" I\n+ proposed a few years ago).\"\n+\n+[4] https://public-inbox.org/git/20180108154822.54829-1-git@jeffhostetler.com/T/#u\n+ A patch to remove the ahead-behind calculation from 'status'.\n-- \n2.16.0.15.g9c3cf44.dirty\n\n"},{"id":"337840","messageId":"1517348383-112294-6-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1517348383-112294-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v2 05/14] commit-graph: implement git-commit-graph --write","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-30T21:39:34Z","receivedAt":"2018-01-30T21:40:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach git-commit-graph to write graph files. Create new test script to verify\nthis command succeeds without failure.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 18 +++++++\n builtin/commit-graph.c | 30 ++++++++++++\n t/t5318-commit-graph.sh | 96 ++++++++++++++++++++++++++++++++++++++\n 3 files changed, 144 insertions(+)\n create mode 100755 t/t5318-commit-graph.sh\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex c8ea548dfb..3f3790d9a8 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -5,3 +5,21 @@ NAME\n ----\n git-commit-graph - Write and verify Git commit graphs (.graph files)\n \n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git commit-graph' --write [--pack-dir ]\n+\n+EXAMPLES\n+--------\n+\n+* Write a commit graph file for the packed commits in your local .git folder.\n++\n+------------------------------------------------\n+$ git commit-graph --write\n+------------------------------------------------\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 2104550d25..7affd512f1 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -6,22 +6,38 @@\n #include \"lockfile.h\"\n #include \"packfile.h\"\n #include \"parse-options.h\"\n+#include \"commit-graph.h\"\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--pack-dir ]\"),\n+\tN_(\"git commit-graph --write [--pack-dir ]\"),\n \tNULL\n };\n \n static struct opts_commit_graph {\n \tconst char *pack_dir;\n+\tint write;\n } opts;\n \n+static int graph_write(void)\n+{\n+\tstruct object_id *graph_hash = construct_commit_graph(opts.pack_dir);\n+\n+\tif (graph_hash)\n+\t\tprintf(\"%s\\n\", oid_to_hex(graph_hash));\n+\n+\tfree(graph_hash);\n+\treturn 0;\n+}\n+\n int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_commit_graph_options[] = {\n \t\t{ OPTION_STRING, 'p', \"pack-dir\", &opts.pack_dir,\n \t\t\tN_(\"dir\"),\n \t\t\tN_(\"The pack directory to store the graph\") },\n+\t\tOPT_BOOL('w', \"write\", &opts.write,\n+\t\t\tN_(\"write commit graph file\")),\n \t\tOPT_END(),\n \t};\n \n@@ -29,5 +45,19 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_commit_graph_usage,\n \t\t\t\t builtin_commit_graph_options);\n \n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_commit_graph_options,\n+\t\t\t builtin_commit_graph_usage, 0);\n+\n+\tif (!opts.pack_dir) {\n+\t\tstruct strbuf path = STRBUF_INIT;\n+\t\tstrbuf_addstr(&path, get_object_directory());\n+\t\tstrbuf_addstr(&path, \"/pack\");\n+\t\topts.pack_dir = strbuf_detach(&path, NULL);\n+\t}\n+\n+\tif (opts.write)\n+\t\treturn graph_write();\n+\n \treturn 0;\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nnew file mode 100755\nindex 0000000000..6bcd1cc264\n--- /dev/null\n+++ b/t/t5318-commit-graph.sh\n@@ -0,0 +1,96 @@\n+#!/bin/sh\n+\n+test_description='commit graph'\n+. ./test-lib.sh\n+\n+test_expect_success 'setup full repo' \\\n+ 'rm -rf .git &&\n+ mkdir full &&\n+ cd full &&\n+ git init &&\n+ git config core.commitgraph true &&\n+ git config pack.threads 1 &&\n+ packdir=\".git/objects/pack\"'\n+\n+test_expect_success 'write graph with no packs' \\\n+ 'git commit-graph --write --pack-dir .'\n+\n+test_expect_success 'create commits and repack' \\\n+ 'for i in $(test_seq 5)\n+ do\n+ echo $i >$i.txt &&\n+ git add $i.txt &&\n+ git commit -m \"commit $i\" &&\n+ git branch commits/$i\n+ done &&\n+ git repack'\n+\n+test_expect_success 'write graph' \\\n+ 'graph1=$(git commit-graph --write) &&\n+ test_path_is_file ${packdir}/graph-${graph1}.graph'\n+\n+t_expect_success 'Add more commits' \\\n+ 'git reset --hard commits/3 &&\n+ for i in $(test_seq 6 10)\n+ do\n+ echo $i >$i.txt &&\n+ git add $i.txt &&\n+ git commit -m \"commit $i\" &&\n+ git branch commits/$i\n+ done &&\n+ git reset --hard commits/3 &&\n+ for i in $(test_seq 11 15)\n+ do\n+ echo $i >$i.txt &&\n+ git add $i.txt &&\n+ git commit -m \"commit $i\" &&\n+ git branch commits/$i\n+ done &&\n+ git reset --hard commits/7 &&\n+ git merge commits/11 &&\n+ git branch merge/1 &&\n+ git reset --hard commits/8 &&\n+ git merge commits/12 &&\n+ git branch merge/2 &&\n+ git reset --hard commits/5 &&\n+ git merge commits/10 commits/15 &&\n+ git branch merge/3 &&\n+ git repack'\n+\n+# Current graph structure:\n+#\n+# M3\n+# / |\\_____\n+# / 10 15\n+# / | |\n+# / 9 M2 14\n+# | |/ \\ |\n+# | 8 M1 | 13\n+# | |/ | \\_|\n+# 5 7 | 12\n+# | | \\__|\n+# 4 6 11\n+# |____/______/\n+# 3\n+# |\n+# 2\n+# |\n+# 1\n+\n+test_expect_success 'write graph with merges' \\\n+ 'graph2=$(git commit-graph --write) &&\n+ test_path_is_file ${packdir}/graph-${graph2}.graph'\n+\n+test_expect_success 'setup bare repo' \\\n+ 'cd .. &&\n+ git clone --bare full bare &&\n+ cd bare &&\n+ git config core.graph true &&\n+ git config pack.threads 1 &&\n+ baredir=\"objects/pack\"'\n+\n+test_expect_success 'write graph in bare repo' \\\n+ 'graphbare=$(git commit-graph --write) &&\n+ test_path_is_file ${baredir}/graph-${graphbare}.graph'\n+\n+test_done\n-- \n2.16.0.15.g9c3cf44.dirty\n\n"},{"id":"337841","messageId":"1517348383-112294-8-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1517348383-112294-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v2 07/14] commit-graph: implement git-commit-graph --update-head","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-30T21:39:36Z","receivedAt":"2018-01-30T21:40:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"It is possible to have multiple commit graph files in a pack directory,\nbut only one is important at a time. Use a 'graph_head' file to point\nto the important file. Teach git-commit-graph to write 'graph_head' upon\nwriting a new commit graph file.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 34 ++++++++++++++++++++++++++++++++++\n builtin/commit-graph.c | 38 +++++++++++++++++++++++++++++++++++---\n commit-graph.c | 25 +++++++++++++++++++++++++\n commit-graph.h | 2 ++\n t/t5318-commit-graph.sh | 12 ++++++++++--\n 5 files changed, 106 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 09aeaf6c82..99ced16ddc 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -12,15 +12,49 @@ SYNOPSIS\n 'git commit-graph' --write [--pack-dir ]\n 'git commit-graph' --read [--pack-dir ]\n \n+OPTIONS\n+-------\n+--pack-dir::\n+\tUse given directory for the location of packfiles, graph-head,\n+\tand graph files.\n+\n+--read::\n+\tRead a graph file given by the graph-head file and output basic\n+\tdetails about the graph file. (Cannot be combined with --write.)\n+\n+--graph-id::\n+\tWhen used with --read, consider the graph file graph-.graph.\n+\n+--write::\n+\tWrite a new graph file to the pack directory. (Cannot be combined\n+\twith --read.)\n+\n+--update-head::\n+\tWhen used with --write, update the graph-head file to point to\n+\tthe written graph file.\n+\n EXAMPLES\n --------\n \n+* Output the hash of the graph file pointed to by /graph-head.\n++\n+------------------------------------------------\n+$ git commit-graph --pack-dir=\n+------------------------------------------------\n+\n * Write a commit graph file for the packed commits in your local .git folder.\n +\n ------------------------------------------------\n $ git commit-graph --write\n ------------------------------------------------\n \n+* Write a graph file for the packed commits in your local .git folder,\n+* and update graph-head.\n++\n+------------------------------------------------\n+$ git commit-graph --write --update-head\n+------------------------------------------------\n+\n * Read basic information from a graph file.\n +\n ------------------------------------------------\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 218740b1f8..d73cbc907d 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -11,7 +11,7 @@\n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--pack-dir ]\"),\n \tN_(\"git commit-graph --read [--graph-hash=]\"),\n-\tN_(\"git commit-graph --write [--pack-dir ]\"),\n+\tN_(\"git commit-graph --write [--pack-dir ] [--update-head]\"),\n \tNULL\n };\n \n@@ -20,6 +20,9 @@ static struct opts_commit_graph {\n \tint read;\n \tconst char *graph_hash;\n \tint write;\n+\tint update_head;\n+\tint has_existing;\n+\tstruct object_id old_graph_hash;\n } opts;\n \n static int graph_read(void)\n@@ -30,8 +33,8 @@ static int graph_read(void)\n \n \tif (opts.graph_hash && strlen(opts.graph_hash) == GIT_MAX_HEXSZ)\n \t\tget_oid_hex(opts.graph_hash, &graph_hash);\n-\telse\n-\t\tdie(\"no graph hash specified\");\n+\telse if (!get_graph_head_hash(opts.pack_dir, &graph_hash))\n+\t\tdie(\"no graph-head exists\");\n \n \tgraph_file = get_commit_graph_filename_hash(opts.pack_dir, &graph_hash);\n \tgraph = load_commit_graph_one(graph_file, opts.pack_dir);\n@@ -62,10 +65,33 @@ static int graph_read(void)\n \treturn 0;\n }\n \n+static void update_head_file(const char *pack_dir, const struct object_id *graph_hash)\n+{\n+\tstruct strbuf head_path = STRBUF_INIT;\n+\tint fd;\n+\tstruct lock_file lk = LOCK_INIT;\n+\n+\tstrbuf_addstr(&head_path, pack_dir);\n+\tstrbuf_addstr(&head_path, \"/\");\n+\tstrbuf_addstr(&head_path, \"graph-head\");\n+\n+\tfd = hold_lock_file_for_update(&lk, head_path.buf, LOCK_DIE_ON_ERROR);\n+\tstrbuf_release(&head_path);\n+\n+\tif (fd < 0)\n+\t\tdie_errno(\"unable to open graph-head\");\n+\n+\twrite_in_full(fd, oid_to_hex(graph_hash), GIT_MAX_HEXSZ);\n+\tcommit_lock_file(&lk);\n+}\n+\n static int graph_write(void)\n {\n \tstruct object_id *graph_hash = construct_commit_graph(opts.pack_dir);\n \n+\tif (opts.update_head)\n+\t\tupdate_head_file(opts.pack_dir, graph_hash);\n+\n \tif (graph_hash)\n \t\tprintf(\"%s\\n\", oid_to_hex(graph_hash));\n \n@@ -83,6 +109,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\t\tN_(\"read graph file\")),\n \t\tOPT_BOOL('w', \"write\", &opts.write,\n \t\t\tN_(\"write commit graph file\")),\n+\t\tOPT_BOOL('u', \"update-head\", &opts.update_head,\n+\t\t\tN_(\"update graph-head to written graph file\")),\n \t\t{ OPTION_STRING, 'H', \"graph-hash\", &opts.graph_hash,\n \t\t\tN_(\"hash\"),\n \t\t\tN_(\"A hash for a specific graph file in the pack-dir.\"),\n@@ -109,10 +137,14 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\topts.pack_dir = strbuf_detach(&path, NULL);\n \t}\n \n+\topts.has_existing = !!get_graph_head_hash(opts.pack_dir, &opts.old_graph_hash);\n+\n \tif (opts.read)\n \t\treturn graph_read();\n \tif (opts.write)\n \t\treturn graph_write();\n \n+\tif (opts.has_existing)\n+\t\tprintf(\"%s\\n\", oid_to_hex(&opts.old_graph_hash));\n \treturn 0;\n }\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 622a650259..764e016ddb 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -35,6 +35,31 @@\n #define GRAPH_MIN_SIZE (GRAPH_CHUNKLOOKUP_SIZE + GRAPH_FANOUT_SIZE + \\\n \t\t\tGRAPH_OID_LEN + sizeof(struct commit_graph_header))\n \n+struct object_id *get_graph_head_hash(const char *pack_dir, struct object_id *hash)\n+{\n+\tstruct strbuf head_filename = STRBUF_INIT;\n+\tchar hex[GIT_MAX_HEXSZ + 1];\n+\tFILE *f;\n+\n+\tstrbuf_addstr(&head_filename, pack_dir);\n+\tstrbuf_addstr(&head_filename, \"/graph-head\");\n+\n+\tf = fopen(head_filename.buf, \"r\");\n+\tstrbuf_release(&head_filename);\n+\n+\tif (!f)\n+\t\treturn 0;\n+\n+\tif (!fgets(hex, sizeof(hex), f))\n+\t\tdie(\"failed to read graph-head\");\n+\n+\tfclose(f);\n+\n+\tif (get_oid_hex(hex, hash))\n+\t\treturn 0;\n+\treturn hash;\n+}\n+\n char* get_commit_graph_filename_hash(const char *pack_dir,\n \t\t\t\t struct object_id *hash)\n {\ndiff --git a/commit-graph.h b/commit-graph.h\nindex e046ae575c..43eb0aec84 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -4,6 +4,8 @@\n #include \"git-compat-util.h\"\n #include \"commit.h\"\n \n+extern struct object_id *get_graph_head_hash(const char *pack_dir,\n+\t\t\t\t\t struct object_id *hash);\n extern char* get_commit_graph_filename_hash(const char *pack_dir,\n \t\t\t\t\t struct object_id *hash);\n \ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex da565624e3..d1a23bcdaf 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -13,7 +13,8 @@ test_expect_success 'setup full repo' \\\n packdir=\".git/objects/pack\"'\n \n test_expect_success 'write graph with no packs' \\\n- 'git commit-graph --write --pack-dir .'\n+ 'git commit-graph --write --pack-dir . &&\n+ test_path_is_missing graph-head'\n \n test_expect_success 'create commits and repack' \\\n 'for i in $(test_seq 5)\n@@ -37,6 +38,7 @@ EOF\n test_expect_success 'write graph' \\\n 'graph1=$(git commit-graph --write) &&\n test_path_is_file ${packdir}/graph-${graph1}.graph &&\n+ test_path_is_missing ${packdir}/graph-head &&\n git commit-graph --read --graph-hash=${graph1} >output &&\n _graph_read_expect \"5\" \"${packdir}\" &&\n cmp expect output'\n@@ -90,8 +92,11 @@ test_expect_success 'Add more commits' \\\n # 1\n \n test_expect_success 'write graph with merges' \\\n- 'graph2=$(git commit-graph --write) &&\n+ 'graph2=$(git commit-graph --write --update-head) &&\n test_path_is_file ${packdir}/graph-${graph2}.graph &&\n+ test_path_is_file ${packdir}/graph-head &&\n+ echo ${graph2} >expect &&\n+ cmp -n 40 expect ${packdir}/graph-head &&\n git commit-graph --read --graph-hash=${graph2} >output &&\n _graph_read_expect \"18\" \"${packdir}\" &&\n cmp expect output'\n@@ -107,6 +112,9 @@ test_expect_success 'setup bare repo' \\\n test_expect_success 'write graph in bare repo' \\\n 'graphbare=$(git commit-graph --write) &&\n test_path_is_file ${baredir}/graph-${graphbare}.graph &&\n+ test_path_is_file ${baredir}/graph-head &&\n+ echo ${graphbare} >expect &&\n+ cmp -n 40 expect ${baredir}/graph-head &&\n git commit-graph --read --graph-hash=${graphbare} >output &&\n _graph_read_expect \"18\" \"${baredir}\" &&\n cmp expect output'\n-- \n2.16.0.15.g9c3cf44.dirty\n\n"},{"id":"337842","messageId":"1517348383-112294-5-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1517348383-112294-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v2 04/14] commit-graph: implement construct_commit_graph()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-01-30T21:39:33Z","receivedAt":"2018-01-30T21:40:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach Git to write a commit graph file by checking all packed objects\nto see if they are commits, then store the file in the given pack\ndirectory.\n\nSigned-off-by: Derrick Stolee \n---\n Makefile | 1 +\n commit-graph.c | 376 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n commit-graph.h | 20 +++\n 3 files changed, 397 insertions(+)\n create mode 100644 commit-graph.c\n create mode 100644 commit-graph.h\n\ndiff --git a/Makefile b/Makefile\nindex aee5d3f7b9..894432b35b 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -773,6 +773,7 @@ LIB_OBJS += color.o\n LIB_OBJS += column.o\n LIB_OBJS += combine-diff.o\n LIB_OBJS += commit.o\n+LIB_OBJS += commit-graph.o\n LIB_OBJS += compat/obstack.o\n LIB_OBJS += compat/terminal.o\n LIB_OBJS += config.o\ndiff --git a/commit-graph.c b/commit-graph.c\nnew file mode 100644\nindex 0000000000..db2b7390c7\n--- /dev/null\n+++ b/commit-graph.c\n@@ -0,0 +1,376 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"git-compat-util.h\"\n+#include \"pack.h\"\n+#include \"packfile.h\"\n+#include \"commit.h\"\n+#include \"object.h\"\n+#include \"commit-graph.h\"\n+\n+#define GRAPH_SIGNATURE 0x43475048 /* \"CGPH\" */\n+#define GRAPH_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n+#define GRAPH_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n+#define GRAPH_CHUNKID_DATA 0x43444154 /* \"CDAT\" */\n+#define GRAPH_CHUNKID_LARGEEDGES 0x45444745 /* \"EDGE\" */\n+\n+#define GRAPH_DATA_WIDTH 36\n+\n+#define GRAPH_VERSION_1 0x1\n+#define GRAPH_VERSION GRAPH_VERSION_1\n+\n+#define GRAPH_OID_VERSION_SHA1 1\n+#define GRAPH_OID_LEN_SHA1 20\n+#define GRAPH_OID_VERSION GRAPH_OID_VERSION_SHA1\n+#define GRAPH_OID_LEN GRAPH_OID_LEN_SHA1\n+\n+#define GRAPH_LARGE_EDGES_NEEDED 0x80000000\n+#define GRAPH_PARENT_MISSING 0x7fffffff\n+#define GRAPH_EDGE_LAST_MASK 0x7fffffff\n+#define GRAPH_PARENT_NONE 0x70000000\n+\n+#define GRAPH_LAST_EDGE 0x80000000\n+\n+#define GRAPH_FANOUT_SIZE (4*256)\n+#define GRAPH_CHUNKLOOKUP_SIZE (5 * 12)\n+#define GRAPH_MIN_SIZE (GRAPH_CHUNKLOOKUP_SIZE + GRAPH_FANOUT_SIZE + \\\n+\t\t\tGRAPH_OID_LEN + sizeof(struct commit_graph_header))\n+\n+char* get_commit_graph_filename_hash(const char *pack_dir,\n+\t\t\t\t struct object_id *hash)\n+{\n+\tsize_t len;\n+\tstruct strbuf head_path = STRBUF_INIT;\n+\tstrbuf_addstr(&head_path, pack_dir);\n+\tstrbuf_addstr(&head_path, \"/graph-\");\n+\tstrbuf_addstr(&head_path, oid_to_hex(hash));\n+\tstrbuf_addstr(&head_path, \".graph\");\n+\n+\treturn strbuf_detach(&head_path, &len);\n+}\n+\n+static void write_graph_chunk_fanout(struct sha1file *f,\n+\t\t\t\t struct commit **commits,\n+\t\t\t\t int nr_commits)\n+{\n+\tuint32_t i, count = 0;\n+\tstruct commit **list = commits;\n+\tstruct commit **last = commits + nr_commits;\n+\n+\t/*\n+\t * Write the first-level table (the list is sorted,\n+\t * but we use a 256-entry lookup to be able to avoid\n+\t * having to do eight extra binary search iterations).\n+\t */\n+\tfor (i = 0; i < 256; i++) {\n+\t\tuint32_t swap_count;\n+\n+\t\twhile (list < last) {\n+\t\t\tif ((*list)->object.oid.hash[0] != i)\n+\t\t\t\tbreak;\n+\t\t\tcount++;\n+\t\t\tlist++;\n+\t\t}\n+\n+\t\tswap_count = htonl(count);\n+\t\tsha1write(f, &swap_count, 4);\n+\t}\n+}\n+\n+static void write_graph_chunk_oids(struct sha1file *f, int hash_len,\n+\t\t\t\t struct commit **commits, int nr_commits)\n+{\n+\tstruct commit **list, **last = commits + nr_commits;\n+\tfor (list = commits; list < last; list++)\n+\t\tsha1write(f, (*list)->object.oid.hash, (int)hash_len);\n+}\n+\n+static int commit_pos(struct commit **commits, int nr_commits,\n+\t\t const struct object_id *oid, uint32_t *pos)\n+{\n+\tuint32_t first = 0, last = nr_commits;\n+\n+\twhile (first < last) {\n+\t\tuint32_t mid = first + (last - first) / 2;\n+\t\tstruct object_id *current;\n+\t\tint cmp;\n+\n+\t\tcurrent = &(commits[mid]->object.oid);\n+\t\tcmp = oidcmp(oid, current);\n+\t\tif (!cmp) {\n+\t\t\t*pos = mid;\n+\t\t\treturn 1;\n+\t\t}\n+\t\tif (cmp > 0) {\n+\t\t\tfirst = mid + 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tlast = mid;\n+\t}\n+\n+\t*pos = first;\n+\treturn 0;\n+}\n+\n+static void write_graph_chunk_data(struct sha1file *f, int hash_len,\n+\t\t\t\t struct commit **commits, int nr_commits)\n+{\n+\tstruct commit **list = commits;\n+\tstruct commit **last = commits + nr_commits;\n+\tuint32_t num_large_edges = 0;\n+\n+\twhile (list < last) {\n+\t\tstruct commit_list *parent;\n+\t\tuint32_t int_id, swap_int_id;\n+\t\tuint32_t packedDate[2];\n+\n+\t\tparse_commit(*list);\n+\t\tsha1write(f, (*list)->tree->object.oid.hash, hash_len);\n+\n+\t\tparent = (*list)->parents;\n+\n+\t\tif (!parent)\n+\t\t\tswap_int_id = htonl(GRAPH_PARENT_NONE);\n+\t\telse if (commit_pos(commits, nr_commits,\n+\t\t\t\t &(parent->item->object.oid), &int_id))\n+\t\t\tswap_int_id = htonl(int_id);\n+\t\telse\n+\t\t\tswap_int_id = htonl(GRAPH_PARENT_MISSING);\n+\n+\t\tsha1write(f, &swap_int_id, 4);\n+\n+\t\tif (parent)\n+\t\t\tparent = parent->next;\n+\n+\t\tif (!parent)\n+\t\t\tswap_int_id = htonl(GRAPH_PARENT_NONE);\n+\t\telse if (parent->next)\n+\t\t\tswap_int_id = htonl(GRAPH_LARGE_EDGES_NEEDED | num_large_edges);\n+\t\telse if (commit_pos(commits, nr_commits,\n+\t\t\t\t &(parent->item->object.oid), &int_id))\n+\t\t\tswap_int_id = htonl(int_id);\n+\t\telse\n+\t\t\tswap_int_id = htonl(GRAPH_PARENT_MISSING);\n+\n+\t\tsha1write(f, &swap_int_id, 4);\n+\n+\t\tif (parent && parent->next) {\n+\t\t\tdo {\n+\t\t\t\tnum_large_edges++;\n+\t\t\t\tparent = parent->next;\n+\t\t\t} while (parent);\n+\t\t}\n+\n+\t\tif (sizeof((*list)->date) > 4)\n+\t\t\tpackedDate[0] = htonl(((*list)->date >> 32) & 0x3);\n+\t\telse\n+\t\t\tpackedDate[0] = 0;\n+\n+\t\tpackedDate[1] = htonl((*list)->date);\n+\t\tsha1write(f, packedDate, 8);\n+\n+\t\tlist++;\n+\t}\n+}\n+\n+static void write_graph_chunk_large_edges(struct sha1file *f,\n+\t\t\t\t\t struct commit **commits,\n+\t\t\t\t\t int nr_commits)\n+{\n+\tstruct commit **list = commits;\n+\tstruct commit **last = commits + nr_commits;\n+\tstruct commit_list *parent;\n+\n+\twhile (list < last) {\n+\t\tint num_parents = 0;\n+\t\tfor (parent = (*list)->parents; num_parents < 3 && parent;\n+\t\t parent = parent->next)\n+\t\t\tnum_parents++;\n+\n+\t\tif (num_parents <= 2) {\n+\t\t\tlist++;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfor (parent = (*list)->parents; parent; parent = parent->next) {\n+\t\t\tuint32_t int_id, swap_int_id;\n+\t\t\tuint32_t last_edge = 0;\n+\n+\t\t\tif (parent == (*list)->parents)\n+\t\t\t\tcontinue;\n+\n+\t\t\tif (!parent->next)\n+\t\t\t\tlast_edge |= GRAPH_LAST_EDGE;\n+\n+\t\t\tif (commit_pos(commits, nr_commits,\n+\t\t\t\t &(parent->item->object.oid),\n+\t\t\t\t &int_id))\n+\t\t\t\tswap_int_id = htonl(int_id | last_edge);\n+\t\t\telse\n+\t\t\t\tswap_int_id = htonl(GRAPH_PARENT_MISSING | last_edge);\n+\n+\t\t\tsha1write(f, &swap_int_id, 4);\n+\t\t}\n+\n+\t\tlist++;\n+\t}\n+}\n+\n+static int commit_compare(const void *_a, const void *_b)\n+{\n+\tstruct object_id *a = *(struct object_id **)_a;\n+\tstruct object_id *b = *(struct object_id **)_b;\n+\treturn oidcmp(a, b);\n+}\n+\n+struct packed_commit_list {\n+\tstruct commit **list;\n+\tint num;\n+\tint size;\n+};\n+\n+struct packed_oid_list {\n+\tstruct object_id **list;\n+\tint num;\n+\tint size;\n+};\n+\n+static int if_packed_commit_add_to_list(const struct object_id *oid,\n+\t\t\t\t\tstruct packed_git *pack,\n+\t\t\t\t\tuint32_t pos,\n+\t\t\t\t\tvoid *data)\n+{\n+\tstruct packed_oid_list *list = (struct packed_oid_list*)data;\n+\tenum object_type type;\n+\tunsigned long size;\n+\tvoid *inner_data;\n+\toff_t offset = nth_packed_object_offset(pack, pos);\n+\tinner_data = unpack_entry(pack, offset, &type, &size);\n+\n+\tif (inner_data)\n+\t\tfree(inner_data);\n+\n+\tif (type != OBJ_COMMIT)\n+\t\treturn 0;\n+\n+\tALLOC_GROW(list->list, list->num + 1, list->size);\n+\tlist->list[list->num] = (struct object_id *)malloc(sizeof(struct object_id));\n+\toidcpy(list->list[list->num], oid);\n+\t(list->num)++;\n+\n+\treturn 0;\n+}\n+\n+struct object_id *construct_commit_graph(const char *pack_dir)\n+{\n+\tstruct packed_oid_list oids;\n+\tstruct packed_commit_list commits;\n+\tstruct commit_graph_header hdr;\n+\tstruct sha1file *f;\n+\tint i, count_distinct = 0;\n+\tstruct strbuf tmp_file = STRBUF_INIT;\n+\tunsigned char final_hash[GIT_MAX_RAWSZ];\n+\tchar *graph_name;\n+\tint fd;\n+\tuint32_t chunk_ids[5];\n+\tuint64_t chunk_offsets[5];\n+\tint num_long_edges;\n+\tstruct object_id *f_hash;\n+\tchar *fname;\n+\tstruct commit_list *parent;\n+\n+\toids.num = 0;\n+\toids.size = 1024;\n+\tALLOC_ARRAY(oids.list, oids.size);\n+\tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n+\tQSORT(oids.list, oids.num, commit_compare);\n+\n+\tcount_distinct = 1;\n+\tfor (i = 1; i < oids.num; i++) {\n+\t\tif (oidcmp(oids.list[i-1], oids.list[i]))\n+\t\t\tcount_distinct++;\n+\t}\n+\n+\tcommits.num = 0;\n+\tcommits.size = count_distinct;\n+\tALLOC_ARRAY(commits.list, commits.size);\n+\n+\tnum_long_edges = 0;\n+\tfor (i = 0; i < oids.num; i++) {\n+\t\tint num_parents = 0;\n+\t\tif (i > 0 && !oidcmp(oids.list[i-1], oids.list[i]))\n+\t\t\tcontinue;\n+\n+\t\tcommits.list[commits.num] = lookup_commit(oids.list[i]);\n+\t\tparse_commit(commits.list[commits.num]);\n+\n+\t\tfor (parent = commits.list[commits.num]->parents;\n+\t\t parent; parent = parent->next)\n+\t\t\tnum_parents++;\n+\n+\t\tif (num_parents > 2)\n+\t\t\tnum_long_edges += num_parents - 1;\n+\n+\t\tcommits.num++;\n+\t}\n+\n+\tstrbuf_addstr(&tmp_file, pack_dir);\n+\tstrbuf_addstr(&tmp_file, \"/tmp_graph_XXXXXX\");\n+\n+\tfd = git_mkstemp_mode(tmp_file.buf, 0444);\n+\tif (fd < 0)\n+\t\tdie_errno(\"unable to create '%s'\", tmp_file.buf);\n+\n+\tgraph_name = strbuf_detach(&tmp_file, NULL);\n+\tf = sha1fd(fd, graph_name);\n+\n+\thdr.graph_signature = htonl(GRAPH_SIGNATURE);\n+\thdr.graph_version = GRAPH_VERSION;\n+\thdr.hash_version = GRAPH_OID_VERSION;\n+\thdr.hash_len = GRAPH_OID_LEN;\n+\thdr.num_chunks = 4;\n+\n+\tassert(sizeof(hdr) == 8);\n+\tsha1write(f, &hdr, sizeof(hdr));\n+\n+\tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n+\tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n+\tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n+\tchunk_ids[3] = GRAPH_CHUNKID_LARGEEDGES;\n+\tchunk_ids[4] = 0;\n+\n+\tchunk_offsets[0] = sizeof(hdr) + GRAPH_CHUNKLOOKUP_SIZE;\n+\tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n+\tchunk_offsets[2] = chunk_offsets[1] + GRAPH_OID_LEN * commits.num;\n+\tchunk_offsets[3] = chunk_offsets[2] + (GRAPH_OID_LEN + 16) * commits.num;\n+\tchunk_offsets[4] = chunk_offsets[3] + 4 * num_long_edges;\n+\n+\tfor (i = 0; i <= hdr.num_chunks; i++) {\n+\t\tuint32_t chunk_write[3];\n+\n+\t\tchunk_write[0] = htonl(chunk_ids[i]);\n+\t\tchunk_write[1] = htonl(chunk_offsets[i] >> 32);\n+\t\tchunk_write[2] = htonl(chunk_offsets[i] & 0xffffffff);\n+\t\tsha1write(f, chunk_write, 12);\n+\t}\n+\n+\twrite_graph_chunk_fanout(f, commits.list, commits.num);\n+\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.num);\n+\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.num);\n+\twrite_graph_chunk_large_edges(f, commits.list, commits.num);\n+\n+\tsha1close(f, final_hash, CSUM_CLOSE | CSUM_FSYNC);\n+\n+\tf_hash = (struct object_id *)malloc(sizeof(struct object_id));\n+\tmemcpy(f_hash->hash, final_hash, GIT_MAX_RAWSZ);\n+\tfname = get_commit_graph_filename_hash(pack_dir, f_hash);\n+\n+\tif (rename(graph_name, fname))\n+\t\tdie(\"failed to rename %s to %s\", graph_name, fname);\n+\n+\tfree(oids.list);\n+\toids.size = 0;\n+\toids.num = 0;\n+\n+\treturn f_hash;\n+}\n+\ndiff --git a/commit-graph.h b/commit-graph.h\nnew file mode 100644\nindex 0000000000..7b3469a7df\n--- /dev/null\n+++ b/commit-graph.h\n@@ -0,0 +1,20 @@\n+#ifndef COMMIT_GRAPH_H\n+#define COMMIT_GRAPH_H\n+\n+#include \"git-compat-util.h\"\n+#include \"commit.h\"\n+\n+extern char* get_commit_graph_filename_hash(const char *pack_dir,\n+\t\t\t\t\t struct object_id *hash);\n+\n+struct commit_graph_header {\n+\tuint32_t graph_signature;\n+\tunsigned char graph_version;\n+\tunsigned char hash_version;\n+\tunsigned char hash_len;\n+\tunsigned char num_chunks;\n+};\n+\n+extern struct object_id *construct_commit_graph(const char *pack_dir);\n+\n+#endif\n-- \n2.16.0.15.g9c3cf44.dirty\n\n"},{"id":"337843","messageId":"CAGZ79karMKMTAFZma8WiThacyWp7FqYPd6NNZ93s4xa48SnOyA@mail.gmail.com","threadId":"47724","inReplyTo":"1517348383-112294-1-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v2 00/14] Serialized Git Commit Graph","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-01-30T21:47:54Z","receivedAt":"2018-01-30T21:48:00Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Jan 30, 2018 at 1:39 PM, Derrick Stolee wrote:\n> Thanks to everyone who gave comments on v1. I tried my best to respond to\n> all of the feedback, but may have missed some while I was doing several\n> renames, including:\n>\n> * builtin/graph.c -> builtin/commit-graph.c\n> * packed-graph.[c|h] -> commit-graph.[c|h]\n> * t/t5319-graph.sh -> t/t5318-commit-graph.sh\n>\n> Because of these renames (and several type/function renames) the diff\n> is too large to conveniently share here.\n>\n> Some issues that came up and are addressed:\n>\n> * Use instead of when referring to the graph-.graph\n> filenames and the contents of graph-head.\n> * 32-bit timestamps will not cause undefined behavior.\n> * timestamp_t is unsigned, so they are never negative.\n> * The config setting \"core.commitgraph\" now only controls consuming the\n> graph during normal operations and will not block the commit-graph\n> plumbing command.\n> * The --stdin-commits is better about sanitizing the input for strings\n> that do not parse to OIDs or are OIDs for non-commit objects.\n>\n> One unresolved comment that I would like consensus on is the use of\n> globals to store the config setting and the graph state. I'm currently\n> using the pattern from packed_git instead of putting these values in\n> the_repository. However, we want to eventually remove globals like\n> packed_git. Should I deviate from the pattern _now_ in order to keep\n> the problem from growing, or should I keep to the known pattern?\n\nI have a series doing the conversion in\nhttps://github.com/stefanbeller/git/tree/object-store\nthat is based on 2.16.\n\nWhile the commits are structured for easy review (to not miss any of\nthe globals that that series is based upon), I did not come up with a\ngood strategy how to take care of series in flight that add more globals.\n\nSo I think for now you'd want to keep it as global vars, such that\nit is consistent with the code base and then we'll figure out how to\ndo the conversion one step at a time.\n\nPlease do not feel stopped or hindered by my slow pace of working\nthrough that series, maybe I'll have to come up with another approach\nthat is better for upstream (rebasing that series is a pain, as upstream\nmoves rather quickly. Maybe I'll have to send that series in smaller chunks).\n\n> Finally, I tried to clean up my incorrect style as I was recreating\n> these commits. Feel free to be merciless in style feedback now that the\n> architecture is more stable.\n\nok, will do.\n\nThanks,\nStefan\n"},{"id":"337898","messageId":"CAGZ79kaAnQYQ2pn+=R6pKdESvxaFRLsiXJ4oMf8cD00fRyyNxA@mail.gmail.com","threadId":"47724","inReplyTo":"1517348383-112294-3-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v2 02/14] graph: add commit graph design document","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-01-31T02:19:48Z","receivedAt":"2018-01-31T02:20:48Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Jan 30, 2018 at 1:39 PM, Derrick Stolee wrote:\n> Add Documentation/technical/commit-graph.txt with details of the planned\n> commit graph feature, including future plans.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/technical/commit-graph.txt | 189 +++++++++++++++++++++++++++++++\n> 1 file changed, 189 insertions(+)\n> create mode 100644 Documentation/technical/commit-graph.txt\n>\n> diff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\n> new file mode 100644\n> index 0000000000..cbf88f7264\n> --- /dev/null\n> +++ b/Documentation/technical/commit-graph.txt\n> @@ -0,0 +1,189 @@\n> +Git Commit Graph Design Notes\n> +=============================\n> +\n> +Git walks the commit graph for many reasons, including:\n> +\n> +1. Listing and filtering commit history.\n> +2. Computing merge bases.\n> +\n> +These operations can become slow as the commit count grows. The merge\n> +base calculation shows up in many user-facing commands, such as 'merge-base'\n> +or 'git show --remerge-diff' and can take minutes to compute depending on\n> +history shape.\n\nSorry for appearing more authoritative than I am here. The --remerge-diff flag\nis just floating around the mailing list, and was never merged. (It is\nsuch a cool\nfeature though, but it would actually confuse users looking for it,\nnot finding it)\n\n\n> +There are two main costs here:\n> +\n> +1. Decompressing and parsing commits.\n> +2. Walking the entire graph to avoid topological order mistakes.\n> +\n> +The commit graph file is a supplemental data structure that accelerates\n> +commit graph walks. If a user downgrades or disables the 'core.commitgraph'\n> +config setting, then the existing ODB is sufficient. The file is stored\n> +next to packfiles either in the .git/objects/pack directory or in the pack\n> +directory of an alternate.\n> +\n> +The commit graph file stores the commit graph structure along with some\n> +extra metadata to speed up graph walks. By listing commit OIDs in lexi-\n> +cographic order, we can identify an integer position for each commit and\n> +refer to the parents of a commit using those integer positions. We use\n> +binary search to find initial commits and then use the integer positions\n> +for fast lookups during the walk.\n> +\n> +A consumer may load the following info for a commit from the graph:\n> +\n> +1. The commit OID.\n> +2. The list of parents, along with their integer position.\n> +3. The commit date.\n> +4. The root tree OID.\n> +5. The generation number (see definition below).\n> +\n> +Values 1-4 satisfy the requirements of parse_commit_gently().\n> +\n> +Define the \"generation number\" of a commit recursively as follows:\n> +\n> + * A commit with no parents (a root commit) has generation number one.\n> +\n> + * A commit with at least one parent has generation number one more than\n> + the largest generation number among its parents.\n> +\n> +Equivalently, the generation number of a commit A is one more than the\n> +length of a longest path from A to a root commit. The recursive definition\n> +is easier to use for computation and observing the following property:\n> +\n> + If A and B are commits with generation numbers N and M, respectively,\n> + and N <= M, then A cannot reach B. That is, we know without searching\n> + that B is not an ancestor of A because it is further from a root commit\n> + than A.\n> +\n> + Conversely, when checking if A is an ancestor of B, then we only need\n> + to walk commits until all commits on the walk boundary have generation\n> + number at most N. If we walk commits using a priority queue seeded by\n> + generation numbers, then we always expand the boundary commit with highest\n> + generation number and can easily detect the stopping condition.\n> +\n> +This property can be used to significantly reduce the time it takes to\n> +walk commits and determine topological relationships. Without generation\n> +numbers, the general heuristic is the following:\n> +\n> + If A and B are commits with commit time X and Y, respectively, and\n> + X < Y, then A _probably_ cannot reach B.\n> +\n> +This heuristic is currently used whenever the computation can make\n> +mistakes with topological orders (such as \"git log\" with default order),\n> +but is not used when the topological order is required (such as merge\n> +base calculations, \"git log --graph\").\n> +\n> +In practice, we expect some commits to be created recently and not stored\n> +in the commit graph. We can treat these commits as having \"infinite\"\n> +generation number and walk until reaching commits with known generation\n> +number.\n> +\n> +Design Details\n> +--------------\n> +\n> +- A graph file is stored in a file named 'graph-.graph' in the pack\n> + directory. This could be stored in an alternate.\n> +\n> +- The most-recent graph file hash is stored in a 'graph-head' file for\n> + immediate access and storing backup graphs. This could be stored in an\n> + alternate, and refers to a 'graph-.graph' file in the same pack\n> + directory.\n> +\n> +- The core.commitgraph config setting must be on to consume graph files.\n> +\n> +- The file format includes parameters for the object id length and hash\n> + algorithm, so a future change of hash algorithm does not require a change\n> + in format.\n> +\n> +Current Limitations\n> +-------------------\n> +\n> +- Only one graph file is used at one time. This allows the integer position\n> + to seek into the single graph file. It is possible to extend the model\n> + for multiple graph files, but that is currently not part of the design.\n> +\n> +- .graph files are managed only by the 'commit-graph' builtin. These are not\n> + updated automatically during clone, fetch, repack, or creating new commits.\n> +\n> +- There is no '--verify' option for the 'commit-graph' builtin to verify the\n> + contents of the graph file agree with the contents in the ODB.\n> +\n> +- When rewriting the graph, we do not check for a commit still existing\n> + in the ODB, so garbage collection may remove commits.\n> +\n> +- Generation numbers are not computed in the current version. The file\n> + format supports storing them, along with a mechanism to upgrade from\n> + a file without generation numbers to one that uses them.\n> +\n> +Future Work\n> +-----------\n> +\n> +- The file format includes room for precomputed generation numbers. These\n> + are not currently computed, so all generation numbers will be marked as\n> + 0 (or \"uncomputed\"). A later patch will include this calculation.\n> +\n> +- The commit graph is currently incompatible with commit grafts. This can be\n> + remedied by duplicating or refactoring the current graft logic.\n> +\n> +- After computing and storing generation numbers, we must make graph\n> + walks aware of generation numbers to gain the performance benefits they\n> + enable. This will mostly be accomplished by swapping a commit-date-ordered\n> + priority queue with one ordered by generation number. The following\n> + operations are important candidates:\n> +\n> + - paint_down_to_common()\n> + - 'log --topo-order'\n> +\n> +- The graph currently only adds commits to a previously existing graph.\n> + When writing a new graph, we could check that the ODB still contains\n> + the commits and choose to remove the commits that are deleted from the\n> + ODB. For performance reasons, this check should remain optional.\n> +\n> +- Currently, parse_commit_gently() requires filling in the root tree\n> + object for a commit. This passes through lookup_tree() and consequently\n> + lookup_object(). Also, it calls lookup_commit() when loading the parents.\n> + These method calls check the ODB for object existence, even if the\n> + consumer does not need the content. For example, we do not need the\n> + tree contents when computing merge bases. Now that commit parsing is\n> + removed from the computation time, these lookup operations are the\n> + slowest operations keeping graph walks from being fast. Consider\n> + loading these objects without verifying their existence in the ODB and\n> + only loading them fully when consumers need them. Consider a method\n> + such as \"ensure_tree_loaded(commit)\" that fully loads a tree before\n> + using commit->tree.\n> +\n> +- The current design uses the 'commit-graph' builtin to generate the graph.\n> + When this feature stabilizes enough to recommend to most users, we should\n> + add automatic graph writes to common operations that create many commits.\n> + For example, one coulde compute a graph on 'clone', 'fetch', or 'repack'\n> + commands.\n> +\n> +- A server could provide a commit graph file as part of the network protocol\n> + to avoid extra calculations by clients.\n> +\n> +Related Links\n> +-------------\n> +[0] https://bugs.chromium.org/p/git/issues/detail?id=8\n> + Chromium work item for: Serialized Commit Graph\n> +\n> +[1] https://public-inbox.org/git/20110713070517.GC18566@sigill.intra.peff.net/\n> + An abandoned patch that introduced generation numbers.\n> +\n> +[2] https://public-inbox.org/git/20170908033403.q7e6dj7benasrjes@sigill.intra.peff.net/\n> + Discussion about generation numbers on commits and how they interact\n> + with fsck.\n> +\n> +[3] https://public-inbox.org/git/20170907094718.b6kuzp2uhvkmwcso@sigill.intra.peff.net/t/#m7a2ea7b355aeda962e6b86404bcbadc648abfbba\n> + More discussion about generation numbers and not storing them inside\n> + commit objects. A valuable quote:\n> +\n> + \"I think we should be moving more in the direction of keeping\n> + repo-local caches for optimizations. Reachability bitmaps have been\n> + a big performance win. I think we should be doing the same with our\n> + properties of commits. Not just generation numbers, but making it\n> + cheap to access the graph structure without zlib-inflating whole\n> + commit objects (i.e., packv4 or something like the \"metapacks\" I\n> + proposed a few years ago).\"\n> +\n> +[4] https://public-inbox.org/git/20180108154822.54829-1-git@jeffhostetler.com/T/#u\n> + A patch to remove the ahead-behind calculation from 'status'.\n> --\n> 2.16.0.15.g9c3cf44.dirty\n>\n"},{"id":"337899","messageId":"CAGZ79kbt3+MLvw-UsigNq7mf7kxK3KhjJ-pEqFe1R03bNFMhKQ@mail.gmail.com","threadId":"47724","inReplyTo":"1517348383-112294-7-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v2 06/14] commit-graph: implement git-commit-graph --read","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-01-31T02:22:49Z","receivedAt":"2018-01-31T02:22:56Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> +static void free_commit_graph(struct commit_graph **g)\n> +{\n> + if (!g || !*g)\n> + return;\n> +\n> + close_commit_graph(*g);\n> +\n> + free(*g);\n> + *g = NULL;\n\nnit: You may want to use FREE_AND_NULL(*g) instead.\n"},{"id":"337980","messageId":"b6216601-603b-f1bd-36e6-58e9f84946d4@gmail.com","threadId":"47724","inReplyTo":"1517348383-112294-11-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v2 10/14] commit-graph: add core.commitgraph setting","fromName":"Igor Djordjevic","fromEmail":"igor.d.djordjevic@gmail.com","sentAt":"2018-01-31T22:44:27Z","receivedAt":"2018-01-31T22:44:38Z","isPatch":true,"sender":{"key":"igor.d.djordjevic@gmail.com","avatar":null},"body":"Hi Derrick,\n\nOn 30/01/2018 22:39, Derrick Stolee wrote:\n>\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index 0e25b2c92b..5b63559a2b 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -898,6 +898,9 @@ core.notesRef::\n> This setting defaults to \"refs/notes/commits\", and it can be overridden by\n> the `GIT_NOTES_REF` environment variable. See linkgit:git-notes[1].\n> \n> +core.commitgraph::\n ^^^\nA small style nitpick - you may want to use \"core.commitGraph\" \nthroughout the series (note capital \"G\"), making it more readable and \naligning with the rest of `git config` variable names (using \"bumpyCaps\" \nas per coding guidelines[1], and as seen a few lines below, at the \nend of this very patch, too, \"core.sparseCheckout\").\n\n> +\tEnable git commit graph feature. Allows reading from .graph files.\n> +\n> core.sparseCheckout::\n> \tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n> \tlinkgit:git-read-tree[1] for more information.\n> diff --git a/cache.h b/cache.h\n> index d8b975a571..e50e447a4f 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -825,6 +825,7 @@ extern char *git_replace_ref_base;\n> extern int fsync_object_files;\n> extern int core_preload_index;\n> extern int core_apply_sparse_checkout;\n> +extern int core_commitgraph;\n ^^^\nSimilar nit here, might be \"core_commit_graph\" (throughout the \nseries) would align better with existing variable names around it \n(note additional underscore between \"commit\" and \"graph\"), but also \nwith your own naming \"scheme\" used for cmd_commit_graph(), \nbuiltin_commit_graph_usage[], construct_commit_graph(), etc.\n\n> extern int precomposed_unicode;\n> extern int protect_hfs;\n> extern int protect_ntfs;\n> diff --git a/config.c b/config.c\n> index e617c2018d..99153fcfdb 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -1223,6 +1223,11 @@ static int git_default_core_config(const char *var, const char *value)\n> \t\treturn 0;\n> \t}\n> \n> +\tif (!strcmp(var, \"core.commitgraph\")) {\n> +\t\tcore_commitgraph = git_config_bool(var, value);\n> +\t\treturn 0;\n> +\t}\n> +\n> \tif (!strcmp(var, \"core.sparsecheckout\")) {\n> \t\tcore_apply_sparse_checkout = git_config_bool(var, value);\n> \t\treturn 0;\n> diff --git a/environment.c b/environment.c\n> index 63ac38a46f..faa4323cc5 100644\n> --- a/environment.c\n> +++ b/environment.c\n> @@ -61,6 +61,7 @@ enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n> char *notes_ref_name;\n> int grafts_replace_parents = 1;\n> int core_apply_sparse_checkout;\n> +int core_commitgraph;\n> int merge_log_config = -1;\n> int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n> unsigned long pack_size_limit_cfg;\n> \n\nThanks, Buga\n\n[1] https://github.com/git/git/blob/master/Documentation/CodingGuidelines\n\n Externally Visible Names\n \n ...\n \n The section and variable names that consist of multiple words are\n formed by concatenating the words without punctuations (e.g. `-`),\n and are broken using bumpyCaps in documentation as a hint to the\n reader.\n"},{"id":"338003","messageId":"CAGZ79kY59aSBQCaC8XKeNPuOz6yJFYCMSvOn2webitWRGbR5KA@mail.gmail.com","threadId":"47724","inReplyTo":"CAGZ79karMKMTAFZma8WiThacyWp7FqYPd6NNZ93s4xa48SnOyA@mail.gmail.com","subject":"Re: [PATCH v2 00/14] Serialized Git Commit Graph","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-01T02:34:03Z","receivedAt":"2018-02-01T02:34:12Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":">> Finally, I tried to clean up my incorrect style as I was recreating\n>> these commits. Feel free to be merciless in style feedback now that the\n>> architecture is more stable.\n>\n> ok, will do.\n\nI reviewed all patches and found no nits worth mentioning.\nThe whole series has been reviewed by me.\n\n>\n> Thanks,\n> Stefan\n"},{"id":"338049","messageId":"20180201134422.1662e936efb55b107cee2efa@google.com","threadId":"47724","inReplyTo":"1517348383-112294-2-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v2 01/14] commit-graph: add format document","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-01T21:44:22Z","receivedAt":"2018-02-01T21:44:29Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 30 Jan 2018 16:39:30 -0500\nDerrick Stolee wrote:\n\n> + Commit Data (ID: {'C', 'G', 'E', 'T' }) (N * (H + 16) bytes)\n> + * The first H bytes are for the OID of the root tree.\n> + * The next 8 bytes are for the int-ids of the first two parents\n> + of the ith commit. Stores value 0xffffffff if no parent in that\n> + position. If there are more than two parents, the second value\n> + has its most-significant bit on and the other bits store an array\n> + position into the Large Edge List chunk.\n\n[snip]\n\n> + Large Edge List (ID: {'E', 'D', 'G', 'E'})\n> + This list of 4-byte values store the second through nth parents for\n> + all octopus merges. The second parent value in the commit data is a\n> + negative number pointing into this list. \n\nLooking at the paragraph which I quoted before the [snip], this sentence\nis confusing (in particular, the second parent value is not interpreted\nas the normal two's-complement negative value, and the semantics of\nnegative values indexing into the list is unclear). Maybe it's better to\nomit it entirely.\n"},{"id":"338051","messageId":"20180201142302.02182cee02d4e44134a1be50@google.com","threadId":"47724","inReplyTo":"1517348383-112294-5-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v2 04/14] commit-graph: implement construct_commit_graph()","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-01T22:23:02Z","receivedAt":"2018-02-01T22:23:10Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 30 Jan 2018 16:39:33 -0500\nDerrick Stolee wrote:\n\n> +#define GRAPH_SIGNATURE 0x43475048 /* \"CGPH\" */\n> +#define GRAPH_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n> +#define GRAPH_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n> +#define GRAPH_CHUNKID_DATA 0x43444154 /* \"CDAT\" */\n> +#define GRAPH_CHUNKID_LARGEEDGES 0x45444745 /* \"EDGE\" */\n\nCould all these just be string constants? sha1write can handle them well\nenough.\n\n> +static void write_graph_chunk_fanout(struct sha1file *f,\n> +\t\t\t\t struct commit **commits,\n> +\t\t\t\t int nr_commits)\n> +{\n> +\tuint32_t i, count = 0;\n> +\tstruct commit **list = commits;\n> +\tstruct commit **last = commits + nr_commits;\n> +\n> +\t/*\n> +\t * Write the first-level table (the list is sorted,\n> +\t * but we use a 256-entry lookup to be able to avoid\n> +\t * having to do eight extra binary search iterations).\n> +\t */\n> +\tfor (i = 0; i < 256; i++) {\n> +\t\tuint32_t swap_count;\n> +\n> +\t\twhile (list < last) {\n> +\t\t\tif ((*list)->object.oid.hash[0] != i)\n> +\t\t\t\tbreak;\n> +\t\t\tcount++;\n> +\t\t\tlist++;\n> +\t\t}\n> +\n> +\t\tswap_count = htonl(count);\n> +\t\tsha1write(f, &swap_count, 4);\n\nYou can use sha1write_be32() instead of swapping.\n\n> +static void write_graph_chunk_large_edges(struct sha1file *f,\n> +\t\t\t\t\t struct commit **commits,\n> +\t\t\t\t\t int nr_commits)\n> +{\n> +\tstruct commit **list = commits;\n> +\tstruct commit **last = commits + nr_commits;\n> +\tstruct commit_list *parent;\n> +\n> +\twhile (list < last) {\n> +\t\tint num_parents = 0;\n> +\t\tfor (parent = (*list)->parents; num_parents < 3 && parent;\n> +\t\t parent = parent->next)\n> +\t\t\tnum_parents++;\n> +\n> +\t\tif (num_parents <= 2) {\n> +\t\t\tlist++;\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tfor (parent = (*list)->parents; parent; parent = parent->next) {\n> +\t\t\tuint32_t int_id, swap_int_id;\n> +\t\t\tuint32_t last_edge = 0;\n> +\n> +\t\t\tif (parent == (*list)->parents)\n> +\t\t\t\tcontinue;\n\nProbably better to just initialize \"parent = (*list)->parents->next\".\nAlso probably best to add a comment describing why you are doing this\n(e.g. \"The first parent is already in the main commit table; the large\nedges table only contains the second parent onwards\").\n\n> +struct packed_commit_list {\n> +\tstruct commit **list;\n> +\tint num;\n> +\tint size;\n> +};\n> +\n> +struct packed_oid_list {\n> +\tstruct object_id **list;\n> +\tint num;\n> +\tint size;\n> +};\n\nWhat are num and size? If they're nr and alloc, maybe use those names\ninstead.\n\n> +static int if_packed_commit_add_to_list(const struct object_id *oid,\n> +\t\t\t\t\tstruct packed_git *pack,\n> +\t\t\t\t\tuint32_t pos,\n> +\t\t\t\t\tvoid *data)\n> +{\n> +\tstruct packed_oid_list *list = (struct packed_oid_list*)data;\n> +\tenum object_type type;\n> +\tunsigned long size;\n> +\tvoid *inner_data;\n> +\toff_t offset = nth_packed_object_offset(pack, pos);\n> +\tinner_data = unpack_entry(pack, offset, &type, &size);\n> +\n> +\tif (inner_data)\n> +\t\tfree(inner_data);\n> +\n> +\tif (type != OBJ_COMMIT)\n> +\t\treturn 0;\n> +\n> +\tALLOC_GROW(list->list, list->num + 1, list->size);\n> +\tlist->list[list->num] = (struct object_id *)malloc(sizeof(struct object_id));\n\nNo need to cast return value of malloc. Also, use xmalloc?\n\n> +struct object_id *construct_commit_graph(const char *pack_dir)\n> +{\n> +\tstruct packed_oid_list oids;\n> +\tstruct packed_commit_list commits;\n> +\tstruct commit_graph_header hdr;\n> +\tstruct sha1file *f;\n> +\tint i, count_distinct = 0;\n> +\tstruct strbuf tmp_file = STRBUF_INIT;\n> +\tunsigned char final_hash[GIT_MAX_RAWSZ];\n> +\tchar *graph_name;\n> +\tint fd;\n> +\tuint32_t chunk_ids[5];\n> +\tuint64_t chunk_offsets[5];\n> +\tint num_long_edges;\n> +\tstruct object_id *f_hash;\n> +\tchar *fname;\n> +\tstruct commit_list *parent;\n> +\n> +\toids.num = 0;\n> +\toids.size = 1024;\n> +\tALLOC_ARRAY(oids.list, oids.size);\n> +\tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n> +\tQSORT(oids.list, oids.num, commit_compare);\n> +\n> +\tcount_distinct = 1;\n> +\tfor (i = 1; i < oids.num; i++) {\n> +\t\tif (oidcmp(oids.list[i-1], oids.list[i]))\n> +\t\t\tcount_distinct++;\n> +\t}\n> +\n> +\tcommits.num = 0;\n> +\tcommits.size = count_distinct;\n> +\tALLOC_ARRAY(commits.list, commits.size);\n> +\n> +\tnum_long_edges = 0;\n> +\tfor (i = 0; i < oids.num; i++) {\n> +\t\tint num_parents = 0;\n> +\t\tif (i > 0 && !oidcmp(oids.list[i-1], oids.list[i]))\n> +\t\t\tcontinue;\n> +\n> +\t\tcommits.list[commits.num] = lookup_commit(oids.list[i]);\n> +\t\tparse_commit(commits.list[commits.num]);\n> +\n> +\t\tfor (parent = commits.list[commits.num]->parents;\n> +\t\t parent; parent = parent->next)\n> +\t\t\tnum_parents++;\n> +\n> +\t\tif (num_parents > 2)\n> +\t\t\tnum_long_edges += num_parents - 1;\n> +\n> +\t\tcommits.num++;\n> +\t}\n> +\n> +\tstrbuf_addstr(&tmp_file, pack_dir);\n> +\tstrbuf_addstr(&tmp_file, \"/tmp_graph_XXXXXX\");\n> +\n> +\tfd = git_mkstemp_mode(tmp_file.buf, 0444);\n> +\tif (fd < 0)\n> +\t\tdie_errno(\"unable to create '%s'\", tmp_file.buf);\n> +\n> +\tgraph_name = strbuf_detach(&tmp_file, NULL);\n> +\tf = sha1fd(fd, graph_name);\n> +\n> +\thdr.graph_signature = htonl(GRAPH_SIGNATURE);\n> +\thdr.graph_version = GRAPH_VERSION;\n> +\thdr.hash_version = GRAPH_OID_VERSION;\n> +\thdr.hash_len = GRAPH_OID_LEN;\n> +\thdr.num_chunks = 4;\n> +\n> +\tassert(sizeof(hdr) == 8);\n> +\tsha1write(f, &hdr, sizeof(hdr));\n\nInstead of assembling these into a data structure, could you just use\nindividual calls to sha1write_be32 and sha1write_u8? Same comment\nthroughout this function.\n\n> diff --git a/commit-graph.h b/commit-graph.h\n> new file mode 100644\n> index 0000000000..7b3469a7df\n> --- /dev/null\n> +++ b/commit-graph.h\n> @@ -0,0 +1,20 @@\n> +#ifndef COMMIT_GRAPH_H\n> +#define COMMIT_GRAPH_H\n> +\n> +#include \"git-compat-util.h\"\n> +#include \"commit.h\"\n> +\n> +extern char* get_commit_graph_filename_hash(const char *pack_dir,\n> +\t\t\t\t\t struct object_id *hash);\n> +\n> +struct commit_graph_header {\n> +\tuint32_t graph_signature;\n> +\tunsigned char graph_version;\n> +\tunsigned char hash_version;\n> +\tunsigned char hash_len;\n> +\tunsigned char num_chunks;\n> +};\n\nThis seems like it should be an internal detail of commit-graph.c.\n(Also, as commented above, this struct might not be necessary at all.)\n"},{"id":"338056","messageId":"20180201153349.c94fe3de6b632e2fd8f843cf@google.com","threadId":"47724","inReplyTo":"1517348383-112294-6-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v2 05/14] commit-graph: implement git-commit-graph --write","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-01T23:33:49Z","receivedAt":"2018-02-01T23:33:57Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 30 Jan 2018 16:39:34 -0500\nDerrick Stolee wrote:\n\n> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> index c8ea548dfb..3f3790d9a8 100644\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -5,3 +5,21 @@ NAME\n> ----\n> git-commit-graph - Write and verify Git commit graphs (.graph files)\n> \n> +\n> +SYNOPSIS\n> +--------\n> +[verse]\n> +'git commit-graph' --write [--pack-dir ]\n\nSubcommands (like those in git submodule) generally don't take \"--\", as\nfar as I know.\n\n> +static int graph_write(void)\n> +{\n> +\tstruct object_id *graph_hash = construct_commit_graph(opts.pack_dir);\n\nI should have noticed this when I was reviewing the previous patch, but\nthis is probably better named write_commit_graph.\n\n> +test_expect_success 'create commits and repack' \\\n> + 'for i in $(test_seq 5)\n> + do\n> + echo $i >$i.txt &&\n> + git add $i.txt &&\n> + git commit -m \"commit $i\" &&\n\nYou can generate commits more easily with test_commit. Also, the final\ncharacter of the test_expect_success line should be the apostrophe that\nstarts the big text block, like in other files. (That also means that\nthe backslash is unnecessary.)\n\n> +# Current graph structure:\n> +#\n> +# M3\n> +# / |\\_____\n> +# / 10 15\n> +# / | |\n> +# / 9 M2 14\n> +# | |/ \\ |\n> +# | 8 M1 | 13\n> +# | |/ | \\_|\n> +# 5 7 | 12\n> +# | | \\__|\n> +# 4 6 11\n> +# |____/______/\n> +# 3\n> +# |\n> +# 2\n> +# |\n> +# 1\n\nI don't think we need such a complicated graph structure - maybe it's\nsufficient to have one 2-way merge and one 3-way merge. E.g.\n\n6\n|\\-.\n5 \\ \\\n|\\ \\ \\\n1 2 3 4\n\nAlso, I wonder if it is possible to test the output to a greater extent.\nWe don't want anything that relies on the ordering of commits\n(especially since we plan on transitioning away from using SHA-1 as the\nhash function) but we could still test, for example, that a 3-way merge\nresults in an edge list of the form \"EDGE_..._...\" (where the first _\ndoes not have its high bit set, but the second one does). This could be\ndone by using my hex_unpack() function as seen here [1] with grep (e.g.\n\"45444745[0-7].......[8-9a-f].......\").\n\n[1] https://public-inbox.org/git/b9ea93edabc42754dc3643d6307c22a947eabaf3.1506714999.git.jonathantanmy@google.com/\n"},{"id":"338057","messageId":"20180201234643.7331-1-szeder.dev@gmail.com","threadId":"47724","inReplyTo":"1517348383-112294-5-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v2 04/14] commit-graph: implement construct_commit_graph()","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-02-01T23:46:43Z","receivedAt":"2018-02-01T23:47:34Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"\n> Teach Git to write a commit graph file by checking all packed objects\n> to see if they are commits, then store the file in the given pack\n> directory.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> Makefile | 1 +\n> commit-graph.c | 376 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n> commit-graph.h | 20 +++\n> 3 files changed, 397 insertions(+)\n> create mode 100644 commit-graph.c\n> create mode 100644 commit-graph.h\n\n\n> diff --git a/commit-graph.c b/commit-graph.c\n> new file mode 100644\n> index 0000000000..db2b7390c7\n> --- /dev/null\n> +++ b/commit-graph.c\n\n> +struct packed_commit_list {\n> +\tstruct commit **list;\n> +\tint num;\n> +\tint size;\n> +};\n> +\n> +struct packed_oid_list {\n> +\tstruct object_id **list;\n> +\tint num;\n> +\tint size;\n> +};\n\nWhen we manage the memory allocation of an array with the ALLOC_GROW\nmacro, then we tend to call the helper fields as 'alloc' and 'nr'.\n\n> +static int if_packed_commit_add_to_list(const struct object_id *oid,\n> +\t\t\t\t\tstruct packed_git *pack,\n> +\t\t\t\t\tuint32_t pos,\n> +\t\t\t\t\tvoid *data)\n> +{\n> +\tstruct packed_oid_list *list = (struct packed_oid_list*)data;\n> +\tenum object_type type;\n> +\tunsigned long size;\n> +\tvoid *inner_data;\n> +\toff_t offset = nth_packed_object_offset(pack, pos);\n> +\tinner_data = unpack_entry(pack, offset, &type, &size);\n> +\n> +\tif (inner_data)\n> +\t\tfree(inner_data);\n> +\n> +\tif (type != OBJ_COMMIT)\n> +\t\treturn 0;\n> +\n> +\tALLOC_GROW(list->list, list->num + 1, list->size);\n> +\tlist->list[list->num] = (struct object_id *)malloc(sizeof(struct object_id));\n> +\toidcpy(list->list[list->num], oid);\n> +\t(list->num)++;\n> +\n> +\treturn 0;\n> +}\n> +\n> +struct object_id *construct_commit_graph(const char *pack_dir)\n> +{\n> +\tstruct packed_oid_list oids;\n> +\tstruct packed_commit_list commits;\n> +\tstruct commit_graph_header hdr;\n> +\tstruct sha1file *f;\n> +\tint i, count_distinct = 0;\n> +\tstruct strbuf tmp_file = STRBUF_INIT;\n> +\tunsigned char final_hash[GIT_MAX_RAWSZ];\n> +\tchar *graph_name;\n> +\tint fd;\n> +\tuint32_t chunk_ids[5];\n> +\tuint64_t chunk_offsets[5];\n> +\tint num_long_edges;\n> +\tstruct object_id *f_hash;\n> +\tchar *fname;\n> +\tstruct commit_list *parent;\n> +\n> +\toids.num = 0;\n> +\toids.size = 1024;\n> +\tALLOC_ARRAY(oids.list, oids.size);\n> +\tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n> +\tQSORT(oids.list, oids.num, commit_compare);\n> +\n> +\tcount_distinct = 1;\n> +\tfor (i = 1; i < oids.num; i++) {\n> +\t\tif (oidcmp(oids.list[i-1], oids.list[i]))\n> +\t\t\tcount_distinct++;\n> +\t}\n> +\n> +\tcommits.num = 0;\n> +\tcommits.size = count_distinct;\n> +\tALLOC_ARRAY(commits.list, commits.size);\n> +\n> +\tnum_long_edges = 0;\n> +\tfor (i = 0; i < oids.num; i++) {\n> +\t\tint num_parents = 0;\n> +\t\tif (i > 0 && !oidcmp(oids.list[i-1], oids.list[i]))\n> +\t\t\tcontinue;\n> +\n> +\t\tcommits.list[commits.num] = lookup_commit(oids.list[i]);\n> +\t\tparse_commit(commits.list[commits.num]);\n> +\n> +\t\tfor (parent = commits.list[commits.num]->parents;\n> +\t\t parent; parent = parent->next)\n> +\t\t\tnum_parents++;\n> +\n> +\t\tif (num_parents > 2)\n> +\t\t\tnum_long_edges += num_parents - 1;\n> +\n> +\t\tcommits.num++;\n> +\t}\n> +\n> +\tstrbuf_addstr(&tmp_file, pack_dir);\n> +\tstrbuf_addstr(&tmp_file, \"/tmp_graph_XXXXXX\");\n> +\n> +\tfd = git_mkstemp_mode(tmp_file.buf, 0444);\n> +\tif (fd < 0)\n> +\t\tdie_errno(\"unable to create '%s'\", tmp_file.buf);\n> +\n> +\tgraph_name = strbuf_detach(&tmp_file, NULL);\n> +\tf = sha1fd(fd, graph_name);\n> +\n> +\thdr.graph_signature = htonl(GRAPH_SIGNATURE);\n> +\thdr.graph_version = GRAPH_VERSION;\n> +\thdr.hash_version = GRAPH_OID_VERSION;\n> +\thdr.hash_len = GRAPH_OID_LEN;\n> +\thdr.num_chunks = 4;\n> +\n> +\tassert(sizeof(hdr) == 8);\n> +\tsha1write(f, &hdr, sizeof(hdr));\n> +\n> +\tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n> +\tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n> +\tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n> +\tchunk_ids[3] = GRAPH_CHUNKID_LARGEEDGES;\n> +\tchunk_ids[4] = 0;\n> +\n> +\tchunk_offsets[0] = sizeof(hdr) + GRAPH_CHUNKLOOKUP_SIZE;\n> +\tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n> +\tchunk_offsets[2] = chunk_offsets[1] + GRAPH_OID_LEN * commits.num;\n> +\tchunk_offsets[3] = chunk_offsets[2] + (GRAPH_OID_LEN + 16) * commits.num;\n> +\tchunk_offsets[4] = chunk_offsets[3] + 4 * num_long_edges;\n> +\n> +\tfor (i = 0; i <= hdr.num_chunks; i++) {\n> +\t\tuint32_t chunk_write[3];\n> +\n> +\t\tchunk_write[0] = htonl(chunk_ids[i]);\n> +\t\tchunk_write[1] = htonl(chunk_offsets[i] >> 32);\n> +\t\tchunk_write[2] = htonl(chunk_offsets[i] & 0xffffffff);\n> +\t\tsha1write(f, chunk_write, 12);\n> +\t}\n> +\n> +\twrite_graph_chunk_fanout(f, commits.list, commits.num);\n> +\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.num);\n> +\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.num);\n> +\twrite_graph_chunk_large_edges(f, commits.list, commits.num);\n> +\n> +\tsha1close(f, final_hash, CSUM_CLOSE | CSUM_FSYNC);\n> +\n> +\tf_hash = (struct object_id *)malloc(sizeof(struct object_id));\n> +\tmemcpy(f_hash->hash, final_hash, GIT_MAX_RAWSZ);\n\nhashcpy(), perhaps?\n\n> +\tfname = get_commit_graph_filename_hash(pack_dir, f_hash);\n> +\n> +\tif (rename(graph_name, fname))\n> +\t\tdie(\"failed to rename %s to %s\", graph_name, fname);\n> +\n> +\tfree(oids.list);\n> +\toids.size = 0;\n> +\toids.num = 0;\n> +\n> +\treturn f_hash;\n> +}\n> +\n> diff --git a/commit-graph.h b/commit-graph.h\n> new file mode 100644\n> index 0000000000..7b3469a7df\n> --- /dev/null\n> +++ b/commit-graph.h\n> @@ -0,0 +1,20 @@\n> +#ifndef COMMIT_GRAPH_H\n> +#define COMMIT_GRAPH_H\n> +\n> +#include \"git-compat-util.h\"\n> +#include \"commit.h\"\n> +\n> +extern char* get_commit_graph_filename_hash(const char *pack_dir,\n> +\t\t\t\t\t struct object_id *hash);\n> +\n> +struct commit_graph_header {\n> +\tuint32_t graph_signature;\n> +\tunsigned char graph_version;\n> +\tunsigned char hash_version;\n> +\tunsigned char hash_len;\n> +\tunsigned char num_chunks;\n> +};\n> +\n> +extern struct object_id *construct_commit_graph(const char *pack_dir);\n> +\n> +#endif\n> -- \n> 2.16.0.15.g9c3cf44.dirty\n\n\n"},{"id":"338058","messageId":"20180201234805.7506-1-szeder.dev@gmail.com","threadId":"47724","inReplyTo":"1517348383-112294-6-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v2 05/14] commit-graph: implement git-commit-graph --write","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-02-01T23:48:05Z","receivedAt":"2018-02-01T23:48:20Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"> Teach git-commit-graph to write graph files. Create new test script to verify\n> this command succeeds without failure.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-commit-graph.txt | 18 +++++++\n> builtin/commit-graph.c | 30 ++++++++++++\n> t/t5318-commit-graph.sh | 96 ++++++++++++++++++++++++++++++++++++++\n> 3 files changed, 144 insertions(+)\n> create mode 100755 t/t5318-commit-graph.sh\n> \n> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> index c8ea548dfb..3f3790d9a8 100644\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -5,3 +5,21 @@ NAME\n> ----\n> git-commit-graph - Write and verify Git commit graphs (.graph files)\n> \n> +\n> +SYNOPSIS\n> +--------\n> +[verse]\n> +'git commit-graph' --write [--pack-dir ]\n> +\n\nWhat do these options do and what is the command's output? IOW, an\n'OPTIONS' section would be nice.\n\n> +EXAMPLES\n> +--------\n> +\n> +* Write a commit graph file for the packed commits in your local .git folder.\n> ++\n> +------------------------------------------------\n> +$ git commit-graph --write\n> +------------------------------------------------\n> +\n> +GIT\n> +---\n> +Part of the linkgit:git[1] suite\n\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> new file mode 100755\n> index 0000000000..6bcd1cc264\n> --- /dev/null\n> +++ b/t/t5318-commit-graph.sh\n> @@ -0,0 +1,96 @@\n> +#!/bin/sh\n> +\n> +test_description='commit graph'\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'setup full repo' \\\n> + 'rm -rf .git &&\n> + mkdir full &&\n> + cd full &&\n> + git init &&\n> + git config core.commitgraph true &&\n> + git config pack.threads 1 &&\n\nDoes this pack.threads=1 make a difference?\n\n> + packdir=\".git/objects/pack\"'\n\nWe tend to put single quotes around tests like this:\n\n test_expect_success 'setup full repo' '\n do-this &&\n check-that\n '\n\nThis is not a mere style nit: those newlines before and after the test\nblock make the test's output with '--verbose-log' slightly more\nreadable.\n\nFurthermore, we prefer tabs for indentation.\n\nFinally, 'cd'-ing around such that it affects subsequent tests is\nusually frowned upon. However, in this particular case (going into\none repo, doing a bunch of tests there, then going into another repo,\nand doing another bunch of tests) I think it's better than changing\ndirectory in a subshell in every single test.\n\n> +\n> +test_expect_success 'write graph with no packs' \\\n> + 'git commit-graph --write --pack-dir .'\n> +\n> +test_expect_success 'create commits and repack' \\\n> + 'for i in $(test_seq 5)\n> + do\n> + echo $i >$i.txt &&\n> + git add $i.txt &&\n> + git commit -m \"commit $i\" &&\n> + git branch commits/$i\n> + done &&\n> + git repack'\n> +\n> +test_expect_success 'write graph' \\\n> + 'graph1=$(git commit-graph --write) &&\n> + test_path_is_file ${packdir}/graph-${graph1}.graph'\n\nStyle nit: those {} around the variable names are unnecessary, but I\nsee you use them a lot.\n\n> +\n> +t_expect_success 'Add more commits' \\\n\nThis must be test_expect_success.\n\n> + 'git reset --hard commits/3 &&\n> + for i in $(test_seq 6 10)\n> + do\n> + echo $i >$i.txt &&\n> + git add $i.txt &&\n> + git commit -m \"commit $i\" &&\n> + git branch commits/$i\n> + done &&\n> + git reset --hard commits/3 &&\n> + for i in $(test_seq 11 15)\n> + do\n> + echo $i >$i.txt &&\n> + git add $i.txt &&\n> + git commit -m \"commit $i\" &&\n> + git branch commits/$i\n> + done &&\n> + git reset --hard commits/7 &&\n> + git merge commits/11 &&\n> + git branch merge/1 &&\n> + git reset --hard commits/8 &&\n> + git merge commits/12 &&\n> + git branch merge/2 &&\n> + git reset --hard commits/5 &&\n> + git merge commits/10 commits/15 &&\n> + git branch merge/3 &&\n> + git repack'\n> +\n> +# Current graph structure:\n> +#\n> +# M3\n> +# / |\\_____\n> +# / 10 15\n> +# / | |\n> +# / 9 M2 14\n> +# | |/ \\ |\n> +# | 8 M1 | 13\n> +# | |/ | \\_|\n> +# 5 7 | 12\n> +# | | \\__|\n> +# 4 6 11\n> +# |____/______/\n> +# 3\n> +# |\n> +# 2\n> +# |\n> +# 1\n> +\n> +test_expect_success 'write graph with merges' \\\n> + 'graph2=$(git commit-graph --write) &&\n> + test_path_is_file ${packdir}/graph-${graph2}.graph'\n> +\n> +test_expect_success 'setup bare repo' \\\n> + 'cd .. &&\n> + git clone --bare full bare &&\n> + cd bare &&\n> + git config core.graph true &&\n> + git config pack.threads 1 &&\n> + baredir=\"objects/pack\"'\n> +\n> +test_expect_success 'write graph in bare repo' \\\n> + 'graphbare=$(git commit-graph --write) &&\n> + test_path_is_file ${baredir}/graph-${graphbare}.graph'\n> +\n> +test_done\n> -- \n> 2.16.0.15.g9c3cf44.dirty\n\n\n"},{"id":"338059","messageId":"20180202000225.9648-1-szeder.dev@gmail.com","threadId":"47724","inReplyTo":"1517348383-112294-7-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v2 06/14] commit-graph: implement git-commit-graph --read","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-02-02T00:02:25Z","receivedAt":"2018-02-02T00:02:45Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"\n> Teach git-commit-graph to read commit graph files and summarize their contents.\n> \n> Use the --read option to verify the contents of a commit graph file in the\n> tests.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-commit-graph.txt | 7 ++\n> builtin/commit-graph.c | 55 +++++++++++++++\n> commit-graph.c | 138 ++++++++++++++++++++++++++++++++++++-\n> commit-graph.h | 25 +++++++\n> t/t5318-commit-graph.sh | 28 ++++++--\n> 5 files changed, 247 insertions(+), 6 deletions(-)\n> \n> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> index 3f3790d9a8..09aeaf6c82 100644\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -10,6 +10,7 @@ SYNOPSIS\n> --------\n> [verse]\n> 'git commit-graph' --write [--pack-dir ]\n> +'git commit-graph' --read [--pack-dir ]\n\nAgain, what does this option do?\n\n> EXAMPLES\n> --------\n> @@ -20,6 +21,12 @@ EXAMPLES\n> $ git commit-graph --write\n> ------------------------------------------------\n> \n> +* Read basic information from a graph file.\n> ++\n> +------------------------------------------------\n> +$ git commit-graph --read --graph-hash=\n> +------------------------------------------------\n> +\n> GIT\n> ---\n> Part of the linkgit:git[1] suite\n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index 7affd512f1..218740b1f8 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n\n> +int close_commit_graph(struct commit_graph *g)\n\nstatic, perhaps? I see it's declared as extern in the headeer file\nbelow, but I don't see it called outside of this source file by the\nend of the patch series.\n\n> +{\n> +\tif (g->graph_fd < 0)\n> +\t\treturn 0;\n> +\n> +\tmunmap((void *)g->data, g->data_len);\n> +\tg->data = 0;\n> +\n> +\tclose(g->graph_fd);\n> +\tg->graph_fd = -1;\n> +\n> +\treturn 1;\n> +}\n> +\n> +static void free_commit_graph(struct commit_graph **g)\n> +{\n> +\tif (!g || !*g)\n> +\t\treturn;\n> +\n> +\tclose_commit_graph(*g);\n> +\n> +\tfree(*g);\n> +\t*g = NULL;\n> +}\n> +\n> +struct commit_graph *load_commit_graph_one(const char *graph_file, const char *pack_dir)\n> +{\n> +\tvoid *graph_map;\n> +\tconst unsigned char *data;\n> +\tstruct commit_graph_header *hdr;\n> +\tsize_t graph_size;\n> +\tstruct stat st;\n> +\tuint32_t i;\n> +\tstruct commit_graph *graph;\n> +\tint fd = git_open(graph_file);\n> +\tuint64_t last_chunk_offset;\n> +\tuint32_t last_chunk_id;\n> +\n> +\tif (fd < 0)\n> +\t\treturn 0;\n> +\tif (fstat(fd, &st)) {\n> +\t\tclose(fd);\n> +\t\treturn 0;\n> +\t}\n> +\tgraph_size = xsize_t(st.st_size);\n> +\n> +\tif (graph_size < GRAPH_MIN_SIZE) {\n> +\t\tclose(fd);\n> +\t\tdie(\"graph file %s is too small\", graph_file);\n> +\t}\n> +\tgraph_map = xmmap(NULL, graph_size, PROT_READ, MAP_PRIVATE, fd, 0);\n> +\tdata = (const unsigned char *)graph_map;\n> +\n> +\thdr = graph_map;\n> +\tif (ntohl(hdr->graph_signature) != GRAPH_SIGNATURE) {\n> +\t\tuint32_t signature = ntohl(hdr->graph_signature);\n> +\t\tmunmap(graph_map, graph_size);\n> +\t\tclose(fd);\n> +\t\tdie(\"graph signature %X does not match signature %X\",\n> +\t\t\tsignature, GRAPH_SIGNATURE);\n> +\t}\n> +\tif (hdr->graph_version != GRAPH_VERSION) {\n> +\t\tunsigned char version = hdr->graph_version;\n> +\t\tmunmap(graph_map, graph_size);\n> +\t\tclose(fd);\n> +\t\tdie(\"graph version %X does not match version %X\",\n> +\t\t\tversion, GRAPH_VERSION);\n> +\t}\n> +\n> +\tgraph = alloc_commit_graph(strlen(pack_dir) + 1);\n> +\n> +\tgraph->hdr = hdr;\n> +\tgraph->graph_fd = fd;\n> +\tgraph->data = graph_map;\n> +\tgraph->data_len = graph_size;\n> +\n> +\tlast_chunk_id = 0;\n> +\tlast_chunk_offset = (uint64_t)sizeof(*hdr);\n> +\tfor (i = 0; i < hdr->num_chunks; i++) {\n> +\t\tuint32_t chunk_id = ntohl(*(uint32_t*)(data + sizeof(*hdr) + 12 * i));\n> +\t\tuint64_t chunk_offset1 = ntohl(*(uint32_t*)(data + sizeof(*hdr) + 12 * i + 4));\n> +\t\tuint32_t chunk_offset2 = ntohl(*(uint32_t*)(data + sizeof(*hdr) + 12 * i + 8));\n\nThere are a lot of magic number in these three lines, but at least\nthey are all multiples of 4.\n\n> +\t\tuint64_t chunk_offset = (chunk_offset1 << 32) | chunk_offset2;\n> +\n> +\t\tif (chunk_offset > graph_size - GIT_MAX_RAWSZ)\n> +\t\t\tdie(\"improper chunk offset %08x%08x\", (uint32_t)(chunk_offset >> 32),\n> +\t\t\t (uint32_t)chunk_offset);\n> +\n> +\t\tswitch (chunk_id) {\n> +\t\t\tcase GRAPH_CHUNKID_OIDFANOUT:\n> +\t\t\t\tgraph->chunk_oid_fanout = data + chunk_offset;\n> +\t\t\t\tbreak;\n> +\n> +\t\t\tcase GRAPH_CHUNKID_OIDLOOKUP:\n> +\t\t\t\tgraph->chunk_oid_lookup = data + chunk_offset;\n> +\t\t\t\tbreak;\n> +\n> +\t\t\tcase GRAPH_CHUNKID_DATA:\n> +\t\t\t\tgraph->chunk_commit_data = data + chunk_offset;\n> +\t\t\t\tbreak;\n> +\n> +\t\t\tcase GRAPH_CHUNKID_LARGEEDGES:\n> +\t\t\t\tgraph->chunk_large_edges = data + chunk_offset;\n> +\t\t\t\tbreak;\n> +\n> +\t\t\tcase 0:\n> +\t\t\t\tbreak;\n> +\n> +\t\t\tdefault:\n> +\t\t\t\tfree_commit_graph(&graph);\n> +\t\t\t\tdie(\"unrecognized graph chunk id: %08x\", chunk_id);\n> +\t\t}\n> +\n> +\t\tif (last_chunk_id == GRAPH_CHUNKID_OIDLOOKUP)\n> +\t\t{\n> +\t\t\tgraph->num_commits = (chunk_offset - last_chunk_offset)\n> +\t\t\t\t\t / hdr->hash_len;\n> +\t\t}\n> +\n> +\t\tlast_chunk_id = chunk_id;\n> +\t\tlast_chunk_offset = chunk_offset;\n> +\t}\n> +\n> +\tstrcpy(graph->pack_dir, pack_dir);\n> +\treturn graph;\n> +}\n> +\n> static void write_graph_chunk_fanout(struct sha1file *f,\n> \t\t\t\t struct commit **commits,\n> \t\t\t\t int nr_commits)\n> @@ -361,7 +497,7 @@ struct object_id *construct_commit_graph(const char *pack_dir)\n> \tsha1close(f, final_hash, CSUM_CLOSE | CSUM_FSYNC);\n> \n> \tf_hash = (struct object_id *)malloc(sizeof(struct object_id));\n> -\tmemcpy(f_hash->hash, final_hash, GIT_MAX_RAWSZ);\n> +\thashcpy(f_hash->hash, final_hash);\n\nOh, look, I told you it's hashcpy()! ;)\n\n> \tfname = get_commit_graph_filename_hash(pack_dir, f_hash);\n> \n> \tif (rename(graph_name, fname))\n> diff --git a/commit-graph.h b/commit-graph.h\n> index 7b3469a7df..e046ae575c 100644\n> --- a/commit-graph.h\n> +++ b/commit-graph.h\n> @@ -15,6 +15,31 @@ struct commit_graph_header {\n> \tunsigned char num_chunks;\n> };\n> \n> +extern struct commit_graph {\n> +\tint graph_fd;\n> +\n> +\tconst unsigned char *data;\n> +\tsize_t data_len;\n> +\n> +\tconst struct commit_graph_header *hdr;\n> +\n> +\tstruct object_id oid;\n> +\n> +\tuint32_t num_commits;\n> +\n> +\tconst unsigned char *chunk_oid_fanout;\n> +\tconst unsigned char *chunk_oid_lookup;\n> +\tconst unsigned char *chunk_commit_data;\n> +\tconst unsigned char *chunk_large_edges;\n> +\n> +\t/* something like \".git/objects/pack\" */\n> +\tchar pack_dir[FLEX_ARRAY]; /* more */\n> +} *commit_graph;\n> +\n> +extern int close_commit_graph(struct commit_graph *g);\n> +\n> +extern struct commit_graph *load_commit_graph_one(const char *graph_file, const char *pack_dir);\n> +\n> extern struct object_id *construct_commit_graph(const char *pack_dir);\n> \n> #endif\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 6bcd1cc264..da565624e3 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -25,11 +25,23 @@ test_expect_success 'create commits and repack' \\\n> done &&\n> git repack'\n> \n> +_graph_read_expect() {\n> + cat >expect <<- EOF\n> +header: 43475048 01 01 14 04\n> +num_commits: $1\n> +chunks: oid_fanout oid_lookup commit_metadata large_edges\n> +pack_dir: $2\n> +EOF\n> +}\n\nStyle nit: since you are already using the '<<-' operator for the\nhere-doc, you could indent it with tabs.\n\n> +\n> test_expect_success 'write graph' \\\n> 'graph1=$(git commit-graph --write) &&\n> - test_path_is_file ${packdir}/graph-${graph1}.graph'\n> + test_path_is_file ${packdir}/graph-${graph1}.graph &&\n> + git commit-graph --read --graph-hash=${graph1} >output &&\n> + _graph_read_expect \"5\" \"${packdir}\" &&\n> + cmp expect output'\n\nPlease use the 'test_cmp' helper throughout the tests instead.\nIf the two files don't match, 'cmp' will only tell you where they\nstart to differ, whereas 'test_cmp' will actually show the difference.\n\n> -t_expect_success 'Add more commits' \\\n> +test_expect_success 'Add more commits' \\\n\nThis should be squashed into the earlier commit.\n\n> 'git reset --hard commits/3 &&\n> for i in $(test_seq 6 10)\n> do\n> @@ -79,7 +91,10 @@ t_expect_success 'Add more commits' \\\n> \n> test_expect_success 'write graph with merges' \\\n> 'graph2=$(git commit-graph --write) &&\n> - test_path_is_file ${packdir}/graph-${graph2}.graph'\n> + test_path_is_file ${packdir}/graph-${graph2}.graph &&\n> + git commit-graph --read --graph-hash=${graph2} >output &&\n> + _graph_read_expect \"18\" \"${packdir}\" &&\n> + cmp expect output'\n> \n> test_expect_success 'setup bare repo' \\\n> 'cd .. &&\n> @@ -87,10 +102,13 @@ test_expect_success 'setup bare repo' \\\n> cd bare &&\n> git config core.graph true &&\n> git config pack.threads 1 &&\n> - baredir=\"objects/pack\"'\n> + baredir=\"./objects/pack\"'\n\nIs this change really necessary? If it is, then perhaps it should\nhave been written this way upon its introduction.\n\n> test_expect_success 'write graph in bare repo' \\\n> 'graphbare=$(git commit-graph --write) &&\n> - test_path_is_file ${baredir}/graph-${graphbare}.graph'\n> + test_path_is_file ${baredir}/graph-${graphbare}.graph &&\n> + git commit-graph --read --graph-hash=${graphbare} >output &&\n> + _graph_read_expect \"18\" \"${baredir}\" &&\n> + cmp expect output'\n> \n> test_done\n> -- \n> 2.16.0.15.g9c3cf44.dirty\n\n\n"},{"id":"338061","messageId":"20180201162324.fa8b38531955ee5378e5c5ff@google.com","threadId":"47724","inReplyTo":"1517348383-112294-7-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v2 06/14] commit-graph: implement git-commit-graph --read","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-02T00:23:24Z","receivedAt":"2018-02-02T00:23:42Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 30 Jan 2018 16:39:35 -0500\nDerrick Stolee wrote:\n\n> Teach git-commit-graph to read commit graph files and summarize their contents.\n\nOne overall question - is the \"read\" command meant to be a command used\nby the end user, or is it here just to test that some aspects of reading\nworks? If the former, I'm not sure how useful it is. And if the latter,\nI think that it is more useful to just implementing something that reads\nit, then make the 11/14 change (modifying parse_commit_gently) and\ninclude a perf test to show that your commit graph reading is both\ncorrect and (performance-)effective.\n"},{"id":"338063","messageId":"20180202005335.23410-1-szeder.dev@gmail.com","threadId":"47724","inReplyTo":"1517348383-112294-4-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v2 03/14] commit-graph: create git-commit-graph builtin","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-02-02T00:53:35Z","receivedAt":"2018-02-02T00:53:46Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> new file mode 100644\n> index 0000000000..c8ea548dfb\n> --- /dev/null\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -0,0 +1,7 @@\n> +git-commit-graph(1)\n> +============\n\nHere the length of the '====' must match the length of the title line\nabove, or AsciiDoc will complain about missing document title.\n\n"},{"id":"338065","messageId":"20180202013552.24931-1-szeder.dev@gmail.com","threadId":"47724","inReplyTo":"1517348383-112294-8-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v2 07/14] commit-graph: implement git-commit-graph --update-head","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-02-02T01:35:52Z","receivedAt":"2018-02-02T01:36:11Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"> It is possible to have multiple commit graph files in a pack directory,\n> but only one is important at a time. Use a 'graph_head' file to point\n> to the important file.\n\nThis implies that all those other files are ignored, right?\n\n> Teach git-commit-graph to write 'graph_head' upon\n> writing a new commit graph file.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-commit-graph.txt | 34 ++++++++++++++++++++++++++++++++++\n> builtin/commit-graph.c | 38 +++++++++++++++++++++++++++++++++++---\n> commit-graph.c | 25 +++++++++++++++++++++++++\n> commit-graph.h | 2 ++\n> t/t5318-commit-graph.sh | 12 ++++++++++--\n> 5 files changed, 106 insertions(+), 5 deletions(-)\n> \n> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> index 09aeaf6c82..99ced16ddc 100644\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -12,15 +12,49 @@ SYNOPSIS\n> 'git commit-graph' --write [--pack-dir ]\n> 'git commit-graph' --read [--pack-dir ]\n> \n> +OPTIONS\n> +-------\n\nOh, look, the 'OPTIONS' section I missed in the previous patches! ;)\n\nThis should be split up and squashed into the previous patches where\nthe individual --options are first mentioned.\n\n> +--pack-dir::\n> +\tUse given directory for the location of packfiles, graph-head,\n> +\tand graph files.\n> +\n> +--read::\n> +\tRead a graph file given by the graph-head file and output basic\n> +\tdetails about the graph file. (Cannot be combined with --write.)\n\nFrom the output of 'git commit-graph --read' it seems that it's not a\ngenerally useful option to the user. Perhaps it should be mentioned\nthat it's only intended as a debugging aid? Or maybe it doesn't\nreally matter, because eventually this command will become irrelevant,\nas other commands (clone, fetch, gc) will invoke it automagically...\n\n> +--graph-id::\n> +\tWhen used with --read, consider the graph file graph-.graph.\n> +\n> +--write::\n> +\tWrite a new graph file to the pack directory. (Cannot be combined\n> +\twith --read.)\n\nI think this should also mention that it prints the generated graph\nfile's checksum.\n\n> +\n> +--update-head::\n> +\tWhen used with --write, update the graph-head file to point to\n> +\tthe written graph file.\n\nSo it should be used with '--write', noted.\n\n> EXAMPLES\n> --------\n> \n> +* Output the hash of the graph file pointed to by /graph-head.\n> ++\n> +------------------------------------------------\n> +$ git commit-graph --pack-dir=\n> +------------------------------------------------\n> +\n> * Write a commit graph file for the packed commits in your local .git folder.\n> +\n> ------------------------------------------------\n> $ git commit-graph --write\n> ------------------------------------------------\n> \n> +* Write a graph file for the packed commits in your local .git folder,\n> +* and update graph-head.\n> ++\n> +------------------------------------------------\n> +$ git commit-graph --write --update-head\n> +------------------------------------------------\n> +\n> * Read basic information from a graph file.\n> +\n> ------------------------------------------------\n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index 218740b1f8..d73cbc907d 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -11,7 +11,7 @@\n> static char const * const builtin_commit_graph_usage[] = {\n> \tN_(\"git commit-graph [--pack-dir ]\"),\n> \tN_(\"git commit-graph --read [--graph-hash=]\"),\n> -\tN_(\"git commit-graph --write [--pack-dir ]\"),\n> +\tN_(\"git commit-graph --write [--pack-dir ] [--update-head]\"),\n> \tNULL\n> };\n> \n> @@ -20,6 +20,9 @@ static struct opts_commit_graph {\n> \tint read;\n> \tconst char *graph_hash;\n> \tint write;\n> +\tint update_head;\n> +\tint has_existing;\n> +\tstruct object_id old_graph_hash;\n> } opts;\n> \n> static int graph_read(void)\n> @@ -30,8 +33,8 @@ static int graph_read(void)\n> \n> \tif (opts.graph_hash && strlen(opts.graph_hash) == GIT_MAX_HEXSZ)\n> \t\tget_oid_hex(opts.graph_hash, &graph_hash);\n> -\telse\n> -\t\tdie(\"no graph hash specified\");\n> +\telse if (!get_graph_head_hash(opts.pack_dir, &graph_hash))\n> +\t\tdie(\"no graph-head exists\");\n> \n> \tgraph_file = get_commit_graph_filename_hash(opts.pack_dir, &graph_hash);\n> \tgraph = load_commit_graph_one(graph_file, opts.pack_dir);\n> @@ -62,10 +65,33 @@ static int graph_read(void)\n> \treturn 0;\n> }\n> \n> +static void update_head_file(const char *pack_dir, const struct object_id *graph_hash)\n> +{\n> +\tstruct strbuf head_path = STRBUF_INIT;\n> +\tint fd;\n> +\tstruct lock_file lk = LOCK_INIT;\n> +\n> +\tstrbuf_addstr(&head_path, pack_dir);\n> +\tstrbuf_addstr(&head_path, \"/\");\n> +\tstrbuf_addstr(&head_path, \"graph-head\");\n\nstrbuf_addstr(&head_path, \"/graph-head\"); ?\n\n> +\n> +\tfd = hold_lock_file_for_update(&lk, head_path.buf, LOCK_DIE_ON_ERROR);\n> +\tstrbuf_release(&head_path);\n> +\n> +\tif (fd < 0)\n> +\t\tdie_errno(\"unable to open graph-head\");\n> +\n> +\twrite_in_full(fd, oid_to_hex(graph_hash), GIT_MAX_HEXSZ);\n> +\tcommit_lock_file(&lk);\n\nThe new graph-head file will be writable. All other files in\n.git/objects/pack are created read-only, including graph files. Just\npointing it out, but I don't think it's a bit deal; other than\nconsistency with the permissions of other files I don't have any\nargument for making it read-only.\n\n> +}\n> +\n> static int graph_write(void)\n> {\n> \tstruct object_id *graph_hash = construct_commit_graph(opts.pack_dir);\n\nFirst the new graph file is written ...\n\n> \n> +\tif (opts.update_head)\n> +\t\tupdate_head_file(opts.pack_dir, graph_hash);\n\n... and then the new graph head, good. There could be a race if it\nwere the other way around.\n\n> +\n> \tif (graph_hash)\n> \t\tprintf(\"%s\\n\", oid_to_hex(graph_hash));\n> \n> @@ -83,6 +109,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n> \t\t\tN_(\"read graph file\")),\n> \t\tOPT_BOOL('w', \"write\", &opts.write,\n> \t\t\tN_(\"write commit graph file\")),\n> +\t\tOPT_BOOL('u', \"update-head\", &opts.update_head,\n> +\t\t\tN_(\"update graph-head to written graph file\")),\n> \t\t{ OPTION_STRING, 'H', \"graph-hash\", &opts.graph_hash,\n> \t\t\tN_(\"hash\"),\n> \t\t\tN_(\"A hash for a specific graph file in the pack-dir.\"),\n> @@ -109,10 +137,14 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n> \t\topts.pack_dir = strbuf_detach(&path, NULL);\n> \t}\n> \n> +\topts.has_existing = !!get_graph_head_hash(opts.pack_dir, &opts.old_graph_hash);\n> +\n> \tif (opts.read)\n> \t\treturn graph_read();\n> \tif (opts.write)\n> \t\treturn graph_write();\n> \n> +\tif (opts.has_existing)\n> +\t\tprintf(\"%s\\n\", oid_to_hex(&opts.old_graph_hash));\n\nIt seems that a command like 'git commit-graph --read --update-head'\nsucceeds and '--update-head' has no effect. I think it should error\nout. 'git commit-graph --update-head' doesn't complain, either.\n\nWould it be more appropriate to have 'read' and 'write' subcommands\ninstead of '--read' and '--write' options? Then parse-options alone\nwould take care of a command line like 'git commit-graph read\n--update-index' and error out because of unrecognized option.\n\n> \treturn 0;\n> }\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 622a650259..764e016ddb 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -35,6 +35,31 @@\n> #define GRAPH_MIN_SIZE (GRAPH_CHUNKLOOKUP_SIZE + GRAPH_FANOUT_SIZE + \\\n> \t\t\tGRAPH_OID_LEN + sizeof(struct commit_graph_header))\n> \n> +struct object_id *get_graph_head_hash(const char *pack_dir, struct object_id *hash)\n> +{\n> +\tstruct strbuf head_filename = STRBUF_INIT;\n> +\tchar hex[GIT_MAX_HEXSZ + 1];\n> +\tFILE *f;\n> +\n> +\tstrbuf_addstr(&head_filename, pack_dir);\n> +\tstrbuf_addstr(&head_filename, \"/graph-head\");\n> +\n> +\tf = fopen(head_filename.buf, \"r\");\n> +\tstrbuf_release(&head_filename);\n> +\n> +\tif (!f)\n> +\t\treturn 0;\n> +\n> +\tif (!fgets(hex, sizeof(hex), f))\n> +\t\tdie(\"failed to read graph-head\");\n> +\n> +\tfclose(f);\n> +\n> +\tif (get_oid_hex(hex, hash))\n> +\t\treturn 0;\n> +\treturn hash;\n> +}\n> +\n> char* get_commit_graph_filename_hash(const char *pack_dir,\n> \t\t\t\t struct object_id *hash)\n> {\n> diff --git a/commit-graph.h b/commit-graph.h\n> index e046ae575c..43eb0aec84 100644\n> --- a/commit-graph.h\n> +++ b/commit-graph.h\n> @@ -4,6 +4,8 @@\n> #include \"git-compat-util.h\"\n> #include \"commit.h\"\n> \n> +extern struct object_id *get_graph_head_hash(const char *pack_dir,\n> +\t\t\t\t\t struct object_id *hash);\n> extern char* get_commit_graph_filename_hash(const char *pack_dir,\n> \t\t\t\t\t struct object_id *hash);\n> \n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index da565624e3..d1a23bcdaf 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -13,7 +13,8 @@ test_expect_success 'setup full repo' \\\n> packdir=\".git/objects/pack\"'\n> \n> test_expect_success 'write graph with no packs' \\\n> - 'git commit-graph --write --pack-dir .'\n> + 'git commit-graph --write --pack-dir . &&\n> + test_path_is_missing graph-head'\n> \n> test_expect_success 'create commits and repack' \\\n> 'for i in $(test_seq 5)\n> @@ -37,6 +38,7 @@ EOF\n> test_expect_success 'write graph' \\\n> 'graph1=$(git commit-graph --write) &&\n> test_path_is_file ${packdir}/graph-${graph1}.graph &&\n> + test_path_is_missing ${packdir}/graph-head &&\n> git commit-graph --read --graph-hash=${graph1} >output &&\n> _graph_read_expect \"5\" \"${packdir}\" &&\n> cmp expect output'\n> @@ -90,8 +92,11 @@ test_expect_success 'Add more commits' \\\n> # 1\n> \n> test_expect_success 'write graph with merges' \\\n> - 'graph2=$(git commit-graph --write) &&\n> + 'graph2=$(git commit-graph --write --update-head) &&\n> test_path_is_file ${packdir}/graph-${graph2}.graph &&\n> + test_path_is_file ${packdir}/graph-head &&\n> + echo ${graph2} >expect &&\n> + cmp -n 40 expect ${packdir}/graph-head &&\n\nThis check is fishy, and that '-n 40' will need adjustment once we\nmigrate to a longer hash function. I presume you used it, because\n'graph-head' contains only 40 hexdigits without a trailing newline,\nbut 'expect' created with 'echo' does contain a newline as well,\nright? Then this would be better instead:\n\n printf $graph2 >expect &&\n test_cmp expect $packdir/graph-head &&\n\n> git commit-graph --read --graph-hash=${graph2} >output &&\n> _graph_read_expect \"18\" \"${packdir}\" &&\n> cmp expect output'\n> @@ -107,6 +112,9 @@ test_expect_success 'setup bare repo' \\\n> test_expect_success 'write graph in bare repo' \\\n> 'graphbare=$(git commit-graph --write) &&\n> test_path_is_file ${baredir}/graph-${graphbare}.graph &&\n> + test_path_is_file ${baredir}/graph-head &&\n> + echo ${graphbare} >expect &&\n> + cmp -n 40 expect ${baredir}/graph-head &&\n\nLikewise.\n\n> git commit-graph --read --graph-hash=${graphbare} >output &&\n> _graph_read_expect \"18\" \"${baredir}\" &&\n> cmp expect output'\n> -- \n> 2.16.0.15.g9c3cf44.dirty\n\n\n"},{"id":"338066","messageId":"20180202014728.30907-1-szeder.dev@gmail.com","threadId":"47724","inReplyTo":"1517348383-112294-6-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v2 05/14] commit-graph: implement git-commit-graph --write","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-02-02T01:47:28Z","receivedAt":"2018-02-02T01:47:46Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> new file mode 100755\n> index 0000000000..6bcd1cc264\n> --- /dev/null\n> +++ b/t/t5318-commit-graph.sh\n> @@ -0,0 +1,96 @@\n> +#!/bin/sh\n> +\n> +test_description='commit graph'\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'setup full repo' \\\n> + 'rm -rf .git &&\n> + mkdir full &&\n> + cd full &&\n> + git init &&\n> + git config core.commitgraph true &&\n\nThis config variable is unknown at this point.\nI think the test shouldn't set it before it's introduced in patch 10.\n\n> + git config pack.threads 1 &&\n> + packdir=\".git/objects/pack\"'\n\n\n> +test_expect_success 'setup bare repo' \\\n> + 'cd .. &&\n> + git clone --bare full bare &&\n> + cd bare &&\n> + git config core.graph true &&\n\nLikewise, and its name should be updated as well.\n\n> + git config pack.threads 1 &&\n> + baredir=\"objects/pack\"'\n"},{"id":"338067","messageId":"20180201175151.89555950582a2e6c5279849a@google.com","threadId":"47724","inReplyTo":"1517348383-112294-12-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v2 11/14] commit: integrate commit graph with commit parsing","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-02T01:51:51Z","receivedAt":"2018-02-02T01:51:58Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 30 Jan 2018 16:39:40 -0500\nDerrick Stolee wrote:\n\n> +/* global storage */\n> +struct commit_graph *commit_graph = 0;\n\nNULL, not 0.\n\n> +static int bsearch_graph(struct commit_graph *g, struct object_id *oid, uint32_t *pos)\n> +{\n> +\tuint32_t last, first = 0;\n> +\n> +\tif (oid->hash[0])\n> +\t\tfirst = ntohl(*(uint32_t*)(g->chunk_oid_fanout + 4 * (oid->hash[0] - 1)));\n> +\tlast = ntohl(*(uint32_t*)(g->chunk_oid_fanout + 4 * oid->hash[0]));\n> +\n> +\twhile (first < last) {\n> +\t\tuint32_t mid = first + (last - first) / 2;\n> +\t\tconst unsigned char *current;\n> +\t\tint cmp;\n> +\n> +\t\tcurrent = g->chunk_oid_lookup + g->hdr->hash_len * mid;\n> +\t\tcmp = hashcmp(oid->hash, current);\n> +\t\tif (!cmp) {\n> +\t\t\t*pos = mid;\n> +\t\t\treturn 1;\n> +\t\t}\n> +\t\tif (cmp > 0) {\n> +\t\t\tfirst = mid + 1;\n> +\t\t\tcontinue;\n> +\t\t}\n> +\t\tlast = mid;\n> +\t}\n> +\n> +\t*pos = first;\n> +\treturn 0;\n> +}\n\nThis would be better in sha1-lookup.c, something like the reverse of commit\nf1068efefe6d (\"sha1_file: drop experimental GIT_USE_LOOKUP search\",\n2017-08-09), except that it can be done using a simple binary search.\n\n> +static int full_parse_commit(struct commit *item, struct commit_graph *g,\n> +\t\t\t uint32_t pos, const unsigned char *commit_data)\n> +{\n> +\tstruct object_id oid;\n> +\tstruct commit *new_parent;\n> +\tuint32_t new_parent_pos;\n> +\tuint32_t *parent_data_ptr;\n> +\tuint64_t date_low, date_high;\n> +\tstruct commit_list **pptr;\n> +\n> +\titem->object.parsed = 1;\n> +\titem->graph_pos = pos;\n> +\n> +\thashcpy(oid.hash, commit_data);\n> +\titem->tree = lookup_tree(&oid);\n> +\n> +\tdate_high = ntohl(*(uint32_t*)(commit_data + g->hdr->hash_len + 8)) & 0x3;\n> +\tdate_low = ntohl(*(uint32_t*)(commit_data + g->hdr->hash_len + 12));\n> +\titem->date = (timestamp_t)((date_high << 32) | date_low);\n> +\n> +\tpptr = &item->parents;\n> +\n> +\tnew_parent_pos = ntohl(*(uint32_t*)(commit_data + g->hdr->hash_len));\n> +\tif (new_parent_pos == GRAPH_PARENT_NONE)\n> +\t\treturn 1;\n> +\tget_nth_commit_oid(g, new_parent_pos, &oid);\n> +\tnew_parent = lookup_commit(&oid);\n> +\tif (new_parent) {\n> +\t\tnew_parent->graph_pos = new_parent_pos;\n> +\t\tpptr = &commit_list_insert(new_parent, pptr)->next;\n> +\t} else {\n> +\t\tdie(\"could not find commit %s\", oid_to_hex(&oid));\n> +\t}\n> +\n> +\tnew_parent_pos = ntohl(*(uint32_t*)(commit_data + g->hdr->hash_len + 4));\n> +\tif (new_parent_pos == GRAPH_PARENT_NONE)\n> +\t\treturn 1;\n> +\tif (!(new_parent_pos & GRAPH_LARGE_EDGES_NEEDED)) {\n> +\t\tget_nth_commit_oid(g, new_parent_pos, &oid);\n> +\t\tnew_parent = lookup_commit(&oid);\n> +\t\tif (new_parent) {\n> +\t\t\tnew_parent->graph_pos = new_parent_pos;\n> +\t\t\tpptr = &commit_list_insert(new_parent, pptr)->next;\n> +\t\t} else\n> +\t\t\tdie(\"could not find commit %s\", oid_to_hex(&oid));\n> +\t\treturn 1;\n> +\t}\n> +\n> +\tparent_data_ptr = (uint32_t*)(g->chunk_large_edges + 4 * (new_parent_pos ^ GRAPH_LARGE_EDGES_NEEDED));\n> +\tdo {\n> +\t\tnew_parent_pos = ntohl(*parent_data_ptr);\n> +\n> +\t\tget_nth_commit_oid(g, new_parent_pos & GRAPH_EDGE_LAST_MASK, &oid);\n> +\t\tnew_parent = lookup_commit(&oid);\n> +\t\tif (new_parent) {\n> +\t\t\tnew_parent->graph_pos = new_parent_pos & GRAPH_EDGE_LAST_MASK;\n> +\t\t\tpptr = &commit_list_insert(new_parent, pptr)->next;\n> +\t\t} else\n> +\t\t\tdie(\"could not find commit %s\", oid_to_hex(&oid));\n> +\t\tparent_data_ptr++;\n> +\t} while (!(new_parent_pos & GRAPH_LAST_EDGE));\n> +\n> +\treturn 1;\n> +}\n\nThe part that converts into \nseems to be duplicated 3 times. Refactor into a function?\n\n> +/**\n> + * Fill 'item' to contain all information that would be parsed by parse_commit_buffer.\n> + */\n> +static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t pos)\n> +{\n> +\tuint32_t new_parent_pos;\n> +\tuint32_t *parent_data_ptr;\n> +\tconst unsigned char *commit_data = g->chunk_commit_data + (g->hdr->hash_len + 16) * pos;\n> +\n> +\tnew_parent_pos = ntohl(*(uint32_t*)(commit_data + g->hdr->hash_len));\n> +\n> +\tif (new_parent_pos == GRAPH_PARENT_MISSING)\n> +\t\treturn 0;\n> +\n> +\tif (new_parent_pos == GRAPH_PARENT_NONE)\n> +\t\treturn full_parse_commit(item, g, pos, commit_data);\n> +\n> +\tnew_parent_pos = ntohl(*(uint32_t*)(commit_data + g->hdr->hash_len + 4));\n> +\n> +\tif (new_parent_pos == GRAPH_PARENT_MISSING)\n> +\t\treturn 0;\n> +\tif (!(new_parent_pos & GRAPH_LARGE_EDGES_NEEDED))\n> +\t\treturn full_parse_commit(item, g, pos, commit_data);\n> +\n> +\tnew_parent_pos = new_parent_pos ^ GRAPH_LARGE_EDGES_NEEDED;\n> +\n> +\tif (new_parent_pos == GRAPH_PARENT_MISSING)\n> +\t\treturn 0;\n> +\n> +\tparent_data_ptr = (uint32_t*)(g->chunk_large_edges + 4 * new_parent_pos);\n> +\tdo {\n> +\t\tnew_parent_pos = ntohl(*parent_data_ptr);\n> +\n> +\t\tif ((new_parent_pos & GRAPH_EDGE_LAST_MASK) == GRAPH_PARENT_MISSING)\n> +\t\t\treturn 0;\n> +\n> +\t\tparent_data_ptr++;\n> +\t} while (!(new_parent_pos & GRAPH_LAST_EDGE));\n> +\n> +\treturn full_parse_commit(item, g, pos, commit_data);\n> +}\n\nThis function seems to just check for GRAPH_PARENT_MISSING - could that\ncheck be folded into full_parse_commit() instead? (Then\nfull_parse_commit can be renamed to fill_commit_in_graph.)\n\n> @@ -439,9 +656,24 @@ struct object_id *construct_commit_graph(const char *pack_dir)\n> \tchar *fname;\n> \tstruct commit_list *parent;\n> \n> +\tprepare_commit_graph();\n> +\n> \toids.num = 0;\n> \toids.size = 1024;\n> +\n> +\tif (commit_graph && oids.size < commit_graph->num_commits)\n> +\t\toids.size = commit_graph->num_commits;\n> +\n> \tALLOC_ARRAY(oids.list, oids.size);\n> +\n> +\tif (commit_graph) {\n> +\t\tfor (i = 0; i < commit_graph->num_commits; i++) {\n> +\t\t\toids.list[i] = malloc(sizeof(struct object_id));\n> +\t\t\tget_nth_commit_oid(commit_graph, i, oids.list[i]);\n> +\t\t}\n> +\t\toids.num = commit_graph->num_commits;\n> +\t}\n> +\n> \tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n> \tQSORT(oids.list, oids.num, commit_compare);\n> \n> @@ -525,6 +757,11 @@ struct object_id *construct_commit_graph(const char *pack_dir)\n> \thashcpy(f_hash->hash, final_hash);\n> \tfname = get_commit_graph_filename_hash(pack_dir, f_hash);\n> \n> +\tif (commit_graph) {\n> +\t\tclose_commit_graph(commit_graph);\n> +\t\tFREE_AND_NULL(commit_graph);\n> +\t}\n> +\n> \tif (rename(graph_name, fname))\n> \t\tdie(\"failed to rename %s to %s\", graph_name, fname);\n\nWhat is the relation of these changes to construct_commit_graph() to the\nrest of the patch?\n\n> diff --git a/commit-graph.h b/commit-graph.h\n> index 43eb0aec84..05ddbbe165 100644\n> --- a/commit-graph.h\n> +++ b/commit-graph.h\n> @@ -4,6 +4,18 @@\n> #include \"git-compat-util.h\"\n> #include \"commit.h\"\n> \n> +/**\n> + * Given a commit struct, try to fill the commit struct info, including:\n> + * 1. tree object\n> + * 2. date\n> + * 3. parents.\n> + *\n> + * Returns 1 if and only if the commit was found in the packed graph.\n> + *\n> + * See parse_commit_buffer() for the fallback after this call.\n> + */\n> +extern int parse_commit_in_graph(struct commit *item);\n> +\n> extern struct object_id *get_graph_head_hash(const char *pack_dir,\n> \t\t\t\t\t struct object_id *hash);\n> extern char* get_commit_graph_filename_hash(const char *pack_dir,\n> @@ -40,7 +52,13 @@ extern struct commit_graph {\n> \n> extern int close_commit_graph(struct commit_graph *g);\n> \n> -extern struct commit_graph *load_commit_graph_one(const char *graph_file, const char *pack_dir);\n> +extern struct commit_graph *load_commit_graph_one(const char *graph_file,\n> +\t\t\t\t\t\t const char *pack_dir);\n> +extern void prepare_commit_graph(void);\n> +\n> +extern struct object_id *get_nth_commit_oid(struct commit_graph *g,\n> +\t\t\t\t\t uint32_t n,\n> +\t\t\t\t\t struct object_id *oid);\n> \n> extern struct object_id *construct_commit_graph(const char *pack_dir);\n\nThis header file now contains functions for reading the commit graph,\nand functions for writing one. It seems to me that those are (and should\nbe) quite disjoint, so it might be better to separate them into two.\n\n> -int parse_commit_gently(struct commit *item, int quiet_on_missing)\n> +int parse_commit_internal(struct commit *item, int quiet_on_missing, int check_packed)\n> {\n> \tenum object_type type;\n> \tvoid *buffer;\n> @@ -385,6 +386,8 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n> \t\treturn -1;\n> \tif (item->object.parsed)\n> \t\treturn 0;\n> +\tif (check_packed && parse_commit_in_graph(item))\n> +\t\treturn 0;\n> \tbuffer = read_sha1_file(item->object.oid.hash, &type, &size);\n> \tif (!buffer)\n> \t\treturn quiet_on_missing ? -1 :\n> @@ -404,6 +407,11 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n> \treturn ret;\n> }\n> \n> +int parse_commit_gently(struct commit *item, int quiet_on_missing)\n> +{\n> +\treturn parse_commit_internal(item, quiet_on_missing, 1);\n> +}\n\nAre you planning to use parse_commit_internal() from elsewhere? (It\ndoesn't seem to be the case, at least from this patch series.)\n\n> diff --git a/log-tree.c b/log-tree.c\n> index fca29d4799..156aed4541 100644\n> --- a/log-tree.c\n> +++ b/log-tree.c\n> @@ -659,8 +659,7 @@ void show_log(struct rev_info *opt)\n> \t\tshow_mergetag(opt, commit);\n> \t}\n> \n> -\tif (!get_cached_commit_buffer(commit, NULL))\n> -\t\treturn;\n> +\tget_commit_buffer(commit, NULL);\n\nThis undoes an optimization that I discuss in my e-mail message here\n[1]. If we decide to do this, it should at least be called out in the\ncommit message.\n\n[1] https://public-inbox.org/git/b88725476d9f13ba4381d85e5fe049f6ef93f621.1506714999.git.jonathantanmy@google.com/\n\n> +_graph_git_two_modes() {\n\nNo need for the name to start with an underscore, I think.\n\n> + git -c core.commitgraph=true $1 >output\n> + git -c core.commitgraph=false $1 >expect\n> + cmp output expect\n\nUse test_cmp.\n\n> +}\n> +\n> +_graph_git_behavior() {\n> + BRANCH=$1\n> + COMPARE=$2\n> + test_expect_success 'check normal git operations' \\\n> + '_graph_git_two_modes \"log --oneline ${BRANCH}\" &&\n> + _graph_git_two_modes \"log --topo-order ${BRANCH}\" &&\n> + _graph_git_two_modes \"branch -vv\" &&\n> + _graph_git_two_modes \"merge-base -a ${BRANCH} ${COMPARE}\"'\n> +}\n\nThis makes it difficult to debug failing tests, since they're all named\nthe same. Better to just run the commands inline, and wrap the\ninvocations of _graph_git_behavior in an appropriately named\ntest_expect_success.\n"},{"id":"338069","messageId":"20180202024548.2663-1-szeder.dev@gmail.com","threadId":"47724","inReplyTo":"1517348383-112294-8-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v2 07/14] commit-graph: implement git-commit-graph --update-head","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-02-02T02:45:48Z","receivedAt":"2018-02-02T02:45:59Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index da565624e3..d1a23bcdaf 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n\n\n> @@ -107,6 +112,9 @@ test_expect_success 'setup bare repo' \\\n> test_expect_success 'write graph in bare repo' \\\n> 'graphbare=$(git commit-graph --write) &&\n> test_path_is_file ${baredir}/graph-${graphbare}.graph &&\n> + test_path_is_file ${baredir}/graph-head &&\n\nThis test and the one preceeding it are wrong.\n\nNote that 'git commit-graph --write' above is missing the\n'--update-head' option, so there should be no graph-head file written,\nyet this 'this test_path_is_file' doesn't fail the test.\n\nThe devil lies in the previous test 'setup bare repo', where this bare\nrepo is created by cloning from a local remote: a simple 'git clone\n--bare full bare' hardlinks all files under .git/objects, including\nall graph and graph-head files that exist in the remote repo.\n\nThe previous test should run 'git clone --bare --no-local full bare'\ninstead, and then this test would fail because of the missing\ngraph-head file, as it should. Specifying '--update-head' will make\nit work again.\n\n\n> + echo ${graphbare} >expect &&\n> + cmp -n 40 expect ${baredir}/graph-head &&\n> git commit-graph --read --graph-hash=${graphbare} >output &&\n> _graph_read_expect \"18\" \"${baredir}\" &&\n> cmp expect output'\n\n\n"},{"id":"338070","messageId":"20180202040100.13793-1-szeder.dev@gmail.com","threadId":"47724","inReplyTo":"1517348383-112294-9-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v2 08/14] commit-graph: implement git-commit-graph --clear","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-02-02T04:01:00Z","receivedAt":"2018-02-02T04:01:16Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"> Teach Git to delete the current 'graph_head' file and the commit graph\n> it references.\n\nAnd will it leave other, non-important graph files behind? Looking at\nthe code it indeed does. What is the use case for keeping the\nnon-important graph files?\n\n> This is a good safety valve if somehow the file is\n> corrupted and needs to be recalculated. Since the commit graph is a\n> summary of contents already in the ODB, it can be regenerated.\n\nWouldn't a simple 'git commit-graph --write --update-head' regenerate\nit on it's own, without cleaning first? It appears, after running a\nfew tests, that a corrupt graph file can be recreated without\ncleaning, which is great. However, if graph-head is corrupt, then the\ncommand errors out with 'failed to read graph-head'. I don't\nunderstand the rationale behind this, it would be overwritten anyway,\nand its content is not necessary for recreating the graph. And\nindeed, after commenting out that get_graph_head_hash() call in\ncmd_commit_graph() it doesn't want to read my corrupted graph-head\nfile, and recreates both the graph and graph-head files just fine.\n\nI think the requirement for explicitly cleaning a corrupt graph-head\nbefore re-writing it is just unnecessary complication.\n\nOn second thought, what's the point of '--write' without\n'--update-head', when consumers (thinking 'log --topo-order...) will\nneed the graph-head anyway? I think '--write' should create a\ngraph-head without requiring an additional option.\n\nHmph, another second thought: the word 'head' has a rather specific\nmeaning in Git, although it's usually capitalized. Using this word in\noptions and filenames may lead to confusion, especially the option\n'--update-head'.\n\n\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-commit-graph.txt | 16 ++++++++++++++--\n> builtin/commit-graph.c | 32 +++++++++++++++++++++++++++++++-\n> t/t5318-commit-graph.sh | 7 ++++++-\n> 3 files changed, 51 insertions(+), 4 deletions(-)\n> \n> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> index 99ced16ddc..33d6567f11 100644\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -11,6 +11,7 @@ SYNOPSIS\n> [verse]\n> 'git commit-graph' --write [--pack-dir ]\n> 'git commit-graph' --read [--pack-dir ]\n> +'git commit-graph' --clear [--pack-dir ]\n> \n> OPTIONS\n> -------\n> @@ -18,16 +19,21 @@ OPTIONS\n> \tUse given directory for the location of packfiles, graph-head,\n> \tand graph files.\n> \n> +--clear::\n> +\tDelete the graph-head file and the graph file it references.\n> +\t(Cannot be combined with --read or --write.)\n> +\n> --read::\n> \tRead a graph file given by the graph-head file and output basic\n> -\tdetails about the graph file. (Cannot be combined with --write.)\n> +\tdetails about the graph file. (Cannot be combined with --clear\n> +\tor --write.)\n> \n> --graph-id::\n> \tWhen used with --read, consider the graph file graph-.graph.\n> \n> --write::\n> \tWrite a new graph file to the pack directory. (Cannot be combined\n> -\twith --read.)\n> +\twith --clear or --read.)\n\nAll these \"cannot be combined with --this and --that\" remarks make\nsubcommands more and more appealing.\n\n> \n> --update-head::\n> \tWhen used with --write, update the graph-head file to point to\n> @@ -61,6 +67,12 @@ $ git commit-graph --write --update-head\n> $ git commit-graph --read --graph-hash=\n> ------------------------------------------------\n> \n> +* Delete /graph-head and the file it references.\n> ++\n> +------------------------------------------------\n> +$ git commit-graph --clear --pack-dir=\n> +------------------------------------------------\n> +\n> GIT\n> ---\n> Part of the linkgit:git[1] suite\n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index d73cbc907d..4970dec133 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -10,6 +10,7 @@\n> \n> static char const * const builtin_commit_graph_usage[] = {\n> \tN_(\"git commit-graph [--pack-dir ]\"),\n> +\tN_(\"git commit-graph --clear [--pack-dir ]\"),\n> \tN_(\"git commit-graph --read [--graph-hash=]\"),\n> \tN_(\"git commit-graph --write [--pack-dir ] [--update-head]\"),\n> \tNULL\n> @@ -17,6 +18,7 @@ static char const * const builtin_commit_graph_usage[] = {\n> \n> static struct opts_commit_graph {\n> \tconst char *pack_dir;\n> +\tint clear;\n> \tint read;\n> \tconst char *graph_hash;\n> \tint write;\n> @@ -25,6 +27,30 @@ static struct opts_commit_graph {\n> \tstruct object_id old_graph_hash;\n> } opts;\n> \n> +static int graph_clear(void)\n> +{\n> +\tstruct strbuf head_path = STRBUF_INIT;\n> +\tchar *old_path;\n> +\n> +\tif (!opts.has_existing)\n> +\t\treturn 0;\n> +\n> +\tstrbuf_addstr(&head_path, opts.pack_dir);\n> +\tstrbuf_addstr(&head_path, \"/\");\n> +\tstrbuf_addstr(&head_path, \"graph-head\");\n\nstrbuf_addstr(&head_path, \"/graph-head\")\n\nAlthough, considering that this is the third place assembling this\npath, maybe a helper function would be worth it.\n\n> +\tif (remove_path(head_path.buf))\n> +\t\tdie(\"failed to remove path %s\", head_path.buf);\n> +\tstrbuf_release(&head_path);\n> +\n> +\told_path = get_commit_graph_filename_hash(opts.pack_dir,\n> +\t\t\t\t\t\t &opts.old_graph_hash);\n> +\tif (remove_path(old_path))\n> +\t\tdie(\"failed to remove path %s\", old_path);\n> +\tfree(old_path);\n> +\n> +\treturn 0;\n> +}\n> +\n> static int graph_read(void)\n> {\n> \tstruct object_id graph_hash;\n> @@ -105,6 +131,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n> \t\t{ OPTION_STRING, 'p', \"pack-dir\", &opts.pack_dir,\n> \t\t\tN_(\"dir\"),\n> \t\t\tN_(\"The pack directory to store the graph\") },\n> +\t\tOPT_BOOL('c', \"clear\", &opts.clear,\n> +\t\t\tN_(\"clear graph file and graph-head\")),\n> \t\tOPT_BOOL('r', \"read\", &opts.read,\n> \t\t\tN_(\"read graph file\")),\n> \t\tOPT_BOOL('w', \"write\", &opts.write,\n> @@ -126,7 +154,7 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n> \t\t\t builtin_commit_graph_options,\n> \t\t\t builtin_commit_graph_usage, 0);\n> \n> -\tif (opts.write + opts.read > 1)\n> +\tif (opts.write + opts.read + opts.clear > 1)\n> \t\tusage_with_options(builtin_commit_graph_usage,\n> \t\t\t\t builtin_commit_graph_options);\n> \n> @@ -139,6 +167,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n> \n> \topts.has_existing = !!get_graph_head_hash(opts.pack_dir, &opts.old_graph_hash);\n> \n> +\tif (opts.clear)\n> +\t\treturn graph_clear();\n> \tif (opts.read)\n> \t\treturn graph_read();\n> \tif (opts.write)\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index d1a23bcdaf..6e3b62b754 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -101,6 +101,11 @@ test_expect_success 'write graph with merges' \\\n> _graph_read_expect \"18\" \"${packdir}\" &&\n> cmp expect output'\n> \n> +test_expect_success 'clear graph' \\\n> + 'git commit-graph --clear &&\n> + test_path_is_missing ${packdir}/graph-${graph2}.graph &&\n> + test_path_is_missing ${packdir}/graph-head'\n> +\n> test_expect_success 'setup bare repo' \\\n> 'cd .. &&\n> git clone --bare full bare &&\n> @@ -110,7 +115,7 @@ test_expect_success 'setup bare repo' \\\n> baredir=\"./objects/pack\"'\n> \n> test_expect_success 'write graph in bare repo' \\\n> - 'graphbare=$(git commit-graph --write) &&\n> + 'graphbare=$(git commit-graph --write --update-head) &&\n\nThis should have been done in the previous patch.\n\n> test_path_is_file ${baredir}/graph-${graphbare}.graph &&\n> test_path_is_file ${baredir}/graph-head &&\n> echo ${graphbare} >expect &&\n> -- \n> 2.16.0.15.g9c3cf44.dirty\n\n"},{"id":"338096","messageId":"20180202150441.27206-1-szeder.dev@gmail.com","threadId":"47724","inReplyTo":"1517348383-112294-10-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v2 09/14] commit-graph: teach git-commit-graph --delete-expired","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-02-02T15:04:41Z","receivedAt":"2018-02-02T15:05:03Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"> Teach git-commit-graph to delete the graph previously referenced by 'graph_head'\n> when writing a new graph file and updating 'graph_head'. This prevents\n> data creep by storing a list of useless graphs. Be careful to not delete\n> the graph if the file did not change.\n\nWe have to be careful with deleting the previously referenced graph\nfile right away after generating the new one. Consider two processes\nrunning concurrently, one writing new graph files with\n--delete-expire', and the other reading the commit graph, e.g. a\nfuture graph-aware 'git gc' and 'git log --topo-order':\n\n 1. 'log' reads the hash of the graph file from graph-head.\n 2. 'gc' writes the new graph and graph head files and deletes the\n old graph file.\n 3. 'log' tries to open the the graph file with the hash it just\n read, but that file is already gone.\n\nAt this point 'log' could simply error out, but that would be rather\nunfriendly. Or it could try harder and could just ignore the missing\ngraph file and walk revisions the old school way. It would be slower,\ndepending on the history size maybe much slower, but it would work.\nGood.\n\nHowever, in addition to the reader trying harder, I think we should\nalso consider making the writer more careful, too, and only delete a\nstale graph file after a certain grace period is elapsed; similar to\nhow 'git gc' only deletes old loose objects. And then perhaps it\nshould delete all graph files that are older than that grace period;\nas it is, neither '--clear' nor '--delete-expired' seem to care about\ngraph files that aren't or weren't referenced by the graph-head.\n\n\n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index 4970dec133..766f09e6fc 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n\n> @@ -121,6 +122,17 @@ static int graph_write(void)\n> \tif (graph_hash)\n> \t\tprintf(\"%s\\n\", oid_to_hex(graph_hash));\n> \n> +\n> +\tif (opts.delete_expired && opts.update_head && opts.has_existing &&\n> +\t oidcmp(graph_hash, &opts.old_graph_hash)) {\n> +\t\tchar *old_path = get_commit_graph_filename_hash(opts.pack_dir,\n> +\t\t\t\t\t\t\t\t&opts.old_graph_hash);\n> +\t\tif (remove_path(old_path))\n> +\t\t\tdie(\"failed to remove path %s\", old_path);\n> +\n> +\t\tfree(old_path);\n> +\t}\n> +\n> \tfree(graph_hash);\n> \treturn 0;\n> }\n> @@ -139,6 +151,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n> \t\t\tN_(\"write commit graph file\")),\n> \t\tOPT_BOOL('u', \"update-head\", &opts.update_head,\n> \t\t\tN_(\"update graph-head to written graph file\")),\n> +\t\tOPT_BOOL('d', \"delete-expired\", &opts.delete_expired,\n> +\t\t\tN_(\"delete expired head graph file\")),\n> \t\t{ OPTION_STRING, 'H', \"graph-hash\", &opts.graph_hash,\n> \t\t\tN_(\"hash\"),\n> \t\t\tN_(\"A hash for a specific graph file in the pack-dir.\"),\n\nLike '--update-head', '--delete-expired' is silently ignored when it's\nnot used with '--write'.\n\n\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 6e3b62b754..b56a6d4217 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n\n> +test_expect_success 'write graph with merges' \\\n> + 'graph3=$(git commit-graph --write --update-head --delete-expired) &&\n> + test_path_is_file ${packdir}/graph-${graph3}.graph &&\n> + test_path_is_missing ${packdir}/graph-${graph2}.graph &&\n> + test_path_is_file ${packdir}/graph-${graph1}.graph &&\n> + test_path_is_file ${packdir}/graph-head &&\n> + echo ${graph3} >expect &&\n> + cmp -n 40 expect ${packdir}/graph-head &&\n\nprintf and test_cmp.\n\n> + git commit-graph --read --graph-hash=${graph3} >output &&\n> + _graph_read_expect \"23\" \"${packdir}\" &&\n> + cmp expect output'\n> +\n> +test_expect_success 'write graph with nothing new' \\\n> + 'graph4=$(git commit-graph --write --update-head --delete-expired) &&\n> + test_path_is_file ${packdir}/graph-${graph4}.graph &&\n> + test_path_is_file ${packdir}/graph-${graph1}.graph &&\n> + test_path_is_file ${packdir}/graph-head &&\n> + echo ${graph4} >expect &&\n> + cmp -n 40 expect ${packdir}/graph-head &&\n\nLikewise.\n\n> + git commit-graph --read --graph-hash=${graph4} >output &&\n> + _graph_read_expect \"23\" \"${packdir}\" &&\n> + cmp expect output'\n> +\n> test_expect_success 'clear graph' \\\n> 'git commit-graph --clear &&\n> test_path_is_missing ${packdir}/graph-${graph2}.graph &&\n> + test_path_is_file ${packdir}/graph-${graph1}.graph &&\n> test_path_is_missing ${packdir}/graph-head'\n> \n> test_expect_success 'setup bare repo' \\\n> @@ -121,7 +185,7 @@ test_expect_success 'write graph in bare repo' \\\n> echo ${graphbare} >expect &&\n> cmp -n 40 expect ${baredir}/graph-head &&\n> git commit-graph --read --graph-hash=${graphbare} >output &&\n> - _graph_read_expect \"18\" \"${baredir}\" &&\n> + _graph_read_expect \"23\" \"${baredir}\" &&\n> cmp expect output'\n> \n> test_done\n> -- \n> 2.16.0.15.g9c3cf44.dirty\n\n\n"},{"id":"338098","messageId":"20180202153212.29746-1-szeder.dev@gmail.com","threadId":"47724","inReplyTo":"1517348383-112294-5-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v2 04/14] commit-graph: implement construct_commit_graph()","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-02-02T15:32:12Z","receivedAt":"2018-02-02T15:32:31Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"\n> Teach Git to write a commit graph file by checking all packed objects\n> to see if they are commits, then store the file in the given pack\n> directory.\n\nI'm afraid that scanning all packed objects is a bit of a roundabout\nway to approach this.\n\nIn my git repo, with 9 pack files at the moment, i.e. not that big a\nrepo and not that many pack files:\n\n $ time ./git commit-graph --write --update-head\n 4df41a3d1cc408b7ad34bea87b51ec4ccbf4b803\n\n real 0m27.550s\n user 0m27.113s\n sys 0m0.376s\n\nIn comparison, performing a good old revision walk to gather all the\ninfo that is written into the graph file:\n\n $ time git log --all --topo-order --format='%H %T %P %cd' |wc -l\n 52954\n\n real 0m0.903s\n user 0m0.972s\n sys 0m0.058s\n\n\n\n> +char* get_commit_graph_filename_hash(const char *pack_dir,\n> +\t\t\t\t struct object_id *hash)\n> +{\n> +\tsize_t len;\n> +\tstruct strbuf head_path = STRBUF_INIT;\n> +\tstrbuf_addstr(&head_path, pack_dir);\n> +\tstrbuf_addstr(&head_path, \"/graph-\");\n> +\tstrbuf_addstr(&head_path, oid_to_hex(hash));\n> +\tstrbuf_addstr(&head_path, \".graph\");\n\nNit: this is assembling the path of a graph file, not that of a\ngraph-head, so the strbuf should be renamed accordingly.\n\n> +\n> +\treturn strbuf_detach(&head_path, &len);\n> +}\n\n"},{"id":"338099","messageId":"20180202160132.31550-1-szeder.dev@gmail.com","threadId":"47724","inReplyTo":"1517348383-112294-11-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v2 10/14] commit-graph: add core.commitgraph setting","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-02-02T16:01:32Z","receivedAt":"2018-02-02T16:01:55Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"> The commit graph feature is controlled by the new core.commitgraph config\n> setting. This defaults to 0, so the feature is opt-in.\n> \n> The intention of core.commitgraph is that a user can always stop checking\n> for or parsing commit graph files if core.commitgraph=0.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/config.txt | 3 +++\n> cache.h | 1 +\n> config.c | 5 +++++\n> environment.c | 1 +\n> 4 files changed, 10 insertions(+)\n\nPlease squash this in to keep the completion script up to date.\n\n\n -- >8 --\n\ndiff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash\nindex 3683c772c..53880f627 100644\n--- a/contrib/completion/git-completion.bash\n+++ b/contrib/completion/git-completion.bash\n@@ -2419,6 +2419,7 @@ _git_config ()\n \t\tcore.bigFileThreshold\n \t\tcore.checkStat\n \t\tcore.commentChar\n+\t\tcore.commitGraph\n \t\tcore.compression\n \t\tcore.createObject\n \t\tcore.deltaBaseCacheLimit\n\n"},{"id":"338105","messageId":"CAGZ79kb8kHAb6HYrxY_oMq5v3p1+AHX_ad40nwUEwiDKe=i+Ng@mail.gmail.com","threadId":"47724","inReplyTo":"20180201153349.c94fe3de6b632e2fd8f843cf@google.com","subject":"Re: [PATCH v2 05/14] commit-graph: implement git-commit-graph --write","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-02T18:36:51Z","receivedAt":"2018-02-02T18:37:00Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Thu, Feb 1, 2018 at 3:33 PM, Jonathan Tan wrote:\n> On Tue, 30 Jan 2018 16:39:34 -0500\n> Derrick Stolee wrote:\n>\n>> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n>> index c8ea548dfb..3f3790d9a8 100644\n>> --- a/Documentation/git-commit-graph.txt\n>> +++ b/Documentation/git-commit-graph.txt\n>> @@ -5,3 +5,21 @@ NAME\n>> ----\n>> git-commit-graph - Write and verify Git commit graphs (.graph files)\n>>\n>> +\n>> +SYNOPSIS\n>> +--------\n>> +[verse]\n>> +'git commit-graph' --write [--pack-dir ]\n>\n> Subcommands (like those in git submodule) generally don't take \"--\", as\n> far as I know.\n\nThen you know only the ugly side of Git. ;)\n\nIt is true for git-submodule and a few others (the minority of commands IIRC)\ngit-tag for example takes subcommands such as --list or --verify.\nhttps://public-inbox.org/git/xmqqiomodkt9.fsf@gitster.dls.corp.google.com/\n"},{"id":"338128","messageId":"xmqq607fc8j8.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"CAGZ79kb8kHAb6HYrxY_oMq5v3p1+AHX_ad40nwUEwiDKe=i+Ng@mail.gmail.com","subject":"Re: [PATCH v2 05/14] commit-graph: implement git-commit-graph --write","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-02T22:48:43Z","receivedAt":"2018-02-02T22:48:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> It is true for git-submodule and a few others (the minority of commands IIRC)\n> git-tag for example takes subcommands such as --list or --verify.\n> https://public-inbox.org/git/xmqqiomodkt9.fsf@gitster.dls.corp.google.com/\n\nThanks. It refers to an article at gmane, which is not readily\naccessible unless you use newsreader. The original discussion it\nrefers to appears at:\n\n https://public-inbox.org/git/7vbo5itjfl.fsf@alter.siamese.dyndns.org/\n\nfor those who are interested.\n\nI am still not sure if it is a good design to add a new command like\nthis series does, though. I would naively have expected that this\nwould be a new pack index format that is produced by pack-objects\nand index-pack, for example, in which case its maintenance would\nalmost be invisible to end users (i.e. just like how the pack bitmap\nfeature was added to the system).\n\n\n"},{"id":"338142","messageId":"25571438-bdb1-81ce-ac5e-18dd0b6292d5@gmail.com","threadId":"47724","inReplyTo":"xmqq607fc8j8.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 05/14] commit-graph: implement git-commit-graph --write","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-03T01:58:52Z","receivedAt":"2018-02-03T01:59:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/2/2018 5:48 PM, Junio C Hamano wrote:\n> Stefan Beller writes:\n> \n>> It is true for git-submodule and a few others (the minority of commands IIRC)\n>> git-tag for example takes subcommands such as --list or --verify.\n>> https://public-inbox.org/git/xmqqiomodkt9.fsf@gitster.dls.corp.google.com/\n> \n> Thanks. It refers to an article at gmane, which is not readily\n> accessible unless you use newsreader. The original discussion it\n> refers to appears at:\n> \n> https://public-inbox.org/git/7vbo5itjfl.fsf@alter.siamese.dyndns.org/\n> \n> for those who are interested.\n\nThanks for the links.\n\n> I am still not sure if it is a good design to add a new command like\n> this series does, though. I would naively have expected that this\n> would be a new pack index format that is produced by pack-objects\n> and index-pack, for example, in which case its maintenance would\n> almost be invisible to end users (i.e. just like how the pack bitmap\n> feature was added to the system).\n\nI agree that the medium-term goal is to have this happen without user \nintervention. Something like a \"core.autoCommitGraph\" setting to trigger \ncommit-graph writes during other cleanup activities, such as a repack or \na gc.\n\nI don't think pairing this with pack-objects or index-pack is a good \ndirection, because the commit graph is not locked into a packfile the \nway the bitmap is. In fact, the entire ODB could be replaced \nindependently and the graph is still valid (the commits in the graph may \nno longer have their paired commits in the ODB due to a GC; you should \nnever navigate to those commits without having a ref pointing to them, \nso this is not immediately a problem).\n\nThis sort of interaction with GC is one reason why I did not include the \nautomatic updates in this patch. The integration with existing \nmaintenance tasks will be worth discussion in its own right. I'd rather \ndemonstrate the value of having a graph (even if it is currently \nmaintained manually) and then follow up with a focus to integrate with \nrepack, gc, etc.\n\nI plan to clean up this patch on Monday given the feedback I received \nthe last two days (Thanks Jonathan and Szeder!). However, if the current \nbuiltin design will block merging, then I'll wait until we can find one \nthat works.\n\nThanks,\n-Stolee\n"},{"id":"338147","messageId":"20180203092806.GA25927@sigill.intra.peff.net","threadId":"47724","inReplyTo":"25571438-bdb1-81ce-ac5e-18dd0b6292d5@gmail.com","subject":"Re: [PATCH v2 05/14] commit-graph: implement git-commit-graph --write","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-02-03T09:28:07Z","receivedAt":"2018-02-03T09:37:08Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Feb 02, 2018 at 08:58:52PM -0500, Derrick Stolee wrote:\n\n> I don't think pairing this with pack-objects or index-pack is a good\n> direction, because the commit graph is not locked into a packfile the way\n> the bitmap is. In fact, the entire ODB could be replaced independently and\n> the graph is still valid (the commits in the graph may no longer have their\n> paired commits in the ODB due to a GC; you should never navigate to those\n> commits without having a ref pointing to them, so this is not immediately a\n> problem).\n\nOne advantage of tying this to packs is that you can piggy-back on the\n.idx to avoid storing object ids a second time. If we imagine that you\nuse a 32-bit index into the .idx instead, that's a savings of 16 bytes\nper object (or more when we switch to a longer hash). You only need to\nrefer to commits and their root trees, though. So on something like\nlinux.git, you're talking about 2 * 700k * 16 = 21 megabytes you could\nsave.\n\nThat may not be worth worrying about too much, compared to the size of\nthe rest of the data. Disk space is obviously cheap, but I'm more\nconcerned about working-set size. However, 21 megabytes probably isn't\nbreaking the bank there these days (and it may even be faster, since the\ncommit-graph lookups can use the more compact index that contains only\ncommits, not other objects).\n\nThe big advantage of your scheme is that you can update the graph index\nwithout repacking. The traditional advice has been that you should\nalways do a full repack during a gc (since it gives the most delta\nopportunities). So metadata like reachability bitmaps were happy to tied\nto packs, since you're repacking anyway during a gc. But my\nunderstanding is that this doesn't really fly with the Windows\nrepository, where it's simply so big that you never obtain a single\npack, and just pass around slices of history in pack format.\n\nSo I think I'm OK with the direction here of keeping metadata caches\nseparate from the pack storage.\n\n> This sort of interaction with GC is one reason why I did not include the\n> automatic updates in this patch. The integration with existing maintenance\n> tasks will be worth discussion in its own right. I'd rather demonstrate the\n> value of having a graph (even if it is currently maintained manually) and\n> then follow up with a focus to integrate with repack, gc, etc.\n> \n> I plan to clean up this patch on Monday given the feedback I received the\n> last two days (Thanks Jonathan and Szeder!). However, if the current builtin\n> design will block merging, then I'll wait until we can find one that works.\n\nIf they're not tied to packs, then I think having a separate builtin\nlike this is the best approach. It gives you a plumbing command to\nexperiment with, and it can easily be called from git-gc.\n\n-Peff\n"},{"id":"338221","messageId":"b7f45961-35c8-6e13-646c-9574fb5d56da@gmail.com","threadId":"47724","inReplyTo":"20180202153212.29746-1-szeder.dev@gmail.com","subject":"Re: [PATCH v2 04/14] commit-graph: implement construct_commit_graph()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-05T16:06:56Z","receivedAt":"2018-02-05T16:07:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/2/2018 10:32 AM, SZEDER Gábor wrote:\n>> Teach Git to write a commit graph file by checking all packed objects\n>> to see if they are commits, then store the file in the given pack\n>> directory.\n> I'm afraid that scanning all packed objects is a bit of a roundabout\n> way to approach this.\n>\n> In my git repo, with 9 pack files at the moment, i.e. not that big a\n> repo and not that many pack files:\n>\n> $ time ./git commit-graph --write --update-head\n> 4df41a3d1cc408b7ad34bea87b51ec4ccbf4b803\n>\n> real 0m27.550s\n> user 0m27.113s\n> sys 0m0.376s\n>\n> In comparison, performing a good old revision walk to gather all the\n> info that is written into the graph file:\n>\n> $ time git log --all --topo-order --format='%H %T %P %cd' |wc -l\n> 52954\n>\n> real 0m0.903s\n> user 0m0.972s\n> sys 0m0.058s\n\nTwo reasons this is in here:\n\n(1) It's easier to get the write implemented this way and add the \nreachable closure later (which I do).\n\n(2) For GVFS, we want to add all commits that arrived in a \"prefetch \npack\" to the graph even if we do not have a ref that points to the \ncommit yet. We expect many commits to become reachable soon and having \nthem in the graph saves a lot of time in merge-base calculations.\n\nSo, (1) is for patch simplicity, and (2) is why I want it to be an \noption in the final version. See the --stdin-packs argument later for a \nway to do this incrementally.\n\nI expect almost all users to use the reachable closure method with \n--stdin-commits (and that's how I will integrate automatic updates with \n'fetch', 'repack', and 'gc' in a later patch).\n\n>\n>> +char* get_commit_graph_filename_hash(const char *pack_dir,\n>> +\t\t\t\t struct object_id *hash)\n>> +{\n>> +\tsize_t len;\n>> +\tstruct strbuf head_path = STRBUF_INIT;\n>> +\tstrbuf_addstr(&head_path, pack_dir);\n>> +\tstrbuf_addstr(&head_path, \"/graph-\");\n>> +\tstrbuf_addstr(&head_path, oid_to_hex(hash));\n>> +\tstrbuf_addstr(&head_path, \".graph\");\n> Nit: this is assembling the path of a graph file, not that of a\n> graph-head, so the strbuf should be renamed accordingly.\n>\n>> +\n>> +\treturn strbuf_detach(&head_path, &len);\n>> +}\n\n"},{"id":"338224","messageId":"6c05d38c-c9fe-e5e5-8081-c862de49d8dc@gmail.com","threadId":"47724","inReplyTo":"20180201234805.7506-1-szeder.dev@gmail.com","subject":"Re: [PATCH v2 05/14] commit-graph: implement git-commit-graph --write","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-05T18:07:45Z","receivedAt":"2018-02-05T18:07:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/1/2018 6:48 PM, SZEDER Gábor wrote:\n>> Teach git-commit-graph to write graph files. Create new test script to verify\n>> this command succeeds without failure.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> Documentation/git-commit-graph.txt | 18 +++++++\n>> builtin/commit-graph.c | 30 ++++++++++++\n>> t/t5318-commit-graph.sh | 96 ++++++++++++++++++++++++++++++++++++++\n>> 3 files changed, 144 insertions(+)\n>> create mode 100755 t/t5318-commit-graph.sh\n>>\n>> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n>> index c8ea548dfb..3f3790d9a8 100644\n>> --- a/Documentation/git-commit-graph.txt\n>> +++ b/Documentation/git-commit-graph.txt\n>> @@ -5,3 +5,21 @@ NAME\n>> ----\n>> git-commit-graph - Write and verify Git commit graphs (.graph files)\n>> \n>> +\n>> +SYNOPSIS\n>> +--------\n>> +[verse]\n>> +'git commit-graph' --write [--pack-dir ]\n>> +\n> What do these options do and what is the command's output? IOW, an\n> 'OPTIONS' section would be nice.\n>\n>> +EXAMPLES\n>> +--------\n>> +\n>> +* Write a commit graph file for the packed commits in your local .git folder.\n>> ++\n>> +------------------------------------------------\n>> +$ git commit-graph --write\n>> +------------------------------------------------\n>> +\n>> +GIT\n>> +---\n>> +Part of the linkgit:git[1] suite\n>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n>> new file mode 100755\n>> index 0000000000..6bcd1cc264\n>> --- /dev/null\n>> +++ b/t/t5318-commit-graph.sh\n>> @@ -0,0 +1,96 @@\n>> +#!/bin/sh\n>> +\n>> +test_description='commit graph'\n>> +. ./test-lib.sh\n>> +\n>> +test_expect_success 'setup full repo' \\\n>> + 'rm -rf .git &&\n>> + mkdir full &&\n>> + cd full &&\n>> + git init &&\n>> + git config core.commitgraph true &&\n>> + git config pack.threads 1 &&\n> Does this pack.threads=1 make a difference?\n>\n>> + packdir=\".git/objects/pack\"'\n> We tend to put single quotes around tests like this:\n>\n> test_expect_success 'setup full repo' '\n> do-this &&\n> check-that\n> '\n>\n> This is not a mere style nit: those newlines before and after the test\n> block make the test's output with '--verbose-log' slightly more\n> readable.\n>\n> Furthermore, we prefer tabs for indentation.\n\nOops! My bad for using t5302-pack-index.sh as my model for creating test \nscripts. It's pretty old, but I do see some of the newer tests using \nthis newer style.\n\n> Finally, 'cd'-ing around such that it affects subsequent tests is\n> usually frowned upon. However, in this particular case (going into\n> one repo, doing a bunch of tests there, then going into another repo,\n> and doing another bunch of tests) I think it's better than changing\n> directory in a subshell in every single test.\n>\n>> +\n>> +test_expect_success 'write graph with no packs' \\\n>> + 'git commit-graph --write --pack-dir .'\n>> +\n>> +test_expect_success 'create commits and repack' \\\n>> + 'for i in $(test_seq 5)\n>> + do\n>> + echo $i >$i.txt &&\n>> + git add $i.txt &&\n>> + git commit -m \"commit $i\" &&\n>> + git branch commits/$i\n>> + done &&\n>> + git repack'\n>> +\n>> +test_expect_success 'write graph' \\\n>> + 'graph1=$(git commit-graph --write) &&\n>> + test_path_is_file ${packdir}/graph-${graph1}.graph'\n> Style nit: those {} around the variable names are unnecessary, but I\n> see you use them a lot.\n>\n>> +\n>> +t_expect_success 'Add more commits' \\\n> This must be test_expect_success.\n>\n>> + 'git reset --hard commits/3 &&\n>> + for i in $(test_seq 6 10)\n>> + do\n>> + echo $i >$i.txt &&\n>> + git add $i.txt &&\n>> + git commit -m \"commit $i\" &&\n>> + git branch commits/$i\n>> + done &&\n>> + git reset --hard commits/3 &&\n>> + for i in $(test_seq 11 15)\n>> + do\n>> + echo $i >$i.txt &&\n>> + git add $i.txt &&\n>> + git commit -m \"commit $i\" &&\n>> + git branch commits/$i\n>> + done &&\n>> + git reset --hard commits/7 &&\n>> + git merge commits/11 &&\n>> + git branch merge/1 &&\n>> + git reset --hard commits/8 &&\n>> + git merge commits/12 &&\n>> + git branch merge/2 &&\n>> + git reset --hard commits/5 &&\n>> + git merge commits/10 commits/15 &&\n>> + git branch merge/3 &&\n>> + git repack'\n>> +\n>> +# Current graph structure:\n>> +#\n>> +# M3\n>> +# / |\\_____\n>> +# / 10 15\n>> +# / | |\n>> +# / 9 M2 14\n>> +# | |/ \\ |\n>> +# | 8 M1 | 13\n>> +# | |/ | \\_|\n>> +# 5 7 | 12\n>> +# | | \\__|\n>> +# 4 6 11\n>> +# |____/______/\n>> +# 3\n>> +# |\n>> +# 2\n>> +# |\n>> +# 1\n>> +\n>> +test_expect_success 'write graph with merges' \\\n>> + 'graph2=$(git commit-graph --write) &&\n>> + test_path_is_file ${packdir}/graph-${graph2}.graph'\n>> +\n>> +test_expect_success 'setup bare repo' \\\n>> + 'cd .. &&\n>> + git clone --bare full bare &&\n>> + cd bare &&\n>> + git config core.graph true &&\n>> + git config pack.threads 1 &&\n>> + baredir=\"objects/pack\"'\n>> +\n>> +test_expect_success 'write graph in bare repo' \\\n>> + 'graphbare=$(git commit-graph --write) &&\n>> + test_path_is_file ${baredir}/graph-${graphbare}.graph'\n>> +\n>> +test_done\n>> -- \n>> 2.16.0.15.g9c3cf44.dirty\n>\n\n"},{"id":"338228","messageId":"xmqqd11jb7ca.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"20180203092806.GA25927@sigill.intra.peff.net","subject":"Re: [PATCH v2 05/14] commit-graph: implement git-commit-graph --write","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-05T18:48:53Z","receivedAt":"2018-02-05T18:50:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> The big advantage of your scheme is that you can update the graph index\n> without repacking. The traditional advice has been that you should\n> always do a full repack during a gc (since it gives the most delta\n> opportunities). So metadata like reachability bitmaps were happy to tied\n> to packs, since you're repacking anyway during a gc. But my\n> understanding is that this doesn't really fly with the Windows\n> repository, where it's simply so big that you never obtain a single\n> pack, and just pass around slices of history in pack format.\n>\n> So I think I'm OK with the direction here of keeping metadata caches\n> separate from the pack storage.\n\nOK. I guess that the topology information surviving repacking is a\nreason good enough to keep this separate from pack files, and I\nagree with your \"If they're not tied to packs,...\" below, too.\n\nThanks.\n\n> If they're not tied to packs, then I think having a separate builtin\n> like this is the best approach. It gives you a plumbing command to\n> experiment with, and it can easily be called from git-gc.\n>\n> -Peff\n"},{"id":"338233","messageId":"9a460925-a143-ecc0-8651-900b8e731caf@gmail.com","threadId":"47724","inReplyTo":"20180201162324.fa8b38531955ee5378e5c5ff@google.com","subject":"Re: [PATCH v2 06/14] commit-graph: implement git-commit-graph --read","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-05T19:29:29Z","receivedAt":"2018-02-05T19:29:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/1/2018 7:23 PM, Jonathan Tan wrote:\n> On Tue, 30 Jan 2018 16:39:35 -0500\n> Derrick Stolee wrote:\n>\n>> Teach git-commit-graph to read commit graph files and summarize their contents.\n> One overall question - is the \"read\" command meant to be a command used\n> by the end user, or is it here just to test that some aspects of reading\n> works? If the former, I'm not sure how useful it is. And if the latter,\n> I think that it is more useful to just implementing something that reads\n> it, then make the 11/14 change (modifying parse_commit_gently) and\n> include a perf test to show that your commit graph reading is both\n> correct and (performance-)effective.\n\nThe \"read\" command is intended for use with the tests to verify that the \ndifferent --write commands write the correct number of commits at a \ntime. For example, we can verify that the closure under reachability \nworks when using --stdin-commits, that we get every commit in a pack \nwhen using --stdin-packs, and that we don't get more commits than we should.\n\nIt doesn't serve much purpose on the user-facing side, but this is \nintended to be a plumbing command that is called by other porcelain \ncommands.\n"},{"id":"338253","messageId":"316c100b-2c27-83e3-53d8-46b545f0353e@gmail.com","threadId":"47724","inReplyTo":"20180202013552.24931-1-szeder.dev@gmail.com","subject":"Re: [PATCH v2 07/14] commit-graph: implement git-commit-graph --update-head","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-05T21:01:31Z","receivedAt":"2018-02-05T21:01:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/1/2018 8:35 PM, SZEDER Gábor wrote:\n>> It is possible to have multiple commit graph files in a pack directory,\n>> but only one is important at a time. Use a 'graph_head' file to point\n>> to the important file.\n> This implies that all those other files are ignored, right?\n\nYes. We do not use directory listings to find graph files.\n\n>\n>> Teach git-commit-graph to write 'graph_head' upon\n>> writing a new commit graph file.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> Documentation/git-commit-graph.txt | 34 ++++++++++++++++++++++++++++++++++\n>> builtin/commit-graph.c | 38 +++++++++++++++++++++++++++++++++++---\n>> commit-graph.c | 25 +++++++++++++++++++++++++\n>> commit-graph.h | 2 ++\n>> t/t5318-commit-graph.sh | 12 ++++++++++--\n>> 5 files changed, 106 insertions(+), 5 deletions(-)\n>>\n>> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n>> index 09aeaf6c82..99ced16ddc 100644\n>> --- a/Documentation/git-commit-graph.txt\n>> +++ b/Documentation/git-commit-graph.txt\n>> @@ -12,15 +12,49 @@ SYNOPSIS\n>> 'git commit-graph' --write [--pack-dir ]\n>> 'git commit-graph' --read [--pack-dir ]\n>> \n>> +OPTIONS\n>> +-------\n> Oh, look, the 'OPTIONS' section I missed in the previous patches! ;)\n>\n> This should be split up and squashed into the previous patches where\n> the individual --options are first mentioned.\n>\n>> +--pack-dir::\n>> +\tUse given directory for the location of packfiles, graph-head,\n>> +\tand graph files.\n>> +\n>> +--read::\n>> +\tRead a graph file given by the graph-head file and output basic\n>> +\tdetails about the graph file. (Cannot be combined with --write.)\n> From the output of 'git commit-graph --read' it seems that it's not a\n> generally useful option to the user. Perhaps it should be mentioned\n> that it's only intended as a debugging aid? Or maybe it doesn't\n> really matter, because eventually this command will become irrelevant,\n> as other commands (clone, fetch, gc) will invoke it automagically...\n\nI'll add some wording to make this clear.\n\n>\n>> +--graph-id::\n>> +\tWhen used with --read, consider the graph file graph-.graph.\n>> +\n>> +--write::\n>> +\tWrite a new graph file to the pack directory. (Cannot be combined\n>> +\twith --read.)\n> I think this should also mention that it prints the generated graph\n> file's checksum.\n>\n>> +\n>> +--update-head::\n>> +\tWhen used with --write, update the graph-head file to point to\n>> +\tthe written graph file.\n> So it should be used with '--write', noted.\n>\n>> EXAMPLES\n>> --------\n>> \n>> +* Output the hash of the graph file pointed to by /graph-head.\n>> ++\n>> +------------------------------------------------\n>> +$ git commit-graph --pack-dir=\n>> +------------------------------------------------\n>> +\n>> * Write a commit graph file for the packed commits in your local .git folder.\n>> +\n>> ------------------------------------------------\n>> $ git commit-graph --write\n>> ------------------------------------------------\n>> \n>> +* Write a graph file for the packed commits in your local .git folder,\n>> +* and update graph-head.\n>> ++\n>> +------------------------------------------------\n>> +$ git commit-graph --write --update-head\n>> +------------------------------------------------\n>> +\n>> * Read basic information from a graph file.\n>> +\n>> ------------------------------------------------\n>> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n>> index 218740b1f8..d73cbc907d 100644\n>> --- a/builtin/commit-graph.c\n>> +++ b/builtin/commit-graph.c\n>> @@ -11,7 +11,7 @@\n>> static char const * const builtin_commit_graph_usage[] = {\n>> \tN_(\"git commit-graph [--pack-dir ]\"),\n>> \tN_(\"git commit-graph --read [--graph-hash=]\"),\n>> -\tN_(\"git commit-graph --write [--pack-dir ]\"),\n>> +\tN_(\"git commit-graph --write [--pack-dir ] [--update-head]\"),\n>> \tNULL\n>> };\n>> \n>> @@ -20,6 +20,9 @@ static struct opts_commit_graph {\n>> \tint read;\n>> \tconst char *graph_hash;\n>> \tint write;\n>> +\tint update_head;\n>> +\tint has_existing;\n>> +\tstruct object_id old_graph_hash;\n>> } opts;\n>> \n>> static int graph_read(void)\n>> @@ -30,8 +33,8 @@ static int graph_read(void)\n>> \n>> \tif (opts.graph_hash && strlen(opts.graph_hash) == GIT_MAX_HEXSZ)\n>> \t\tget_oid_hex(opts.graph_hash, &graph_hash);\n>> -\telse\n>> -\t\tdie(\"no graph hash specified\");\n>> +\telse if (!get_graph_head_hash(opts.pack_dir, &graph_hash))\n>> +\t\tdie(\"no graph-head exists\");\n>> \n>> \tgraph_file = get_commit_graph_filename_hash(opts.pack_dir, &graph_hash);\n>> \tgraph = load_commit_graph_one(graph_file, opts.pack_dir);\n>> @@ -62,10 +65,33 @@ static int graph_read(void)\n>> \treturn 0;\n>> }\n>> \n>> +static void update_head_file(const char *pack_dir, const struct object_id *graph_hash)\n>> +{\n>> +\tstruct strbuf head_path = STRBUF_INIT;\n>> +\tint fd;\n>> +\tstruct lock_file lk = LOCK_INIT;\n>> +\n>> +\tstrbuf_addstr(&head_path, pack_dir);\n>> +\tstrbuf_addstr(&head_path, \"/\");\n>> +\tstrbuf_addstr(&head_path, \"graph-head\");\n> strbuf_addstr(&head_path, \"/graph-head\"); ?\n>\n>> +\n>> +\tfd = hold_lock_file_for_update(&lk, head_path.buf, LOCK_DIE_ON_ERROR);\n>> +\tstrbuf_release(&head_path);\n>> +\n>> +\tif (fd < 0)\n>> +\t\tdie_errno(\"unable to open graph-head\");\n>> +\n>> +\twrite_in_full(fd, oid_to_hex(graph_hash), GIT_MAX_HEXSZ);\n>> +\tcommit_lock_file(&lk);\n> The new graph-head file will be writable. All other files in\n> .git/objects/pack are created read-only, including graph files. Just\n> pointing it out, but I don't think it's a bit deal; other than\n> consistency with the permissions of other files I don't have any\n> argument for making it read-only.\n\nI don't have strong opinions on the permissions difference, except that \na graph-.graph file should not change contents (or the hash will \nbe wrong) but a user or external tool could change the graph-head \ncontents to point to a different file. I can't think of a case where \nthat is important.\n"},{"id":"338515","messageId":"11c4f90e-558c-b80c-61f0-f70cdcf6edd9@gmail.com","threadId":"47724","inReplyTo":"20180201175151.89555950582a2e6c5279849a@google.com","subject":"Re: [PATCH v2 11/14] commit: integrate commit graph with commit parsing","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-06T14:53:01Z","receivedAt":"2018-02-06T14:53:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/1/2018 8:51 PM, Jonathan Tan wrote:\n> On Tue, 30 Jan 2018 16:39:40 -0500\n> Derrick Stolee wrote:\n>\n>> +/* global storage */\n>> +struct commit_graph *commit_graph = 0;\n> NULL, not 0.\n>\n>> +static int bsearch_graph(struct commit_graph *g, struct object_id *oid, uint32_t *pos)\n>> +{\n>> +\tuint32_t last, first = 0;\n>> +\n>> +\tif (oid->hash[0])\n>> +\t\tfirst = ntohl(*(uint32_t*)(g->chunk_oid_fanout + 4 * (oid->hash[0] - 1)));\n>> +\tlast = ntohl(*(uint32_t*)(g->chunk_oid_fanout + 4 * oid->hash[0]));\n>> +\n>> +\twhile (first < last) {\n>> +\t\tuint32_t mid = first + (last - first) / 2;\n>> +\t\tconst unsigned char *current;\n>> +\t\tint cmp;\n>> +\n>> +\t\tcurrent = g->chunk_oid_lookup + g->hdr->hash_len * mid;\n>> +\t\tcmp = hashcmp(oid->hash, current);\n>> +\t\tif (!cmp) {\n>> +\t\t\t*pos = mid;\n>> +\t\t\treturn 1;\n>> +\t\t}\n>> +\t\tif (cmp > 0) {\n>> +\t\t\tfirst = mid + 1;\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\t\tlast = mid;\n>> +\t}\n>> +\n>> +\t*pos = first;\n>> +\treturn 0;\n>> +}\n> This would be better in sha1-lookup.c, something like the reverse of commit\n> f1068efefe6d (\"sha1_file: drop experimental GIT_USE_LOOKUP search\",\n> 2017-08-09), except that it can be done using a simple binary search.\n\nI rebased my patch onto your binary search patch, so I'll use that in \nthe future.\n\n>\n>> +static int full_parse_commit(struct commit *item, struct commit_graph *g,\n>> +\t\t\t uint32_t pos, const unsigned char *commit_data)\n>> +{\n>> +\tstruct object_id oid;\n>> +\tstruct commit *new_parent;\n>> +\tuint32_t new_parent_pos;\n>> +\tuint32_t *parent_data_ptr;\n>> +\tuint64_t date_low, date_high;\n>> +\tstruct commit_list **pptr;\n>> +\n>> +\titem->object.parsed = 1;\n>> +\titem->graph_pos = pos;\n>> +\n>> +\thashcpy(oid.hash, commit_data);\n>> +\titem->tree = lookup_tree(&oid);\n>> +\n>> +\tdate_high = ntohl(*(uint32_t*)(commit_data + g->hdr->hash_len + 8)) & 0x3;\n>> +\tdate_low = ntohl(*(uint32_t*)(commit_data + g->hdr->hash_len + 12));\n>> +\titem->date = (timestamp_t)((date_high << 32) | date_low);\n>> +\n>> +\tpptr = &item->parents;\n>> +\n>> +\tnew_parent_pos = ntohl(*(uint32_t*)(commit_data + g->hdr->hash_len));\n>> +\tif (new_parent_pos == GRAPH_PARENT_NONE)\n>> +\t\treturn 1;\n>> +\tget_nth_commit_oid(g, new_parent_pos, &oid);\n>> +\tnew_parent = lookup_commit(&oid);\n>> +\tif (new_parent) {\n>> +\t\tnew_parent->graph_pos = new_parent_pos;\n>> +\t\tpptr = &commit_list_insert(new_parent, pptr)->next;\n>> +\t} else {\n>> +\t\tdie(\"could not find commit %s\", oid_to_hex(&oid));\n>> +\t}\n>> +\n>> +\tnew_parent_pos = ntohl(*(uint32_t*)(commit_data + g->hdr->hash_len + 4));\n>> +\tif (new_parent_pos == GRAPH_PARENT_NONE)\n>> +\t\treturn 1;\n>> +\tif (!(new_parent_pos & GRAPH_LARGE_EDGES_NEEDED)) {\n>> +\t\tget_nth_commit_oid(g, new_parent_pos, &oid);\n>> +\t\tnew_parent = lookup_commit(&oid);\n>> +\t\tif (new_parent) {\n>> +\t\t\tnew_parent->graph_pos = new_parent_pos;\n>> +\t\t\tpptr = &commit_list_insert(new_parent, pptr)->next;\n>> +\t\t} else\n>> +\t\t\tdie(\"could not find commit %s\", oid_to_hex(&oid));\n>> +\t\treturn 1;\n>> +\t}\n>> +\n>> +\tparent_data_ptr = (uint32_t*)(g->chunk_large_edges + 4 * (new_parent_pos ^ GRAPH_LARGE_EDGES_NEEDED));\n>> +\tdo {\n>> +\t\tnew_parent_pos = ntohl(*parent_data_ptr);\n>> +\n>> +\t\tget_nth_commit_oid(g, new_parent_pos & GRAPH_EDGE_LAST_MASK, &oid);\n>> +\t\tnew_parent = lookup_commit(&oid);\n>> +\t\tif (new_parent) {\n>> +\t\t\tnew_parent->graph_pos = new_parent_pos & GRAPH_EDGE_LAST_MASK;\n>> +\t\t\tpptr = &commit_list_insert(new_parent, pptr)->next;\n>> +\t\t} else\n>> +\t\t\tdie(\"could not find commit %s\", oid_to_hex(&oid));\n>> +\t\tparent_data_ptr++;\n>> +\t} while (!(new_parent_pos & GRAPH_LAST_EDGE));\n>> +\n>> +\treturn 1;\n>> +}\n> The part that converts into \n> seems to be duplicated 3 times. Refactor into a function?\n\nWill do.\n\n>\n>> +/**\n>> + * Fill 'item' to contain all information that would be parsed by parse_commit_buffer.\n>> + */\n>> +static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t pos)\n>> +{\n>> +\tuint32_t new_parent_pos;\n>> +\tuint32_t *parent_data_ptr;\n>> +\tconst unsigned char *commit_data = g->chunk_commit_data + (g->hdr->hash_len + 16) * pos;\n>> +\n>> +\tnew_parent_pos = ntohl(*(uint32_t*)(commit_data + g->hdr->hash_len));\n>> +\n>> +\tif (new_parent_pos == GRAPH_PARENT_MISSING)\n>> +\t\treturn 0;\n>> +\n>> +\tif (new_parent_pos == GRAPH_PARENT_NONE)\n>> +\t\treturn full_parse_commit(item, g, pos, commit_data);\n>> +\n>> +\tnew_parent_pos = ntohl(*(uint32_t*)(commit_data + g->hdr->hash_len + 4));\n>> +\n>> +\tif (new_parent_pos == GRAPH_PARENT_MISSING)\n>> +\t\treturn 0;\n>> +\tif (!(new_parent_pos & GRAPH_LARGE_EDGES_NEEDED))\n>> +\t\treturn full_parse_commit(item, g, pos, commit_data);\n>> +\n>> +\tnew_parent_pos = new_parent_pos ^ GRAPH_LARGE_EDGES_NEEDED;\n>> +\n>> +\tif (new_parent_pos == GRAPH_PARENT_MISSING)\n>> +\t\treturn 0;\n>> +\n>> +\tparent_data_ptr = (uint32_t*)(g->chunk_large_edges + 4 * new_parent_pos);\n>> +\tdo {\n>> +\t\tnew_parent_pos = ntohl(*parent_data_ptr);\n>> +\n>> +\t\tif ((new_parent_pos & GRAPH_EDGE_LAST_MASK) == GRAPH_PARENT_MISSING)\n>> +\t\t\treturn 0;\n>> +\n>> +\t\tparent_data_ptr++;\n>> +\t} while (!(new_parent_pos & GRAPH_LAST_EDGE));\n>> +\n>> +\treturn full_parse_commit(item, g, pos, commit_data);\n>> +}\n> This function seems to just check for GRAPH_PARENT_MISSING - could that\n> check be folded into full_parse_commit() instead? (Then\n> full_parse_commit can be renamed to fill_commit_in_graph.)\n\nI'd rather not have a really long method, but I could make the two steps \ntheir own static methods (one for checking and one for full parsing) to \nmake it more clear that there are two steps here.\n\n>\n>> @@ -439,9 +656,24 @@ struct object_id *construct_commit_graph(const char *pack_dir)\n>> \tchar *fname;\n>> \tstruct commit_list *parent;\n>> \n>> +\tprepare_commit_graph();\n>> +\n>> \toids.num = 0;\n>> \toids.size = 1024;\n>> +\n>> +\tif (commit_graph && oids.size < commit_graph->num_commits)\n>> +\t\toids.size = commit_graph->num_commits;\n>> +\n>> \tALLOC_ARRAY(oids.list, oids.size);\n>> +\n>> +\tif (commit_graph) {\n>> +\t\tfor (i = 0; i < commit_graph->num_commits; i++) {\n>> +\t\t\toids.list[i] = malloc(sizeof(struct object_id));\n>> +\t\t\tget_nth_commit_oid(commit_graph, i, oids.list[i]);\n>> +\t\t}\n>> +\t\toids.num = commit_graph->num_commits;\n>> +\t}\n>> +\n>> \tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n>> \tQSORT(oids.list, oids.num, commit_compare);\n\nThis change auto-includes the commits that were in the existing graph \ninto the new graph.\n\n>> \n>> @@ -525,6 +757,11 @@ struct object_id *construct_commit_graph(const char *pack_dir)\n>> \thashcpy(f_hash->hash, final_hash);\n>> \tfname = get_commit_graph_filename_hash(pack_dir, f_hash);\n>> \n>> +\tif (commit_graph) {\n>> +\t\tclose_commit_graph(commit_graph);\n>> +\t\tFREE_AND_NULL(commit_graph);\n>> +\t}\n>> +\n>> \tif (rename(graph_name, fname))\n>> \t\tdie(\"failed to rename %s to %s\", graph_name, fname);\n\nThis change is necessary if we are going to use --delete-expired, as we \nneed to unmap the file before we can delete it. Perhaps it would be \nbetter to close the graph in the builtin instead so that relationship is \nclearer.\n\n> What is the relation of these changes to construct_commit_graph() to the\n> rest of the patch?\n\n(answered above, since the two changes have different purposes)\n\n>> diff --git a/commit-graph.h b/commit-graph.h\n>> index 43eb0aec84..05ddbbe165 100644\n>> --- a/commit-graph.h\n>> +++ b/commit-graph.h\n>> @@ -4,6 +4,18 @@\n>> #include \"git-compat-util.h\"\n>> #include \"commit.h\"\n>> \n>> +/**\n>> + * Given a commit struct, try to fill the commit struct info, including:\n>> + * 1. tree object\n>> + * 2. date\n>> + * 3. parents.\n>> + *\n>> + * Returns 1 if and only if the commit was found in the packed graph.\n>> + *\n>> + * See parse_commit_buffer() for the fallback after this call.\n>> + */\n>> +extern int parse_commit_in_graph(struct commit *item);\n>> +\n>> extern struct object_id *get_graph_head_hash(const char *pack_dir,\n>> \t\t\t\t\t struct object_id *hash);\n>> extern char* get_commit_graph_filename_hash(const char *pack_dir,\n>> @@ -40,7 +52,13 @@ extern struct commit_graph {\n>> \n>> extern int close_commit_graph(struct commit_graph *g);\n>> \n>> -extern struct commit_graph *load_commit_graph_one(const char *graph_file, const char *pack_dir);\n>> +extern struct commit_graph *load_commit_graph_one(const char *graph_file,\n>> +\t\t\t\t\t\t const char *pack_dir);\n>> +extern void prepare_commit_graph(void);\n>> +\n>> +extern struct object_id *get_nth_commit_oid(struct commit_graph *g,\n>> +\t\t\t\t\t uint32_t n,\n>> +\t\t\t\t\t struct object_id *oid);\n>> \n>> extern struct object_id *construct_commit_graph(const char *pack_dir);\n> This header file now contains functions for reading the commit graph,\n> and functions for writing one. It seems to me that those are (and should\n> be) quite disjoint, so it might be better to separate them into two.\n\nThis header file provides a unified API surface for interacting with \ncommit graphs. I'm not a fan of how other write commands are hidden in \nthe builtins (like 'builtin/pack-objects.c' for writing packs). If there \nis a better example of how this split has been done in the root \ndirectory, I'm happy to consider it.\n\n>\n>> -int parse_commit_gently(struct commit *item, int quiet_on_missing)\n>> +int parse_commit_internal(struct commit *item, int quiet_on_missing, int check_packed)\n>> {\n>> \tenum object_type type;\n>> \tvoid *buffer;\n>> @@ -385,6 +386,8 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n>> \t\treturn -1;\n>> \tif (item->object.parsed)\n>> \t\treturn 0;\n>> +\tif (check_packed && parse_commit_in_graph(item))\n>> +\t\treturn 0;\n>> \tbuffer = read_sha1_file(item->object.oid.hash, &type, &size);\n>> \tif (!buffer)\n>> \t\treturn quiet_on_missing ? -1 :\n>> @@ -404,6 +407,11 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n>> \treturn ret;\n>> }\n>> \n>> +int parse_commit_gently(struct commit *item, int quiet_on_missing)\n>> +{\n>> +\treturn parse_commit_internal(item, quiet_on_missing, 1);\n>> +}\n> Are you planning to use parse_commit_internal() from elsewhere? (It\n> doesn't seem to be the case, at least from this patch series.)\n\nAt one point I was using it, but I removed the one caller and forgot to \nclean up.\n\n>\n>> diff --git a/log-tree.c b/log-tree.c\n>> index fca29d4799..156aed4541 100644\n>> --- a/log-tree.c\n>> +++ b/log-tree.c\n>> @@ -659,8 +659,7 @@ void show_log(struct rev_info *opt)\n>> \t\tshow_mergetag(opt, commit);\n>> \t}\n>> \n>> -\tif (!get_cached_commit_buffer(commit, NULL))\n>> -\t\treturn;\n>> +\tget_commit_buffer(commit, NULL);\n> This undoes an optimization that I discuss in my e-mail message here\n> [1]. If we decide to do this, it should at least be called out in the\n> commit message.\n>\n> [1] https://public-inbox.org/git/b88725476d9f13ba4381d85e5fe049f6ef93f621.1506714999.git.jonathantanmy@google.com/\n\nI will call this out more clearly in my commit message next time. My \nproblem with the existing code is that it doesn't just ignore the commit \ncontents but will actually not write a newline. I noticed during testing \n'git log --oneline' with the graph enabled and the output listed several \nshort-shas in one line.\n\n>\n>> +_graph_git_two_modes() {\n> No need for the name to start with an underscore, I think.\n>\n>> + git -c core.commitgraph=true $1 >output\n>> + git -c core.commitgraph=false $1 >expect\n>> + cmp output expect\n> Use test_cmp.\n>\n>> +}\n>> +\n>> +_graph_git_behavior() {\n>> + BRANCH=$1\n>> + COMPARE=$2\n>> + test_expect_success 'check normal git operations' \\\n>> + '_graph_git_two_modes \"log --oneline ${BRANCH}\" &&\n>> + _graph_git_two_modes \"log --topo-order ${BRANCH}\" &&\n>> + _graph_git_two_modes \"branch -vv\" &&\n>> + _graph_git_two_modes \"merge-base -a ${BRANCH} ${COMPARE}\"'\n>> +}\n> This makes it difficult to debug failing tests, since they're all named\n> the same. Better to just run the commands inline, and wrap the\n> invocations of _graph_git_behavior in an appropriately named\n> test_expect_success.\n\nI'll add a parameter that adds a message to each test about the state of \nthe repo and graph.\n\nThanks,\n-Stolee\n"},{"id":"338528","messageId":"4dce1997-e0fe-8a7f-622f-94f409f822ec@gmail.com","threadId":"47724","inReplyTo":"xmqqd11jb7ca.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 05/14] commit-graph: implement git-commit-graph --write","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-06T18:55:38Z","receivedAt":"2018-02-06T18:55:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/5/2018 1:48 PM, Junio C Hamano wrote:\n> Jeff King writes:\n>\n>> The big advantage of your scheme is that you can update the graph index\n>> without repacking. The traditional advice has been that you should\n>> always do a full repack during a gc (since it gives the most delta\n>> opportunities). So metadata like reachability bitmaps were happy to tied\n>> to packs, since you're repacking anyway during a gc. But my\n>> understanding is that this doesn't really fly with the Windows\n>> repository, where it's simply so big that you never obtain a single\n>> pack, and just pass around slices of history in pack format.\n>>\n>> So I think I'm OK with the direction here of keeping metadata caches\n>> separate from the pack storage.\n> OK. I guess that the topology information surviving repacking is a\n> reason good enough to keep this separate from pack files, and I\n> agree with your \"If they're not tied to packs,...\" below, too.\n>\n> Thanks.\n>\n>> If they're not tied to packs, then I think having a separate builtin\n>> like this is the best approach. It gives you a plumbing command to\n>> experiment with, and it can easily be called from git-gc.\n>>\n>> -Peff\n\nThanks for all the advice here. In addition to the many cleanups that \nwere suggested, I'm going to take a try at the \"subcommand\" approach. \nI'll use git-submodule--helper and git-remote as models for my \nimplementation.\n\nThanks,\n-Stolee\n"},{"id":"338629","messageId":"CAM0VKj=wmkBNH=psCRztXFrC13RiG1EaSw89Q6LJaNsdJDEFHg@mail.gmail.com","threadId":"47724","inReplyTo":"b7f45961-35c8-6e13-646c-9574fb5d56da@gmail.com","subject":"Re: [PATCH v2 04/14] commit-graph: implement construct_commit_graph()","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-02-07T15:08:07Z","receivedAt":"2018-02-07T15:08:36Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Feb 5, 2018 at 5:06 PM, Derrick Stolee wrote:\n> On 2/2/2018 10:32 AM, SZEDER Gábor wrote:\n\n>> In my git repo, with 9 pack files at the moment, i.e. not that big a\n>> repo and not that many pack files:\n>>\n>> $ time ./git commit-graph --write --update-head\n>> 4df41a3d1cc408b7ad34bea87b51ec4ccbf4b803\n>>\n>> real 0m27.550s\n>> user 0m27.113s\n>> sys 0m0.376s\n>>\n>> In comparison, performing a good old revision walk to gather all the\n>> info that is written into the graph file:\n>>\n>> $ time git log --all --topo-order --format='%H %T %P %cd' |wc -l\n>> 52954\n>>\n>> real 0m0.903s\n>> user 0m0.972s\n>> sys 0m0.058s\n>\n>\n> Two reasons this is in here:\n>\n> (1) It's easier to get the write implemented this way and add the reachable\n> closure later (which I do).\n>\n> (2) For GVFS, we want to add all commits that arrived in a \"prefetch pack\"\n> to the graph even if we do not have a ref that points to the commit yet. We\n> expect many commits to become reachable soon and having them in the graph\n> saves a lot of time in merge-base calculations.\n>\n> So, (1) is for patch simplicity, and (2) is why I want it to be an option in\n> the final version. See the --stdin-packs argument later for a way to do this\n> incrementally.\n>\n> I expect almost all users to use the reachable closure method with\n> --stdin-commits (and that's how I will integrate automatic updates with\n> 'fetch', 'repack', and 'gc' in a later patch).\n\nI see. I was about to ask about the expected use-cases of the\n'--stdin-packs' option, considering how much slower it is to enumerate\nall objects in pack files, but run out of time after patch 10.\n\nThe run-time using '--stdin-commits' is indeed great:\n\n $ time { git for-each-ref --format='%(objectname)' refs/heads/ | ./git\n commit-graph --write --update-head --stdin-commits ; }\n 82fe9a5cd715ff578f01f7f44e0611d7902d20c8\n\n real 0m0.985s\n user 0m0.916s\n sys 0m0.024s\n\nConsidering the run-time difference, I think in the end it would be a\nbetter default for a plain 'git commit-graph --write' to traverse\nhistory from all refs, and it should enumerate pack files only if\nexplicitly told so via '--stdin-packs'.\n\nTo be clear: I'm not saying that traversing history should already be\nthe default when introducing construct_commit_graph() and '--write'. If\nenumerating pack files keeps the earlier patches simpler and easier to\nreview, then by all means stick with it, and only change the\n'--stdin-*'-less behavior near the end of the series, when all the\nbuilding blocks are already in place (but then mention this in the early\ncommit messages).\n\n\nI have also noticed a segfault when feeding non-commit object names to\n'--stdin-commits', i.e. when I run the above command without restricting\n'git for-each-ref' to branches and it listed object names of tags as\nwell.\n\n $ git rev-parse v2.16.1 |./git commit-graph --write --update-head\n--stdin-commits\n error: Object eb5fcb24f69e13335cf6a6a1b1d4553fa2b0f202 not a commit\n error: Object eb5fcb24f69e13335cf6a6a1b1d4553fa2b0f202 not a commit\n error: Object eb5fcb24f69e13335cf6a6a1b1d4553fa2b0f202 not a commit\n Segmentation fault\n\n(gdb) bt\n#0 __memcpy_avx_unaligned ()\n at ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:126\n#1 0x00000000004ea97c in sha1write (f=0x356bbf0, buf=0x4, count=20)\n at csum-file.c:104\n#2 0x00000000004d98e1 in write_graph_chunk_data (f=0x356bbf0, hash_len=20,\n commits=0x3508de0, nr_commits=50615) at commit-graph.c:506\n#3 0x00000000004da9ca in construct_commit_graph (\n pack_dir=0x8ff360 \".git/objects/pack\", pack_indexes=0x0, nr_packs=0,\n commit_hex=0x8ff790, nr_commits=1) at commit-graph.c:818\n#4 0x000000000044184e in graph_write () at builtin/commit-graph.c:149\n#5 0x0000000000441a8c in cmd_commit_graph (argc=0, argv=0x7fffffffe310,\n prefix=0x0) at builtin/commit-graph.c:224\n#6 0x0000000000405a0a in run_builtin (p=0x8ad950 , argc=4,\n argv=0x7fffffffe310) at git.c:346\n#7 0x0000000000405ce4 in handle_builtin (argc=4, argv=0x7fffffffe310)\n at git.c:555\n#8 0x0000000000405ec8 in run_argv (argcp=0x7fffffffe1cc, argv=0x7fffffffe1c0)\n at git.c:607\n#9 0x0000000000406079 in cmd_main (argc=4, argv=0x7fffffffe310) at git.c:684\n#10 0x00000000004a85c8 in main (argc=5, argv=0x7fffffffe308)\n at common-main.c:43\n"},{"id":"338630","messageId":"31192658-5fe6-fcc6-7b9c-8e6b298e2474@gmail.com","threadId":"47724","inReplyTo":"CAM0VKj=wmkBNH=psCRztXFrC13RiG1EaSw89Q6LJaNsdJDEFHg@mail.gmail.com","subject":"Re: [PATCH v2 04/14] commit-graph: implement construct_commit_graph()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-07T15:10:29Z","receivedAt":"2018-02-07T15:10:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/7/2018 10:08 AM, SZEDER Gábor wrote:\n> On Mon, Feb 5, 2018 at 5:06 PM, Derrick Stolee wrote:\n>> On 2/2/2018 10:32 AM, SZEDER Gábor wrote:\n>>> In my git repo, with 9 pack files at the moment, i.e. not that big a\n>>> repo and not that many pack files:\n>>>\n>>> $ time ./git commit-graph --write --update-head\n>>> 4df41a3d1cc408b7ad34bea87b51ec4ccbf4b803\n>>>\n>>> real 0m27.550s\n>>> user 0m27.113s\n>>> sys 0m0.376s\n>>>\n>>> In comparison, performing a good old revision walk to gather all the\n>>> info that is written into the graph file:\n>>>\n>>> $ time git log --all --topo-order --format='%H %T %P %cd' |wc -l\n>>> 52954\n>>>\n>>> real 0m0.903s\n>>> user 0m0.972s\n>>> sys 0m0.058s\n>>\n>> Two reasons this is in here:\n>>\n>> (1) It's easier to get the write implemented this way and add the reachable\n>> closure later (which I do).\n>>\n>> (2) For GVFS, we want to add all commits that arrived in a \"prefetch pack\"\n>> to the graph even if we do not have a ref that points to the commit yet. We\n>> expect many commits to become reachable soon and having them in the graph\n>> saves a lot of time in merge-base calculations.\n>>\n>> So, (1) is for patch simplicity, and (2) is why I want it to be an option in\n>> the final version. See the --stdin-packs argument later for a way to do this\n>> incrementally.\n>>\n>> I expect almost all users to use the reachable closure method with\n>> --stdin-commits (and that's how I will integrate automatic updates with\n>> 'fetch', 'repack', and 'gc' in a later patch).\n> I see. I was about to ask about the expected use-cases of the\n> '--stdin-packs' option, considering how much slower it is to enumerate\n> all objects in pack files, but run out of time after patch 10.\n>\n> The run-time using '--stdin-commits' is indeed great:\n>\n> $ time { git for-each-ref --format='%(objectname)' refs/heads/ | ./git\n> commit-graph --write --update-head --stdin-commits ; }\n> 82fe9a5cd715ff578f01f7f44e0611d7902d20c8\n>\n> real 0m0.985s\n> user 0m0.916s\n> sys 0m0.024s\n>\n> Considering the run-time difference, I think in the end it would be a\n> better default for a plain 'git commit-graph --write' to traverse\n> history from all refs, and it should enumerate pack files only if\n> explicitly told so via '--stdin-packs'.\n>\n> To be clear: I'm not saying that traversing history should already be\n> the default when introducing construct_commit_graph() and '--write'. If\n> enumerating pack files keeps the earlier patches simpler and easier to\n> review, then by all means stick with it, and only change the\n> '--stdin-*'-less behavior near the end of the series, when all the\n> building blocks are already in place (but then mention this in the early\n> commit messages).\n\nI will consider this.\n\n> I have also noticed a segfault when feeding non-commit object names to\n> '--stdin-commits', i.e. when I run the above command without restricting\n> 'git for-each-ref' to branches and it listed object names of tags as\n> well.\n>\n> $ git rev-parse v2.16.1 |./git commit-graph --write --update-head\n> --stdin-commits\n> error: Object eb5fcb24f69e13335cf6a6a1b1d4553fa2b0f202 not a commit\n> error: Object eb5fcb24f69e13335cf6a6a1b1d4553fa2b0f202 not a commit\n> error: Object eb5fcb24f69e13335cf6a6a1b1d4553fa2b0f202 not a commit\n> Segmentation fault\n>\n> (gdb) bt\n> #0 __memcpy_avx_unaligned ()\n> at ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:126\n> #1 0x00000000004ea97c in sha1write (f=0x356bbf0, buf=0x4, count=20)\n> at csum-file.c:104\n> #2 0x00000000004d98e1 in write_graph_chunk_data (f=0x356bbf0, hash_len=20,\n> commits=0x3508de0, nr_commits=50615) at commit-graph.c:506\n> #3 0x00000000004da9ca in construct_commit_graph (\n> pack_dir=0x8ff360 \".git/objects/pack\", pack_indexes=0x0, nr_packs=0,\n> commit_hex=0x8ff790, nr_commits=1) at commit-graph.c:818\n> #4 0x000000000044184e in graph_write () at builtin/commit-graph.c:149\n> #5 0x0000000000441a8c in cmd_commit_graph (argc=0, argv=0x7fffffffe310,\n> prefix=0x0) at builtin/commit-graph.c:224\n> #6 0x0000000000405a0a in run_builtin (p=0x8ad950 , argc=4,\n> argv=0x7fffffffe310) at git.c:346\n> #7 0x0000000000405ce4 in handle_builtin (argc=4, argv=0x7fffffffe310)\n> at git.c:555\n> #8 0x0000000000405ec8 in run_argv (argcp=0x7fffffffe1cc, argv=0x7fffffffe1c0)\n> at git.c:607\n> #9 0x0000000000406079 in cmd_main (argc=4, argv=0x7fffffffe310) at git.c:684\n> #10 0x00000000004a85c8 in main (argc=5, argv=0x7fffffffe308)\n> at common-main.c:43\n\nI noticed this while preparing v3. I have a fix, but you now remind me \nthat I need to add tags to the test script.\n\nThanks,\n-Stolee\n"},{"id":"338757","messageId":"1518122258-157281-1-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1517348383-112294-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v3 00/14] Serialized Git Commit Graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-08T20:37:24Z","receivedAt":"2018-02-08T20:37:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Thanks to everyone who gave comments on v1 and v2.\n\nHopefully the following points have been addressed:\n\n* Fixed inter-commit problems where certain fixes did not arrive until\n later commits.\n\n* Converted from submode flags (\"git commit-graph --write\") to\n subcommands (\"git commit-graph write\").\n\n* Fixed a bug where a non-commit OID would cause a segfault when using\n --stdin-commits. Added a test for an annotated tag.\n\n* Numerous style issues, especially in the test script.\n\nI also based my patches on the branch jt/binsearch-with-fanout to make\nuse of the bsearch_hash() method.\n\nI look forward to your feedback.\n\nThanks,\n-Stolee\n\n-- >8 --\n\nAs promised [1], this patch contains a way to serialize the commit graph.\nThe current implementation defines a new file format to store the graph\nstructure (parent relationships) and basic commit metadata (commit date,\nroot tree OID) in order to prevent parsing raw commits while performing\nbasic graph walks. For example, we do not need to parse the full commit\nwhen performing these walks:\n\n* 'git log --topo-order -1000' walks all reachable commits to avoid\n incorrect topological orders, but only needs the commit message for\n the top 1000 commits.\n\n* 'git merge-base ' may walk many commits to find the correct\n boundary between the commits reachable from A and those reachable\n from B. No commit messages are needed.\n\n* 'git branch -vv' checks ahead/behind status for all local branches\n compared to their upstream remote branches. This is essentially as\n hard as computing merge bases for each.\n\nThe current patch speeds up these calculations by injecting a check in\nparse_commit_gently() to check if there is a graph file and using that\nto provide the required metadata to the struct commit.\n\nThe file format has room to store generation numbers, which will be\nprovided as a patch after this framework is merged. Generation numbers\nare referenced by the design document but not implemented in order to\nmake the current patch focus on the graph construction process. Once\nthat is stable, it will be easier to add generation numbers and make\ngraph walks aware of generation numbers one-by-one.\n\nHere are some performance results for a copy of the Linux repository\nwhere 'master' has 704,766 reachable commits and is behind 'origin/master'\nby 19,610 commits.\n\n| Command | Before | After | Rel % |\n|----------------------------------|--------|--------|-------|\n| log --oneline --topo-order -1000 | 5.9s | 0.7s | -88% |\n| branch -vv | 0.42s | 0.27s | -35% |\n| rev-list --all | 6.4s | 1.0s | -84% |\n| rev-list --all --objects | 32.6s | 27.6s | -15% |\n\nTo test this yourself, run the following on your repo:\n\n git config core.commitGraph true\n git show-ref -s | git commit-graph write --update-head --stdin-commits\n\nThe second command writes a commit graph file containing every commit\nreachable from your refs. Now, all git commands that walk commits will\ncheck your graph first before consulting the ODB. You can run your own\nperformance comparisions by toggling the 'core.commitgraph' setting.\n\n[1] https://public-inbox.org/git/d154319e-bb9e-b300-7c37-27b1dcd2a2ce@jeffhostetler.com/\n Re: What's cooking in git.git (Jan 2018, #03; Tue, 23)\n\n[2] https://github.com/derrickstolee/git/pull/2\n A GitHub pull request containing the latest version of this patch.\n\nDerrick Stolee (14):\n commit-graph: add format document\n graph: add commit graph design document\n commit-graph: create git-commit-graph builtin\n commit-graph: implement write_commit_graph()\n commit-graph: implement 'git-commit-graph write'\n commit-graph: implement 'git-commit-graph read'\n commit-graph: update graph-head during write\n commit-graph: implement 'git-commit-graph clear'\n commit-graph: implement --delete-expired\n commit-graph: add core.commitGraph setting\n commit: integrate commit graph with commit parsing\n commit-graph: close under reachability\n commit-graph: read only from specific pack-indexes\n commit-graph: build graph from starting commits\n\n .gitignore | 1 +\n Documentation/config.txt | 3 +\n Documentation/git-commit-graph.txt | 115 ++++\n Documentation/technical/commit-graph-format.txt | 91 +++\n Documentation/technical/commit-graph.txt | 189 ++++++\n Makefile | 2 +\n alloc.c | 1 +\n builtin.h | 1 +\n builtin/commit-graph.c | 335 ++++++++++\n cache.h | 1 +\n command-list.txt | 1 +\n commit-graph.c | 828 ++++++++++++++++++++++++\n commit-graph.h | 60 ++\n commit.c | 3 +\n commit.h | 3 +\n config.c | 5 +\n environment.c | 1 +\n git.c | 1 +\n log-tree.c | 3 +-\n packfile.c | 4 +-\n packfile.h | 2 +\n t/t5318-commit-graph.sh | 228 +++++++\n 22 files changed, 1874 insertions(+), 4 deletions(-)\n create mode 100644 Documentation/git-commit-graph.txt\n create mode 100644 Documentation/technical/commit-graph-format.txt\n create mode 100644 Documentation/technical/commit-graph.txt\n create mode 100644 builtin/commit-graph.c\n create mode 100644 commit-graph.c\n create mode 100644 commit-graph.h\n create mode 100755 t/t5318-commit-graph.sh\n\n-- \n2.7.4\n\n"},{"id":"338758","messageId":"1518122258-157281-2-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1518122258-157281-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v3 01/14] commit-graph: add format document","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-08T20:37:25Z","receivedAt":"2018-02-08T20:38:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Add document specifying the binary format for commit graphs. This\nformat allows for:\n\n* New versions.\n* New hash functions and hash lengths.\n* Optional extensions.\n\nBasic header information is followed by a binary table of contents\ninto \"chunks\" that include:\n\n* An ordered list of commit object IDs.\n* A 256-entry fanout into that list of OIDs.\n* A list of metadata for the commits.\n* A list of \"large edges\" to enable octopus merges.\n\nThe format automatically includes two parent positions for every\ncommit. This favors speed over space, since using only one position\nper commit would cause an extra level of indirection for every merge\ncommit. (Octopus merges suffer from this indirection, but they are\nvery rare.)\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph-format.txt | 91 +++++++++++++++++++++++++\n 1 file changed, 91 insertions(+)\n create mode 100644 Documentation/technical/commit-graph-format.txt\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nnew file mode 100644\nindex 0000000000..349fa0c14c\n--- /dev/null\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -0,0 +1,91 @@\n+Git commit graph format\n+=======================\n+\n+The Git commit graph stores a list of commit OIDs and some associated\n+metadata, including:\n+\n+- The generation number of the commit. Commits with no parents have\n+ generation number 1; commits with parents have generation number\n+ one more than the maximum generation number of its parents. We\n+ reserve zero as special, and can be used to mark a generation\n+ number invalid or as \"not computed\".\n+\n+- The root tree OID.\n+\n+- The commit date.\n+\n+- The parents of the commit, stored using positional references within\n+ the graph file.\n+\n+== graph-*.graph files have the following format:\n+\n+In order to allow extensions that add extra data to the graph, we organize\n+the body into \"chunks\" and provide a binary lookup table at the beginning\n+of the body. The header includes certain values, such as number of chunks,\n+hash lengths and types.\n+\n+All 4-byte numbers are in network order.\n+\n+HEADER:\n+\n+ 4-byte signature:\n+ The signature is: {'C', 'G', 'P', 'H'}\n+\n+ 1-byte version number:\n+ Currently, the only valid version is 1.\n+\n+ 1-byte Object Id Version (1 = SHA-1)\n+\n+ 1-byte Object Id Length (H)\n+\n+ 1-byte number (C) of \"chunks\"\n+\n+CHUNK LOOKUP:\n+\n+ (C + 1) * 12 bytes listing the table of contents for the chunks:\n+ First 4 bytes describe chunk id. Value 0 is a terminating label.\n+ Other 8 bytes provide offset in current file for chunk to start.\n+ (Chunks are ordered contiguously in the file, so you can infer\n+ the length using the next chunk position if necessary.)\n+\n+ The remaining data in the body is described one chunk at a time, and\n+ these chunks may be given in any order. Chunks are required unless\n+ otherwise specified.\n+\n+CHUNK DATA:\n+\n+ OID Fanout (ID: {'O', 'I', 'D', 'F'}) (256 * 4 bytes)\n+ The ith entry, F[i], stores the number of OIDs with first\n+ byte at most i. Thus F[255] stores the total\n+ number of commits (N).\n+\n+ OID Lookup (ID: {'O', 'I', 'D', 'L'}) (N * H bytes)\n+ The OIDs for all commits in the graph, sorted in ascending order.\n+\n+ Commit Data (ID: {'C', 'G', 'E', 'T' }) (N * (H + 16) bytes)\n+ * The first H bytes are for the OID of the root tree.\n+ * The next 8 bytes are for the int-ids of the first two parents\n+ of the ith commit. Stores value 0xffffffff if no parent in that\n+ position. If there are more than two parents, the second value\n+ has its most-significant bit on and the other bits store an array\n+ position into the Large Edge List chunk.\n+ * The next 8 bytes store the generation number of the commit and\n+ the commit time in seconds since EPOCH. The generation number\n+ uses the higher 30 bits of the first 4 bytes, while the commit\n+ time uses the 32 bits of the second 4 bytes, along with the lowest\n+ 2 bits of the lowest byte, storing the 33rd and 34th bit of the\n+ commit time.\n+\n+ Large Edge List (ID: {'E', 'D', 'G', 'E'})\n+ This list of 4-byte values store the second through nth parents for\n+ all octopus merges. The second parent value in the commit data stores\n+ an array position within this list along with the most-significant bit\n+ on. Starting at that array position, iterate through this list of int-ids\n+ for the parents until reaching a value with the most-significant bit on.\n+ The other bits correspond to the int-id of the last parent. This chunk\n+ should always be present, but may be empty.\n+\n+TRAILER:\n+\n+\tH-byte HASH-checksum of all of the above.\n+\n-- \n2.15.1.45.g9b7079f\n\n"},{"id":"338759","messageId":"1518122258-157281-4-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1518122258-157281-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v3 03/14] commit-graph: create git-commit-graph builtin","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-08T20:37:27Z","receivedAt":"2018-02-08T20:38:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach git the 'commit-graph' builtin that will be used for writing and\nreading packed graph files. The current implementation is mostly\nempty, except for a '--pack-dir' option.\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-commit-graph.txt | 11 +++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/commit-graph.c | 37 +++++++++++++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n 7 files changed, 53 insertions(+)\n create mode 100644 Documentation/git-commit-graph.txt\n create mode 100644 builtin/commit-graph.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 833ef3b0b7..e82f90184d 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -34,6 +34,7 @@\n /git-clone\n /git-column\n /git-commit\n+/git-commit-graph\n /git-commit-tree\n /git-config\n /git-count-objects\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nnew file mode 100644\nindex 0000000000..e1c3078ca1\n--- /dev/null\n+++ b/Documentation/git-commit-graph.txt\n@@ -0,0 +1,11 @@\n+git-commit-graph(1)\n+===================\n+\n+NAME\n+----\n+git-commit-graph - Write and verify Git commit graphs (.graph files)\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\n+\ndiff --git a/Makefile b/Makefile\nindex ee9d5eb11e..fc40b816dc 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -932,6 +932,7 @@ BUILTIN_OBJS += builtin/clone.o\n BUILTIN_OBJS += builtin/column.o\n BUILTIN_OBJS += builtin/commit-tree.o\n BUILTIN_OBJS += builtin/commit.o\n+BUILTIN_OBJS += builtin/commit-graph.o\n BUILTIN_OBJS += builtin/config.o\n BUILTIN_OBJS += builtin/count-objects.o\n BUILTIN_OBJS += builtin/credential.o\ndiff --git a/builtin.h b/builtin.h\nindex 42378f3aa4..079855b6d4 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -149,6 +149,7 @@ extern int cmd_clone(int argc, const char **argv, const char *prefix);\n extern int cmd_clean(int argc, const char **argv, const char *prefix);\n extern int cmd_column(int argc, const char **argv, const char *prefix);\n extern int cmd_commit(int argc, const char **argv, const char *prefix);\n+extern int cmd_commit_graph(int argc, const char **argv, const char *prefix);\n extern int cmd_commit_tree(int argc, const char **argv, const char *prefix);\n extern int cmd_config(int argc, const char **argv, const char *prefix);\n extern int cmd_count_objects(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nnew file mode 100644\nindex 0000000000..a01c5d9981\n--- /dev/null\n+++ b/builtin/commit-graph.c\n@@ -0,0 +1,37 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"parse-options.h\"\n+\n+static char const * const builtin_commit_graph_usage[] = {\n+\tN_(\"git commit-graph [--pack-dir ]\"),\n+\tNULL\n+};\n+\n+static struct opts_commit_graph {\n+\tconst char *pack_dir;\n+} opts;\n+\n+\n+int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic struct option builtin_commit_graph_options[] = {\n+\t\t{ OPTION_STRING, 'p', \"pack-dir\", &opts.pack_dir,\n+\t\t\tN_(\"dir\"),\n+\t\t\tN_(\"The pack directory to store the graph\") },\n+\t\tOPT_END(),\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_commit_graph_usage,\n+\t\t\t\t builtin_commit_graph_options);\n+\n+\tgit_config(git_default_config, NULL);\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_commit_graph_options,\n+\t\t\t builtin_commit_graph_usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n+\n+\tusage_with_options(builtin_commit_graph_usage,\n+\t\t\t builtin_commit_graph_options);\n+}\n+\ndiff --git a/command-list.txt b/command-list.txt\nindex a1fad28fd8..835c5890be 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -34,6 +34,7 @@ git-clean mainporcelain\n git-clone mainporcelain init\n git-column purehelpers\n git-commit mainporcelain history\n+git-commit-graph plumbingmanipulators\n git-commit-tree plumbingmanipulators\n git-config ancillarymanipulators\n git-count-objects ancillaryinterrogators\ndiff --git a/git.c b/git.c\nindex 9e96dd4090..d4832c1e0d 100644\n--- a/git.c\n+++ b/git.c\n@@ -388,6 +388,7 @@ static struct cmd_struct commands[] = {\n \t{ \"clone\", cmd_clone },\n \t{ \"column\", cmd_column, RUN_SETUP_GENTLY },\n \t{ \"commit\", cmd_commit, RUN_SETUP | NEED_WORK_TREE },\n+\t{ \"commit-graph\", cmd_commit_graph, RUN_SETUP },\n \t{ \"commit-tree\", cmd_commit_tree, RUN_SETUP },\n \t{ \"config\", cmd_config, RUN_SETUP_GENTLY },\n \t{ \"count-objects\", cmd_count_objects, RUN_SETUP },\n-- \n2.15.1.45.g9b7079f\n\n"},{"id":"338760","messageId":"1518122258-157281-8-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1518122258-157281-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v3 07/14] commit-graph: update graph-head during write","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-08T20:37:31Z","receivedAt":"2018-02-08T20:38:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"It is possible to have multiple commit graph files in a pack directory,\nbut only one is important at a time. Use a 'graph_head' file to point\nto the important file. Teach git-commit-graph to write 'graph_head' upon\nwriting a new commit graph file.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 11 ++++++++++-\n builtin/commit-graph.c | 27 +++++++++++++++++++++++++--\n commit-graph.c | 8 ++++++++\n commit-graph.h | 1 +\n t/t5318-commit-graph.sh | 25 +++++++++++++++++++------\n 5 files changed, 63 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 67e107f06a..5e32c43b27 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -33,7 +33,9 @@ COMMANDS\n Write a commit graph file based on the commits found in packfiles.\n Includes all commits from the existing commit graph file. Outputs the\n checksum hash of the written file.\n-\n++\n+With `--update-head` option, update the graph-head file to point\n+to the written graph file.\n \n 'read'::\n \n@@ -53,6 +55,13 @@ EXAMPLES\n $ git commit-graph write\n ------------------------------------------------\n \n+* Write a graph file for the packed commits in your local .git folder\n+* and update graph-head.\n++\n+------------------------------------------------\n+$ git commit-graph write --update-head\n+------------------------------------------------\n+\n * Read basic information from a graph file.\n +\n ------------------------------------------------\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 3ffa7ec433..776ca087e8 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -1,12 +1,13 @@\n #include \"builtin.h\"\n #include \"config.h\"\n+#include \"lockfile.h\"\n #include \"parse-options.h\"\n #include \"commit-graph.h\"\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--pack-dir ]\"),\n \tN_(\"git commit-graph read [--graph-hash=]\"),\n-\tN_(\"git commit-graph write [--pack-dir ]\"),\n+\tN_(\"git commit-graph write [--pack-dir ] [--update-head]\"),\n \tNULL\n };\n \n@@ -16,13 +17,14 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--pack-dir ]\"),\n+\tN_(\"git commit-graph write [--pack-dir ] [--update-head]\"),\n \tNULL\n };\n \n static struct opts_commit_graph {\n \tconst char *pack_dir;\n \tconst char *graph_hash;\n+\tint update_head;\n } opts;\n \n static int graph_read(int argc, const char **argv)\n@@ -87,6 +89,22 @@ static int graph_read(int argc, const char **argv)\n \treturn 0;\n }\n \n+static void update_head_file(const char *pack_dir, const struct object_id *graph_hash)\n+{\n+\tint fd;\n+\tstruct lock_file lk = LOCK_INIT;\n+\tchar *head_fname = get_graph_head_filename(pack_dir);\n+\n+\tfd = hold_lock_file_for_update(&lk, head_fname, LOCK_DIE_ON_ERROR);\n+\tFREE_AND_NULL(head_fname);\n+\n+\tif (fd < 0)\n+\t\tdie_errno(\"unable to open graph-head\");\n+\n+\twrite_in_full(fd, oid_to_hex(graph_hash), GIT_MAX_HEXSZ);\n+\tcommit_lock_file(&lk);\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tstruct object_id *graph_hash;\n@@ -95,6 +113,8 @@ static int graph_write(int argc, const char **argv)\n \t\t{ OPTION_STRING, 'p', \"pack-dir\", &opts.pack_dir,\n \t\t\tN_(\"dir\"),\n \t\t\tN_(\"The pack directory to store the graph\") },\n+\t\tOPT_BOOL('u', \"update-head\", &opts.update_head,\n+\t\t\tN_(\"update graph-head to written graph file\")),\n \t\tOPT_END(),\n \t};\n \n@@ -111,6 +131,9 @@ static int graph_write(int argc, const char **argv)\n \n \tgraph_hash = write_commit_graph(opts.pack_dir);\n \n+\tif (opts.update_head)\n+\t\tupdate_head_file(opts.pack_dir, graph_hash);\n+\n \tif (graph_hash) {\n \t\tprintf(\"%s\\n\", oid_to_hex(graph_hash));\n \t\tFREE_AND_NULL(graph_hash);\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 9a337cea4d..9789fe37f9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -38,6 +38,14 @@\n #define GRAPH_MIN_SIZE (GRAPH_CHUNKLOOKUP_SIZE + GRAPH_FANOUT_SIZE + \\\n \t\t\tGRAPH_OID_LEN + 8)\n \n+char *get_graph_head_filename(const char *pack_dir)\n+{\n+\tstruct strbuf fname = STRBUF_INIT;\n+\tstrbuf_addstr(&fname, pack_dir);\n+\tstrbuf_addstr(&fname, \"/graph-head\");\n+\treturn strbuf_detach(&fname, 0);\n+}\n+\n char* get_commit_graph_filename_hash(const char *pack_dir,\n \t\t\t\t struct object_id *hash)\n {\ndiff --git a/commit-graph.h b/commit-graph.h\nindex c1608976b3..726b8aa0f4 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -4,6 +4,7 @@\n #include \"git-compat-util.h\"\n #include \"commit.h\"\n \n+extern char *get_graph_head_filename(const char *pack_dir);\n extern char* get_commit_graph_filename_hash(const char *pack_dir,\n \t\t\t\t\t struct object_id *hash);\n \ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex ad1d0e621d..21352d5a3c 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -11,7 +11,8 @@ test_expect_success 'setup full repo' '\n \tpackdir=\".git/objects/pack\"'\n \n test_expect_success 'write graph with no packs' '\n-\tgit commit-graph write --pack-dir .'\n+\tgit commit-graph write --pack-dir . &&\n+\ttest_path_is_missing graph-head'\n \n test_expect_success 'create commits and repack' '\n \tfor i in $(test_seq 3)\n@@ -32,6 +33,7 @@ graph_read_expect() {\n \n test_expect_success 'write graph' '\n \tgraph1=$(git commit-graph write) &&\n+\ttest_path_is_missing graph-head &&\n \ttest_path_is_file $packdir/graph-$graph1.graph &&\n \tgit commit-graph read --graph-hash=$graph1 >output &&\n \tgraph_read_expect \"3\" \"$packdir\" &&\n@@ -73,8 +75,11 @@ test_expect_success 'Add more commits' '\n \n \n test_expect_success 'write graph with merges' '\n-\tgraph2=$(git commit-graph write)&&\n+\tgraph2=$(git commit-graph write --update-head)&&\n \ttest_path_is_file $packdir/graph-$graph2.graph &&\n+\ttest_path_is_file $packdir/graph-head &&\n+\tprintf $graph2 >expect &&\n+\ttest_cmp expect $packdir/graph-head &&\n \tgit commit-graph read --graph-hash=$graph2 >output &&\n \tgraph_read_expect \"10\" \"$packdir\" &&\n \ttest_cmp expect output'\n@@ -99,19 +104,24 @@ test_expect_success 'Add one more commit' '\n # 1\n \n test_expect_success 'write graph with new commit' '\n-\tgraph3=$(git commit-graph write) &&\n-\ttest_path_is_file $packdir/graph-$graph3.graph &&\n+\tgraph3=$(git commit-graph write --update-head) &&\n \ttest_path_is_file $packdir/graph-$graph3.graph &&\n+ test_path_is_file $packdir/graph-head &&\n+ printf $graph3 >expect &&\n+ test_cmp expect $packdir/graph-head &&\n \tgit commit-graph read --graph-hash=$graph3 >output &&\n \tgraph_read_expect \"11\" \"$packdir\" &&\n \ttest_cmp expect output'\n \n test_expect_success 'write graph with nothing new' '\n-\tgraph4=$(git commit-graph write) &&\n+\tgraph4=$(git commit-graph write --update-head) &&\n \ttest_path_is_file $packdir/graph-$graph4.graph &&\n \tprintf $graph3 >expect &&\n \tprintf $graph4 >output &&\n \ttest_cmp expect output &&\n+ test_path_is_file $packdir/graph-head &&\n+ printf $graph4 >expect &&\n+ test_cmp expect $packdir/graph-head &&\n \tgit commit-graph read --graph-hash=$graph4 >output &&\n \tgraph_read_expect \"11\" \"$packdir\" &&\n \ttest_cmp expect output'\n@@ -123,8 +133,11 @@ test_expect_success 'setup bare repo' '\n \tbaredir=\"./objects/pack\"'\n \n test_expect_success 'write graph in bare repo' '\n-\tgraphbare=$(git commit-graph write) &&\n+\tgraphbare=$(git commit-graph write --update-head) &&\n \ttest_path_is_file $baredir/graph-$graphbare.graph &&\n+ test_path_is_file $baredir/graph-head &&\n+ printf $graphbare >expect &&\n+ test_cmp expect $baredir/graph-head &&\n \tgit commit-graph read --graph-hash=$graphbare >output &&\n \tgraph_read_expect \"11\" \"$baredir\" &&\n \ttest_cmp expect output'\n-- \n2.15.1.45.g9b7079f\n\n"},{"id":"338761","messageId":"1518122258-157281-11-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1518122258-157281-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v3 10/14] commit-graph: add core.commitGraph setting","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-08T20:37:34Z","receivedAt":"2018-02-08T20:38:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit graph feature is controlled by the new core.commitGraph config\nsetting. This defaults to 0, so the feature is opt-in.\n\nThe intention of core.commitGraph is that a user can always stop checking\nfor or parsing commit graph files if core.commitGraph=0.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 3 +++\n cache.h | 1 +\n config.c | 5 +++++\n environment.c | 1 +\n 4 files changed, 10 insertions(+)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 9593bfabaa..e90d0d1262 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -883,6 +883,9 @@ core.notesRef::\n This setting defaults to \"refs/notes/commits\", and it can be overridden by\n the `GIT_NOTES_REF` environment variable. See linkgit:git-notes[1].\n \n+core.commitGraph::\n+\tEnable git commit graph feature. Allows reading from .graph files.\n+\n core.sparseCheckout::\n \tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n \tlinkgit:git-read-tree[1] for more information.\ndiff --git a/cache.h b/cache.h\nindex 6440e2bf21..1063873316 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -771,6 +771,7 @@ extern char *git_replace_ref_base;\n \n extern int fsync_object_files;\n extern int core_preload_index;\n+extern int core_commit_graph;\n extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n extern int protect_hfs;\ndiff --git a/config.c b/config.c\nindex 41862d4a32..614cf59ac4 100644\n--- a/config.c\n+++ b/config.c\n@@ -1213,6 +1213,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.commitgraph\")) {\n+\t\tcore_commit_graph = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.sparsecheckout\")) {\n \t\tcore_apply_sparse_checkout = git_config_bool(var, value);\n \t\treturn 0;\ndiff --git a/environment.c b/environment.c\nindex 8289c25b44..81fed83c50 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -60,6 +60,7 @@ enum push_default_type push_default = PUSH_DEFAULT_UNSPECIFIED;\n enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n char *notes_ref_name;\n int grafts_replace_parents = 1;\n+int core_commit_graph;\n int core_apply_sparse_checkout;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n-- \n2.15.1.45.g9b7079f\n\n"},{"id":"338762","messageId":"1518122258-157281-13-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1518122258-157281-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v3 12/14] commit-graph: close under reachability","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-08T20:37:36Z","receivedAt":"2018-02-08T20:38:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach write_commit_graph() to walk all parents from the commits\ndiscovered in packfiles. This prevents gaps given by loose objects or\npreviously-missed packfiles.\n\nAlso automatically add commits from the existing graph file, if it\nexists.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 37 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 37 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex aff67c458e..d711a2cd81 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -633,6 +633,28 @@ static int if_packed_commit_add_to_list(const struct object_id *oid,\n \treturn 0;\n }\n \n+static void close_reachable(struct packed_oid_list *oids)\n+{\n+\tint i;\n+\tstruct rev_info revs;\n+\tstruct commit *commit;\n+\tinit_revisions(&revs, NULL);\n+\tfor (i = 0; i < oids->nr; i++) {\n+\t\tcommit = lookup_commit(oids->list[i]);\n+\t\tif (commit && !parse_commit(commit))\n+\t\t\trevs.commits = commit_list_insert(commit, &revs.commits);\n+\t}\n+\n+\tif (prepare_revision_walk(&revs))\n+\t\tdie(_(\"revision walk setup failed\"));\n+\n+\twhile ((commit = get_revision(&revs)) != NULL) {\n+\t\tALLOC_GROW(oids->list, oids->nr + 1, oids->alloc);\n+\t\toids->list[oids->nr] = &(commit->object.oid);\n+\t\t(oids->nr)++;\n+\t}\n+}\n+\n struct object_id *write_commit_graph(const char *pack_dir)\n {\n \tstruct packed_oid_list oids;\n@@ -650,12 +672,27 @@ struct object_id *write_commit_graph(const char *pack_dir)\n \tchar *fname;\n \tstruct commit_list *parent;\n \n+\tprepare_commit_graph();\n+\n \toids.nr = 0;\n \toids.alloc = 1024;\n+\n+\tif (commit_graph && oids.alloc < commit_graph->num_commits)\n+\t\toids.alloc = commit_graph->num_commits;\n+\n \tALLOC_ARRAY(oids.list, oids.alloc);\n \n+\tif (commit_graph) {\n+\t\tfor (i = 0; i < commit_graph->num_commits; i++) {\n+\t\t\toids.list[i] = malloc(sizeof(struct object_id));\n+\t\t\tget_nth_commit_oid(commit_graph, i, oids.list[i]);\n+\t\t}\n+\t\toids.nr = commit_graph->num_commits;\n+\t}\n+\n \tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n \n+\tclose_reachable(&oids);\n \tQSORT(oids.list, oids.nr, commit_compare);\n \n \tcount_distinct = 1;\n-- \n2.15.1.45.g9b7079f\n\n"},{"id":"338763","messageId":"1518122258-157281-12-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1518122258-157281-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v3 11/14] commit: integrate commit graph with commit parsing","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-08T20:37:35Z","receivedAt":"2018-02-08T20:38:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach Git to inspect a commit graph file to supply the contents of a\nstruct commit when calling parse_commit_gently(). This implementation\nsatisfies all post-conditions on the struct commit, including loading\nparents, the root tree, and the commit date. The only loosely-expected\ncondition is that the commit buffer is loaded into the cache. This\nwas checked in log-tree.c:show_log(), but the \"return;\" on failure\nproduced unexpected results (i.e. the message line was never terminated).\nThe new behavior of loading the buffer when needed prevents the\nunexpected behavior.\n\nIf core.commitGraph is false, then do not check graph files.\n\nIn test script t5318-commit-graph.sh, add output-matching conditions on\nread-only graph operations.\n\nBy loading commits from the graph instead of parsing commit buffers, we\nsave a lot of time on long commit walks. Here are some performance\nresults for a copy of the Linux repository where 'master' has 704,766\nreachable commits and is behind 'origin/master' by 19,610 commits.\n\n| Command | Before | After | Rel % |\n|----------------------------------|--------|--------|-------|\n| log --oneline --topo-order -1000 | 5.9s | 0.7s | -88% |\n| branch -vv | 0.42s | 0.27s | -35% |\n| rev-list --all | 6.4s | 1.0s | -84% |\n| rev-list --all --objects | 32.6s | 27.6s | -15% |\n\nSigned-off-by: Derrick Stolee \n---\n alloc.c | 1 +\n commit-graph.c | 202 ++++++++++++++++++++++++++++++++++++++++++++++++\n commit-graph.h | 21 ++++-\n commit.c | 3 +\n commit.h | 3 +\n log-tree.c | 3 +-\n t/t5318-commit-graph.sh | 44 ++++++++++-\n 7 files changed, 272 insertions(+), 5 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 12afadfacd..cf4f8b61e1 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -93,6 +93,7 @@ void *alloc_commit_node(void)\n \tstruct commit *c = alloc_node(&commit_state, sizeof(struct commit));\n \tc->object.type = OBJ_COMMIT;\n \tc->index = alloc_commit_index();\n+\tc->graph_pos = COMMIT_NOT_FROM_GRAPH;\n \treturn c;\n }\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 95b813c2c7..aff67c458e 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -38,6 +38,9 @@\n #define GRAPH_MIN_SIZE (GRAPH_CHUNKLOOKUP_SIZE + GRAPH_FANOUT_SIZE + \\\n \t\t\tGRAPH_OID_LEN + 8)\n \n+/* global storage */\n+struct commit_graph *commit_graph = NULL;\n+\n char *get_graph_head_filename(const char *pack_dir)\n {\n \tstruct strbuf fname = STRBUF_INIT;\n@@ -229,6 +232,205 @@ struct commit_graph *load_commit_graph_one(const char *graph_file, const char *p\n \treturn graph;\n }\n \n+static void prepare_commit_graph_one(const char *obj_dir)\n+{\n+\tchar *graph_file;\n+\tstruct object_id oid;\n+\tstruct strbuf pack_dir = STRBUF_INIT;\n+\tstrbuf_addstr(&pack_dir, obj_dir);\n+\tstrbuf_add(&pack_dir, \"/pack\", 5);\n+\n+\tif (!get_graph_head_hash(pack_dir.buf, &oid))\n+\t\treturn;\n+\n+\tgraph_file = get_commit_graph_filename_hash(pack_dir.buf, &oid);\n+\n+\tcommit_graph = load_commit_graph_one(graph_file, pack_dir.buf);\n+\tstrbuf_release(&pack_dir);\n+}\n+\n+static int prepare_commit_graph_run_once = 0;\n+void prepare_commit_graph(void)\n+{\n+\tstruct alternate_object_database *alt;\n+\tchar *obj_dir;\n+\n+\tif (prepare_commit_graph_run_once)\n+\t\treturn;\n+\tprepare_commit_graph_run_once = 1;\n+\n+\tobj_dir = get_object_directory();\n+\tprepare_commit_graph_one(obj_dir);\n+\tprepare_alt_odb();\n+\tfor (alt = alt_odb_list; !commit_graph && alt; alt = alt->next)\n+\t\tprepare_commit_graph_one(alt->path);\n+}\n+\n+static int bsearch_graph(struct commit_graph *g, struct object_id *oid, uint32_t *pos)\n+{\n+\tint result = bsearch_hash(oid->hash, g->chunk_oid_fanout,\n+\t\t\t\t g->chunk_oid_lookup, g->hash_len);\n+\n+\tif (result >= 0) {\n+\t\t*pos = result;\n+\t\treturn 1;\n+\t} else {\n+\t\t*pos = -result - 1;\n+\t\treturn 0;\n+\t}\n+}\n+\n+struct object_id *get_nth_commit_oid(struct commit_graph *g,\n+\t\t\t\t uint32_t n,\n+\t\t\t\t struct object_id *oid)\n+{\n+\thashcpy(oid->hash, g->chunk_oid_lookup + g->hash_len * n);\n+\treturn oid;\n+}\n+\n+static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n+\t\t\t\t\t int pos,\n+\t\t\t\t\t struct commit_list **pptr)\n+{\n+\tstruct commit *c;\n+\tstruct object_id oid;\n+\tget_nth_commit_oid(g, pos, &oid);\n+\tc = lookup_commit(&oid);\n+\tif (!c)\n+\t\tdie(\"could not find commit %s\", oid_to_hex(&oid));\n+\tc->graph_pos = pos;\n+\treturn &commit_list_insert(c, pptr)->next;\n+}\n+\n+static int check_commit_parents(struct commit *item, struct commit_graph *g,\n+\t\t\t\tuint32_t pos, const unsigned char *commit_data)\n+{\n+\tuint32_t new_parent_pos;\n+\tuint32_t *parent_data_ptr;\n+\n+\tnew_parent_pos = ntohl(*(uint32_t*)(commit_data + g->hash_len));\n+\n+\tif (new_parent_pos == GRAPH_PARENT_MISSING)\n+\t\treturn 0;\n+\n+\tif (new_parent_pos == GRAPH_PARENT_NONE)\n+\t\treturn 1;\n+\n+\tnew_parent_pos = ntohl(*(uint32_t*)(commit_data + g->hash_len + 4));\n+\n+\tif (new_parent_pos == GRAPH_PARENT_MISSING)\n+\t\treturn 0;\n+\tif (!(new_parent_pos & GRAPH_LARGE_EDGES_NEEDED))\n+\t\treturn 1;\n+\n+\tnew_parent_pos = new_parent_pos ^ GRAPH_LARGE_EDGES_NEEDED;\n+\n+\tif (new_parent_pos == GRAPH_PARENT_MISSING)\n+\t\treturn 0;\n+\n+\tparent_data_ptr = (uint32_t*)(g->chunk_large_edges + 4 * new_parent_pos);\n+\tdo {\n+\t\tnew_parent_pos = ntohl(*parent_data_ptr);\n+\n+\t\tif ((new_parent_pos & GRAPH_EDGE_LAST_MASK) == GRAPH_PARENT_MISSING)\n+\t\t\treturn 0;\n+\n+\t\tparent_data_ptr++;\n+\t} while (!(new_parent_pos & GRAPH_LAST_EDGE));\n+\n+\treturn 1;\n+}\n+\n+static int full_parse_commit(struct commit *item, struct commit_graph *g,\n+\t\t\t uint32_t pos, const unsigned char *commit_data)\n+{\n+\tstruct object_id oid;\n+\tuint32_t new_parent_pos;\n+\tuint32_t *parent_data_ptr;\n+\tuint64_t date_low, date_high;\n+\tstruct commit_list **pptr;\n+\n+\titem->object.parsed = 1;\n+\titem->graph_pos = pos;\n+\n+\thashcpy(oid.hash, commit_data);\n+\titem->tree = lookup_tree(&oid);\n+\n+\tdate_high = ntohl(*(uint32_t*)(commit_data + g->hash_len + 8)) & 0x3;\n+\tdate_low = ntohl(*(uint32_t*)(commit_data + g->hash_len + 12));\n+\titem->date = (timestamp_t)((date_high << 32) | date_low);\n+\n+\tpptr = &item->parents;\n+\n+\tnew_parent_pos = ntohl(*(uint32_t*)(commit_data + g->hash_len));\n+\tif (new_parent_pos == GRAPH_PARENT_NONE)\n+\t\treturn 1;\n+\tpptr = insert_parent_or_die(g, new_parent_pos, pptr);\n+\n+\tnew_parent_pos = ntohl(*(uint32_t*)(commit_data + g->hash_len + 4));\n+\tif (new_parent_pos == GRAPH_PARENT_NONE)\n+\t\treturn 1;\n+\tif (!(new_parent_pos & GRAPH_LARGE_EDGES_NEEDED)) {\n+\t\tpptr = insert_parent_or_die(g, new_parent_pos, pptr);\n+\t\treturn 1;\n+\t}\n+\n+\tparent_data_ptr = (uint32_t*)(g->chunk_large_edges +\n+\t\t\t 4 * (new_parent_pos ^ GRAPH_LARGE_EDGES_NEEDED));\n+\tdo {\n+\t\tnew_parent_pos = ntohl(*parent_data_ptr);\n+\t\tpptr = insert_parent_or_die(g, new_parent_pos, pptr);\n+\t\tparent_data_ptr++;\n+\t} while (!(new_parent_pos & GRAPH_LAST_EDGE));\n+\n+\treturn 1;\n+}\n+\n+static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t pos)\n+{\n+\tconst unsigned char *commit_data = g->chunk_commit_data + (g->hash_len + 16) * pos;\n+\n+\tif (!check_commit_parents(item, g, pos, commit_data))\n+\t\treturn 0;\n+\n+\treturn full_parse_commit(item, g, pos, commit_data);\n+}\n+\n+/*\n+ * Given a commit struct, try to fill the commit struct info, including:\n+ * 1. tree object\n+ * 2. date\n+ * 3. parents.\n+ *\n+ * Returns 1 if and only if the commit was found in the commit graph.\n+ *\n+ * See parse_commit_buffer() for the fallback after this call.\n+ */\n+int parse_commit_in_graph(struct commit *item)\n+{\n+\tif (!core_commit_graph)\n+\t\treturn 0;\n+\tif (item->object.parsed)\n+\t\treturn 1;\n+\n+\tprepare_commit_graph();\n+\tif (commit_graph) {\n+\t\tuint32_t pos;\n+\t\tint found;\n+\t\tif (item->graph_pos != COMMIT_NOT_FROM_GRAPH) {\n+\t\t\tpos = item->graph_pos;\n+\t\t\tfound = 1;\n+\t\t} else {\n+\t\t\tfound = bsearch_graph(commit_graph, &(item->object.oid), &pos);\n+\t\t}\n+\n+\t\tif (found)\n+\t\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n+\t}\n+\n+\treturn 0;\n+}\n+\n static void write_graph_chunk_fanout(struct sha1file *f,\n \t\t\t\t struct commit **commits,\n \t\t\t\t int nr_commits)\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 75427cd5f6..7c4c9c38ab 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -4,13 +4,25 @@\n #include \"git-compat-util.h\"\n #include \"commit.h\"\n \n+/*\n+ * Given a commit struct, try to fill the commit struct info, including:\n+ * 1. tree object\n+ * 2. date\n+ * 3. parents.\n+ *\n+ * Returns 1 if and only if the commit was found in the packed graph.\n+ *\n+ * See parse_commit_buffer() for the fallback after this call.\n+ */\n+extern int parse_commit_in_graph(struct commit *item);\n+\n extern char *get_graph_head_filename(const char *pack_dir);\n extern struct object_id *get_graph_head_hash(const char *pack_dir,\n \t\t\t\t\t struct object_id *hash);\n extern char* get_commit_graph_filename_hash(const char *pack_dir,\n \t\t\t\t\t struct object_id *hash);\n \n-struct commit_graph {\n+extern struct commit_graph {\n \tint graph_fd;\n \n \tconst unsigned char *data;\n@@ -28,10 +40,15 @@ struct commit_graph {\n \n \t/* something like \".git/objects/pack\" */\n \tchar pack_dir[FLEX_ARRAY]; /* more */\n-};\n+} *commit_graph;\n \n extern struct commit_graph *load_commit_graph_one(const char *graph_file,\n \t\t\t\t\t\t const char *pack_dir);\n+extern void prepare_commit_graph(void);\n+\n+extern struct object_id *get_nth_commit_oid(struct commit_graph *g,\n+\t\t\t\t\t uint32_t n,\n+\t\t\t\t\t struct object_id *oid);\n \n extern struct object_id *write_commit_graph(const char *pack_dir);\n \ndiff --git a/commit.c b/commit.c\nindex cab8d4455b..a8b464d901 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"tag.h\"\n #include \"commit.h\"\n+#include \"commit-graph.h\"\n #include \"pkt-line.h\"\n #include \"utf8.h\"\n #include \"diff.h\"\n@@ -385,6 +386,8 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \t\treturn -1;\n \tif (item->object.parsed)\n \t\treturn 0;\n+\tif (parse_commit_in_graph(item))\n+\t\treturn 0;\n \tbuffer = read_sha1_file(item->object.oid.hash, &type, &size);\n \tif (!buffer)\n \t\treturn quiet_on_missing ? -1 :\ndiff --git a/commit.h b/commit.h\nindex 99a3fea68d..57963d86c3 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -8,6 +8,8 @@\n #include \"gpg-interface.h\"\n #include \"string-list.h\"\n \n+#define COMMIT_NOT_FROM_GRAPH 0xFFFFFFFF\n+\n struct commit_list {\n \tstruct commit *item;\n \tstruct commit_list *next;\n@@ -20,6 +22,7 @@ struct commit {\n \ttimestamp_t date;\n \tstruct commit_list *parents;\n \tstruct tree *tree;\n+\tuint32_t graph_pos;\n };\n \n extern int save_commit_buffer;\ndiff --git a/log-tree.c b/log-tree.c\nindex 580b3a98a0..14735d412b 100644\n--- a/log-tree.c\n+++ b/log-tree.c\n@@ -647,8 +647,7 @@ void show_log(struct rev_info *opt)\n \t\tshow_mergetag(opt, commit);\n \t}\n \n-\tif (!get_cached_commit_buffer(commit, NULL))\n-\t\treturn;\n+\tget_commit_buffer(commit, NULL);\n \n \tif (opt->show_notes) {\n \t\tint raw;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 10dfb6c5cf..1e3fe59d70 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -8,6 +8,7 @@ test_expect_success 'setup full repo' '\n \tmkdir full &&\n \tcd full &&\n \tgit init &&\n+\tgit config core.commitGraph true &&\n \tpackdir=\".git/objects/pack\"'\n \n test_expect_success 'write graph with no packs' '\n@@ -22,6 +23,26 @@ test_expect_success 'create commits and repack' '\n \tdone &&\n \tgit repack'\n \n+graph_git_two_modes() {\n+\tgit -c core.graph=true $1 >output\n+\tgit -c core.graph=false $1 >expect\n+\ttest_cmp output expect\n+}\n+\n+graph_git_behavior() {\n+\tMSG=$1\n+\tBRANCH=$2\n+\tCOMPARE=$3\n+\ttest_expect_success \"check normal git operations: $MSG\" '\n+\t\tgraph_git_two_modes \"log --oneline $BRANCH\" &&\n+\t\tgraph_git_two_modes \"log --topo-order $BRANCH\" &&\n+\t\tgraph_git_two_modes \"log --graph $COMPARE..$BRANCH\" &&\n+\t\tgraph_git_two_modes \"branch -vv\" &&\n+\t\tgraph_git_two_modes \"merge-base -a $BRANCH $COMPARE\"'\n+}\n+\n+graph_git_behavior 'no graph' commits/3 commits/1\n+\n graph_read_expect() {\n \tcat >expect <<- EOF\n \theader: 43475048 01 01 14 04\n@@ -39,6 +60,8 @@ test_expect_success 'write graph' '\n \tgraph_read_expect \"3\" \"$packdir\" &&\n \ttest_cmp expect output'\n \n+graph_git_behavior 'graph exists, no head' commits/3 commits/1\n+\n test_expect_success 'Add more commits' '\n \tgit reset --hard commits/1 &&\n \tfor i in $(test_seq 4 5)\n@@ -73,7 +96,6 @@ test_expect_success 'Add more commits' '\n # |___/____/\n # 1\n \n-\n test_expect_success 'write graph with merges' '\n \tgraph2=$(git commit-graph write --update-head)&&\n \ttest_path_is_file $packdir/graph-$graph2.graph &&\n@@ -84,6 +106,10 @@ test_expect_success 'write graph with merges' '\n \tgraph_read_expect \"10\" \"$packdir\" &&\n \ttest_cmp expect output'\n \n+graph_git_behavior 'merge 1 vs 2' merge/1 merge/2\n+graph_git_behavior 'merge 1 vs 3' merge/1 merge/3\n+graph_git_behavior 'merge 2 vs 3' merge/2 merge/3\n+\n test_expect_success 'Add one more commit' '\n \ttest_commit 8 &&\n \tgit branch commits/8 &&\n@@ -103,6 +129,9 @@ test_expect_success 'Add one more commit' '\n # |___/____/\n # 1\n \n+graph_git_behavior 'mixed mode, commit 8 vs merge 1' commits/8 merge/1\n+graph_git_behavior 'mixed mode, commit 8 vs merge 2' commits/8 merge/2\n+\n test_expect_success 'write graph with new commit' '\n \tgraph3=$(git commit-graph write --update-head --delete-expired) &&\n \ttest_path_is_file $packdir/graph-$graph3.graph &&\n@@ -115,6 +144,9 @@ test_expect_success 'write graph with new commit' '\n \tgraph_read_expect \"11\" \"$packdir\" &&\n \ttest_cmp expect output'\n \n+graph_git_behavior 'full graph, commit 8 vs merge 1' commits/8 merge/1\n+graph_git_behavior 'full graph, commit 8 vs merge 2' commits/8 merge/2\n+\n test_expect_success 'write graph with nothing new' '\n \tgraph4=$(git commit-graph write --update-head --delete-expired) &&\n \ttest_path_is_file $packdir/graph-$graph4.graph &&\n@@ -134,12 +166,19 @@ test_expect_success 'clear graph' '\n \ttest_path_is_missing $packdir/graph-$graph4.graph &&\n \ttest_path_is_missing $packdir/graph-head'\n \n+graph_git_behavior 'cleared graph, commit 8 vs merge 1' commits/8 merge/1\n+graph_git_behavior 'cleared graph, commit 8 vs merge 2' commits/8 merge/2\n+\n test_expect_success 'setup bare repo' '\n \tcd .. &&\n \tgit clone --bare --no-local full bare &&\n \tcd bare &&\n+\tgit config core.commitGraph true &&\n \tbaredir=\"./objects/pack\"'\n \n+graph_git_behavior 'bare repo, commit 8 vs merge 1' commits/8 merge/1\n+graph_git_behavior 'bare repo, commit 8 vs merge 2' commits/8 merge/2\n+\n test_expect_success 'write graph in bare repo' '\n \tgraphbare=$(git commit-graph write --update-head) &&\n \ttest_path_is_file $baredir/graph-$graphbare.graph &&\n@@ -150,5 +189,8 @@ test_expect_success 'write graph in bare repo' '\n \tgraph_read_expect \"11\" \"$baredir\" &&\n \ttest_cmp expect output'\n \n+graph_git_behavior 'bare repo with graph, commit 8 vs merge 1' commits/8 merge/1\n+graph_git_behavior 'bare repo with graph, commit 8 vs merge 2' commits/8 merge/2\n+\n test_done\n \n-- \n2.15.1.45.g9b7079f\n\n"},{"id":"338764","messageId":"1518122258-157281-6-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1518122258-157281-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v3 05/14] commit-graph: implement 'git-commit-graph write'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-08T20:37:29Z","receivedAt":"2018-02-08T20:38:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach git-commit-graph to write graph files. Create new test script to verify\nthis command succeeds without failure.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 39 +++++++++++++\n builtin/commit-graph.c | 43 +++++++++++++++\n t/t5318-commit-graph.sh | 109 +++++++++++++++++++++++++++++++++++++\n 3 files changed, 191 insertions(+)\n create mode 100755 t/t5318-commit-graph.sh\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex e1c3078ca1..55dfe5c3d8 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -5,6 +5,45 @@ NAME\n ----\n git-commit-graph - Write and verify Git commit graphs (.graph files)\n \n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git commit-graph write' [--pack-dir ]\n+\n+\n+DESCRIPTION\n+-----------\n+\n+Manage the serialized commit graph file.\n+\n+\n+OPTIONS\n+-------\n+--pack-dir::\n+\tUse given directory for the location of packfiles, graph-head,\n+\tand graph files.\n+\n+\n+COMMANDS\n+--------\n+'write'::\n+\n+Write a commit graph file based on the commits found in packfiles.\n+Includes all commits from the existing commit graph file. Outputs the\n+checksum hash of the written file.\n+\n+\n+EXAMPLES\n+--------\n+\n+* Write a commit graph file for the packed commits in your local .git folder.\n++\n+------------------------------------------------\n+$ git commit-graph write\n+------------------------------------------------\n+\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex a01c5d9981..5dac033bfe 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -1,9 +1,16 @@\n #include \"builtin.h\"\n #include \"config.h\"\n #include \"parse-options.h\"\n+#include \"commit-graph.h\"\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--pack-dir ]\"),\n+\tN_(\"git commit-graph write [--pack-dir ]\"),\n+\tNULL\n+};\n+\n+static const char * const builtin_commit_graph_write_usage[] = {\n+\tN_(\"git commit-graph write [--pack-dir ]\"),\n \tNULL\n };\n \n@@ -11,6 +18,37 @@ static struct opts_commit_graph {\n \tconst char *pack_dir;\n } opts;\n \n+static int graph_write(int argc, const char **argv)\n+{\n+\tstruct object_id *graph_hash;\n+\n+\tstatic struct option builtin_commit_graph_write_options[] = {\n+\t\t{ OPTION_STRING, 'p', \"pack-dir\", &opts.pack_dir,\n+\t\t\tN_(\"dir\"),\n+\t\t\tN_(\"The pack directory to store the graph\") },\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t builtin_commit_graph_write_options,\n+\t\t\t builtin_commit_graph_write_usage, 0);\n+\n+\tif (!opts.pack_dir) {\n+\t\tstruct strbuf path = STRBUF_INIT;\n+\t\tstrbuf_addstr(&path, get_object_directory());\n+\t\tstrbuf_addstr(&path, \"/pack\");\n+\t\topts.pack_dir = strbuf_detach(&path, NULL);\n+\t}\n+\n+\tgraph_hash = write_commit_graph(opts.pack_dir);\n+\n+\tif (graph_hash) {\n+\t\tprintf(\"%s\\n\", oid_to_hex(graph_hash));\n+\t\tFREE_AND_NULL(graph_hash);\n+\t}\n+\n+\treturn 0;\n+}\n \n int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n {\n@@ -31,6 +69,11 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\t\t builtin_commit_graph_usage,\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n+\tif (argc > 0) {\n+\t\tif (!strcmp(argv[0], \"write\"))\n+\t\t\treturn graph_write(argc, argv);\n+\t}\n+\n \tusage_with_options(builtin_commit_graph_usage,\n \t\t\t builtin_commit_graph_options);\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nnew file mode 100755\nindex 0000000000..b762587595\n--- /dev/null\n+++ b/t/t5318-commit-graph.sh\n@@ -0,0 +1,109 @@\n+#!/bin/sh\n+\n+test_description='commit graph'\n+. ./test-lib.sh\n+\n+test_expect_success 'setup full repo' '\n+\trm -rf .git &&\n+\tmkdir full &&\n+\tcd full &&\n+\tgit init &&\n+\tpackdir=\".git/objects/pack\"'\n+\n+test_expect_success 'write graph with no packs' '\n+\tgit commit-graph write --pack-dir .'\n+\n+test_expect_success 'create commits and repack' '\n+\tfor i in $(test_seq 3)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i\n+\tdone &&\n+\tgit repack'\n+\n+test_expect_success 'write graph' '\n+\tgraph1=$(git commit-graph write) &&\n+\ttest_path_is_file $packdir/graph-$graph1.graph'\n+\n+test_expect_success 'Add more commits' '\n+\tgit reset --hard commits/1 &&\n+\tfor i in $(test_seq 4 5)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i\n+\tdone &&\n+\tgit reset --hard commits/2 &&\n+\tfor i in $(test_seq 6 7)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i\n+\tdone &&\n+\tgit reset --hard commits/2 &&\n+\tgit merge commits/4 &&\n+\tgit branch merge/1 &&\n+\tgit reset --hard commits/4 &&\n+\tgit merge commits/6 &&\n+\tgit branch merge/2 &&\n+\tgit reset --hard commits/3 &&\n+\tgit merge commits/5 commits/7 &&\n+\tgit branch merge/3 &&\n+\tgit repack'\n+\n+# Current graph structure:\n+#\n+# __M3___\n+# / | \\\n+# 3 M1 5 M2 7\n+# |/ \\|/ \\|\n+# 2 4 6\n+# |___/____/\n+# 1\n+\n+\n+test_expect_success 'write graph with merges' '\n+\tgraph2=$(git commit-graph write)&&\n+\ttest_path_is_file $packdir/graph-$graph2.graph'\n+\n+test_expect_success 'Add one more commit' '\n+\ttest_commit 8 &&\n+\tgit branch commits/8 &&\n+\tls $packdir | grep idx >existing-idx &&\n+\tgit repack &&\n+\tls $packdir | grep idx | grep -v --file=existing-idx >new-idx'\n+\n+# Current graph structure:\n+#\n+# 8\n+# |\n+# __M3___\n+# / | \\\n+# 3 M1 5 M2 7\n+# |/ \\|/ \\|\n+# 2 4 6\n+# |___/____/\n+# 1\n+\n+test_expect_success 'write graph with new commit' '\n+\tgraph3=$(git commit-graph write) &&\n+\ttest_path_is_file $packdir/graph-$graph3.graph'\n+\n+\n+test_expect_success 'write graph with nothing new' '\n+\tgraph4=$(git commit-graph write) &&\n+\ttest_path_is_file $packdir/graph-$graph4.graph &&\n+\tprintf $graph3 >expect &&\n+\tprintf $graph4 >output &&\n+\ttest_cmp expect output'\n+\n+test_expect_success 'setup bare repo' '\n+\tcd .. &&\n+\tgit clone --bare --no-local full bare &&\n+\tcd bare &&\n+\tbaredir=\"./objects/pack\"'\n+\n+test_expect_success 'write graph in bare repo' '\n+\tgraphbare=$(git commit-graph write) &&\n+\ttest_path_is_file $baredir/graph-$graphbare.graph'\n+\n+test_done\n+\n-- \n2.15.1.45.g9b7079f\n\n"},{"id":"338765","messageId":"1518122258-157281-9-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1518122258-157281-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v3 08/14] commit-graph: implement 'git-commit-graph clear'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-08T20:37:32Z","receivedAt":"2018-02-08T20:38:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach Git to delete the current 'graph_head' file and the commit graph\nit references. This is a good safety valve if somehow the file is\ncorrupted and needs to be recalculated. Since the commit graph is a\nsummary of contents already in the ODB, it can be regenerated.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 11 +++++++++\n builtin/commit-graph.c | 50 ++++++++++++++++++++++++++++++++++++++\n commit-graph.c | 23 ++++++++++++++++++\n commit-graph.h | 2 ++\n t/t5318-commit-graph.sh | 5 ++++\n 5 files changed, 91 insertions(+)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 5e32c43b27..8c2cbbc923 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -9,6 +9,7 @@ git-commit-graph - Write and verify Git commit graphs (.graph files)\n SYNOPSIS\n --------\n [verse]\n+'git commit-graph clear' [--pack-dir ]\n 'git commit-graph read' [--pack-dir ]\n 'git commit-graph write' [--pack-dir ]\n \n@@ -45,6 +46,10 @@ details about the graph file.\n With `--graph-hash=` option, consider the graph file\n graph-.graph in the pack directory.\n \n+'clear'::\n+\n+Delete the graph-head file and the graph file it references.\n+\n \n EXAMPLES\n --------\n@@ -68,6 +73,12 @@ $ git commit-graph write --update-head\n $ git commit-graph read --graph-hash=\n ------------------------------------------------\n \n+* Delete /graph-head and the file it references.\n++\n+------------------------------------------------\n+$ git commit-graph clear --pack-dir=\n+------------------------------------------------\n+\n \n GIT\n ---\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 776ca087e8..529cb80de6 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -1,16 +1,23 @@\n #include \"builtin.h\"\n #include \"config.h\"\n+#include \"dir.h\"\n #include \"lockfile.h\"\n #include \"parse-options.h\"\n #include \"commit-graph.h\"\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--pack-dir ]\"),\n+\tN_(\"git commit-graph clear [--pack-dir ]\"),\n \tN_(\"git commit-graph read [--graph-hash=]\"),\n \tN_(\"git commit-graph write [--pack-dir ] [--update-head]\"),\n \tNULL\n };\n \n+static const char * const builtin_commit_graph_clear_usage[] = {\n+\tN_(\"git commit-graph clear [--pack-dir ]\"),\n+\tNULL\n+};\n+\n static const char * const builtin_commit_graph_read_usage[] = {\n \tN_(\"git commit-graph read [--pack-dir ]\"),\n \tNULL\n@@ -27,6 +34,47 @@ static struct opts_commit_graph {\n \tint update_head;\n } opts;\n \n+static int graph_clear(int argc, const char **argv)\n+{\n+\tchar *old_path;\n+\tchar *head_fname;\n+\tstruct object_id old_graph_hash;\n+\n+\tstatic struct option builtin_commit_graph_clear_options[] = {\n+\t\t{ OPTION_STRING, 'p', \"pack-dir\", &opts.pack_dir,\n+\t\t\tN_(\"dir\"),\n+\t\t\tN_(\"The pack directory to store the graph\") },\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t builtin_commit_graph_clear_options,\n+\t\t\t builtin_commit_graph_clear_usage, 0);\n+\n+\tif (!opts.pack_dir) {\n+\t\tstruct strbuf path = STRBUF_INIT;\n+\t\tstrbuf_addstr(&path, get_object_directory());\n+\t\tstrbuf_addstr(&path, \"/pack\");\n+\t\topts.pack_dir = strbuf_detach(&path, NULL);\n+\t}\n+\n+\tif (!get_graph_head_hash(opts.pack_dir, &old_graph_hash))\n+\t\treturn 0;\n+\n+\thead_fname = get_graph_head_filename(opts.pack_dir);\n+\tif (remove_path(head_fname))\n+\t\tdie(\"failed to remove path %s\", head_fname);\n+\tFREE_AND_NULL(head_fname);\n+\n+\told_path = get_commit_graph_filename_hash(opts.pack_dir,\n+\t\t\t\t\t\t &old_graph_hash);\n+\tif (remove_path(old_path))\n+\t\tdie(\"failed to remove path %s\", old_path);\n+\tfree(old_path);\n+\n+\treturn 0;\n+}\n+\n static int graph_read(int argc, const char **argv)\n {\n \tstruct object_id graph_hash;\n@@ -162,6 +210,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n \tif (argc > 0) {\n+\t\tif (!strcmp(argv[0], \"clear\"))\n+\t\t\treturn graph_clear(argc, argv);\n \t\tif (!strcmp(argv[0], \"read\"))\n \t\t\treturn graph_read(argc, argv);\n \t\tif (!strcmp(argv[0], \"write\"))\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 9789fe37f9..95b813c2c7 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -46,6 +46,29 @@ char *get_graph_head_filename(const char *pack_dir)\n \treturn strbuf_detach(&fname, 0);\n }\n \n+struct object_id *get_graph_head_hash(const char *pack_dir, struct object_id *hash)\n+{\n+\tchar hex[GIT_MAX_HEXSZ + 1];\n+\tchar *fname;\n+\tFILE *f;\n+\n+\tfname = get_graph_head_filename(pack_dir);\n+\tf = fopen(fname, \"r\");\n+\tFREE_AND_NULL(fname);\n+\n+\tif (!f)\n+\t\treturn 0;\n+\n+\tif (!fgets(hex, sizeof(hex), f))\n+\t\tdie(\"failed to read graph-head\");\n+\n+\tfclose(f);\n+\n+\tif (get_oid_hex(hex, hash))\n+\t\treturn 0;\n+\treturn hash;\n+}\n+\n char* get_commit_graph_filename_hash(const char *pack_dir,\n \t\t\t\t struct object_id *hash)\n {\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 726b8aa0f4..75427cd5f6 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -5,6 +5,8 @@\n #include \"commit.h\"\n \n extern char *get_graph_head_filename(const char *pack_dir);\n+extern struct object_id *get_graph_head_hash(const char *pack_dir,\n+\t\t\t\t\t struct object_id *hash);\n extern char* get_commit_graph_filename_hash(const char *pack_dir,\n \t\t\t\t\t struct object_id *hash);\n \ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 21352d5a3c..de81253790 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -126,6 +126,11 @@ test_expect_success 'write graph with nothing new' '\n \tgraph_read_expect \"11\" \"$packdir\" &&\n \ttest_cmp expect output'\n \n+test_expect_success 'clear graph' '\n+\tgit commit-graph clear &&\n+\ttest_path_is_missing $packdir/graph-$graph4.graph &&\n+\ttest_path_is_missing $packdir/graph-head'\n+\n test_expect_success 'setup bare repo' '\n \tcd .. &&\n \tgit clone --bare --no-local full bare &&\n-- \n2.15.1.45.g9b7079f\n\n"},{"id":"338766","messageId":"1518122258-157281-10-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1518122258-157281-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v3 09/14] commit-graph: implement --delete-expired","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-08T20:37:33Z","receivedAt":"2018-02-08T20:38:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach git-commit-graph to delete the graph files in the pack directory\nthat were not referenced by 'graph_head' during this process. This cleans\nup space for the user while not causing race conditions with other running\nGit processes that may be referencing the previous graph file.\n\nTo delete old graph files, a user (or managing process) would call\n\n\tgit commit-graph write --update-head --delete-expired\n\nbut there is some responsibility that the caller must consider. Specifically,\nensure that processes that started before a previous 'commit-graph write'\ncommand have completed. Otherwise, they may have an open handle on a graph file\nthat will be deleted by the new call.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 11 ++++--\n builtin/commit-graph.c | 73 ++++++++++++++++++++++++++++++++++++--\n t/t5318-commit-graph.sh | 7 ++--\n 3 files changed, 84 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 8c2cbbc923..7ae8f7484d 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -37,6 +37,11 @@ checksum hash of the written file.\n +\n With `--update-head` option, update the graph-head file to point\n to the written graph file.\n++\n+With the `--delete-expired` option, delete the graph files in the pack\n+directory that are not referred to by the graph-head file. To avoid race\n+conditions, do not delete the file previously referred to by the\n+graph-head file if it is updated by the `--update-head` option.\n \n 'read'::\n \n@@ -60,11 +65,11 @@ EXAMPLES\n $ git commit-graph write\n ------------------------------------------------\n \n-* Write a graph file for the packed commits in your local .git folder\n-* and update graph-head.\n+* Write a graph file for the packed commits in your local .git folder,\n+* update graph-head, and delete state graph files.\n +\n ------------------------------------------------\n-$ git commit-graph write --update-head\n+$ git commit-graph write --update-head --delete-expired\n ------------------------------------------------\n \n * Read basic information from a graph file.\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 529cb80de6..15f647fd81 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -9,7 +9,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--pack-dir ]\"),\n \tN_(\"git commit-graph clear [--pack-dir ]\"),\n \tN_(\"git commit-graph read [--graph-hash=]\"),\n-\tN_(\"git commit-graph write [--pack-dir ] [--update-head]\"),\n+\tN_(\"git commit-graph write [--pack-dir ] [--update-head] [--delete-expired]\"),\n \tNULL\n };\n \n@@ -24,7 +24,7 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--pack-dir ] [--update-head]\"),\n+\tN_(\"git commit-graph write [--pack-dir ] [--update-head] [--delete-expired]\"),\n \tNULL\n };\n \n@@ -32,6 +32,7 @@ static struct opts_commit_graph {\n \tconst char *pack_dir;\n \tconst char *graph_hash;\n \tint update_head;\n+\tint delete_expired;\n } opts;\n \n static int graph_clear(int argc, const char **argv)\n@@ -153,9 +154,68 @@ static void update_head_file(const char *pack_dir, const struct object_id *graph\n \tcommit_lock_file(&lk);\n }\n \n+/*\n+ * To avoid race conditions and deleting graph files that are being\n+ * used by other processes, look inside a pack directory for all files\n+ * of the form \"graph-.graph\" that do not match the old or new\n+ * graph hashes and delete them.\n+ */\n+static void do_delete_expired(const char *pack_dir,\n+\t\t\t struct object_id *old_graph_hash,\n+\t\t\t struct object_id *new_graph_hash)\n+{\n+\tDIR *dir;\n+\tstruct dirent *de;\n+\tint dirnamelen;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tchar *old_graph_path, *new_graph_path;\n+\n+\tif (old_graph_hash)\n+\t\told_graph_path = get_commit_graph_filename_hash(pack_dir, old_graph_hash);\n+\telse\n+\t\told_graph_path = NULL;\n+\tnew_graph_path = get_commit_graph_filename_hash(pack_dir, new_graph_hash);\n+\n+\tdir = opendir(pack_dir);\n+\tif (!dir) {\n+\t\tif (errno != ENOENT)\n+\t\t\terror_errno(\"unable to open object pack directory: %s\",\n+\t\t\t\t pack_dir);\n+\t\treturn;\n+\t}\n+\n+\tstrbuf_addstr(&path, pack_dir);\n+\tstrbuf_addch(&path, '/');\n+\n+\tdirnamelen = path.len;\n+\twhile ((de = readdir(dir)) != NULL) {\n+\t\tsize_t base_len;\n+\n+\t\tif (is_dot_or_dotdot(de->d_name))\n+\t\t\tcontinue;\n+\n+\t\tstrbuf_setlen(&path, dirnamelen);\n+\t\tstrbuf_addstr(&path, de->d_name);\n+\n+\t\tbase_len = path.len;\n+\t\tif (strip_suffix_mem(path.buf, &base_len, \".graph\") &&\n+\t\t strcmp(new_graph_path, path.buf) &&\n+\t\t (!old_graph_path || strcmp(old_graph_path, path.buf)) &&\n+\t\t remove_path(path.buf))\n+\t\t\tdie(\"failed to remove path %s\", path.buf);\n+\t}\n+\n+\tstrbuf_release(&path);\n+\tif (old_graph_path)\n+\t\tFREE_AND_NULL(old_graph_path);\n+\tFREE_AND_NULL(new_graph_path);\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tstruct object_id *graph_hash;\n+\tstruct object_id old_graph_hash;\n+\tint has_existing;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\t{ OPTION_STRING, 'p', \"pack-dir\", &opts.pack_dir,\n@@ -163,6 +223,8 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"The pack directory to store the graph\") },\n \t\tOPT_BOOL('u', \"update-head\", &opts.update_head,\n \t\t\tN_(\"update graph-head to written graph file\")),\n+\t\tOPT_BOOL('d', \"delete-expired\", &opts.delete_expired,\n+\t\t\tN_(\"delete expired head graph file\")),\n \t\tOPT_END(),\n \t};\n \n@@ -177,11 +239,18 @@ static int graph_write(int argc, const char **argv)\n \t\topts.pack_dir = strbuf_detach(&path, NULL);\n \t}\n \n+\thas_existing = !!get_graph_head_hash(opts.pack_dir, &old_graph_hash);\n+\n \tgraph_hash = write_commit_graph(opts.pack_dir);\n \n \tif (opts.update_head)\n \t\tupdate_head_file(opts.pack_dir, graph_hash);\n \n+\tif (opts.delete_expired && graph_hash)\n+\t\tdo_delete_expired(opts.pack_dir,\n+\t\t\t\t has_existing ? &old_graph_hash : NULL,\n+\t\t\t\t graph_hash);\n+\n \tif (graph_hash) {\n \t\tprintf(\"%s\\n\", oid_to_hex(graph_hash));\n \t\tFREE_AND_NULL(graph_hash);\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex de81253790..10dfb6c5cf 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -104,8 +104,10 @@ test_expect_success 'Add one more commit' '\n # 1\n \n test_expect_success 'write graph with new commit' '\n-\tgraph3=$(git commit-graph write --update-head) &&\n+\tgraph3=$(git commit-graph write --update-head --delete-expired) &&\n \ttest_path_is_file $packdir/graph-$graph3.graph &&\n+\ttest_path_is_file $packdir/graph-$graph2.graph &&\n+\ttest_path_is_missing $packdir/graph-$graph1.graph &&\n test_path_is_file $packdir/graph-head &&\n printf $graph3 >expect &&\n test_cmp expect $packdir/graph-head &&\n@@ -114,8 +116,9 @@ test_expect_success 'write graph with new commit' '\n \ttest_cmp expect output'\n \n test_expect_success 'write graph with nothing new' '\n-\tgraph4=$(git commit-graph write --update-head) &&\n+\tgraph4=$(git commit-graph write --update-head --delete-expired) &&\n \ttest_path_is_file $packdir/graph-$graph4.graph &&\n+\ttest_path_is_missing $packdir/graph-$graph2.graph &&\n \tprintf $graph3 >expect &&\n \tprintf $graph4 >output &&\n \ttest_cmp expect output &&\n-- \n2.15.1.45.g9b7079f\n\n"},{"id":"338767","messageId":"1518122258-157281-14-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1518122258-157281-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v3 13/14] commit-graph: read only from specific pack-indexes","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-08T20:37:37Z","receivedAt":"2018-02-08T20:38:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach git-commit-graph to inspect the objects only in a certain list\nof pack-indexes within the given pack directory. This allows updating\nthe commit graph iteratively, since we add all commits stored in a\nprevious commit graph.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 11 +++++++++++\n builtin/commit-graph.c | 32 +++++++++++++++++++++++++++++---\n commit-graph.c | 25 +++++++++++++++++++++++--\n commit-graph.h | 4 +++-\n packfile.c | 4 ++--\n packfile.h | 2 ++\n t/t5318-commit-graph.sh | 13 +++++++++++++\n 7 files changed, 83 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 7ae8f7484d..727d5d70bb 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -42,6 +42,10 @@ With the `--delete-expired` option, delete the graph files in the pack\n directory that are not referred to by the graph-head file. To avoid race\n conditions, do not delete the file previously referred to by the\n graph-head file if it is updated by the `--update-head` option.\n++\n+With the `--stdin-packs` option, generate the new commit graph by\n+walking objects only in the specified packfiles and any commits in\n+the existing graph-head.\n \n 'read'::\n \n@@ -72,6 +76,13 @@ $ git commit-graph write\n $ git commit-graph write --update-head --delete-expired\n ------------------------------------------------\n \n+* Write a graph file, extending the current graph file using commits\n+* in , update graph-head, and delete stale graph files.\n++\n+------------------------------------------------\n+$ echo | git commit-graph write --update-head --delete-expired --stdin-packs\n+------------------------------------------------\n+\n * Read basic information from a graph file.\n +\n ------------------------------------------------\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 15f647fd81..fe5f00551c 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -9,7 +9,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--pack-dir ]\"),\n \tN_(\"git commit-graph clear [--pack-dir ]\"),\n \tN_(\"git commit-graph read [--graph-hash=]\"),\n-\tN_(\"git commit-graph write [--pack-dir ] [--update-head] [--delete-expired]\"),\n+\tN_(\"git commit-graph write [--pack-dir ] [--update-head] [--delete-expired] [--stdin-packs]\"),\n \tNULL\n };\n \n@@ -24,7 +24,7 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--pack-dir ] [--update-head] [--delete-expired]\"),\n+\tN_(\"git commit-graph write [--pack-dir ] [--update-head] [--delete-expired] [--stdin-packs]\"),\n \tNULL\n };\n \n@@ -33,6 +33,7 @@ static struct opts_commit_graph {\n \tconst char *graph_hash;\n \tint update_head;\n \tint delete_expired;\n+\tint stdin_packs;\n } opts;\n \n static int graph_clear(int argc, const char **argv)\n@@ -216,6 +217,11 @@ static int graph_write(int argc, const char **argv)\n \tstruct object_id *graph_hash;\n \tstruct object_id old_graph_hash;\n \tint has_existing;\n+\tconst char **pack_indexes = NULL;\n+\tint nr_packs = 0;\n+\tconst char **lines = NULL;\n+\tint nr_lines = 0;\n+\tint alloc_lines = 0;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\t{ OPTION_STRING, 'p', \"pack-dir\", &opts.pack_dir,\n@@ -225,6 +231,8 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"update graph-head to written graph file\")),\n \t\tOPT_BOOL('d', \"delete-expired\", &opts.delete_expired,\n \t\t\tN_(\"delete expired head graph file\")),\n+\t\tOPT_BOOL('s', \"stdin-packs\", &opts.stdin_packs,\n+\t\t\tN_(\"only scan packfiles listed by stdin\")),\n \t\tOPT_END(),\n \t};\n \n@@ -241,7 +249,25 @@ static int graph_write(int argc, const char **argv)\n \n \thas_existing = !!get_graph_head_hash(opts.pack_dir, &old_graph_hash);\n \n-\tgraph_hash = write_commit_graph(opts.pack_dir);\n+\tif (opts.stdin_packs) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tnr_lines = 0;\n+\t\talloc_lines = 128;\n+\t\tALLOC_ARRAY(lines, alloc_lines);\n+\n+\t\twhile (strbuf_getline(&buf, stdin) != EOF) {\n+\t\t\tALLOC_GROW(lines, nr_lines + 1, alloc_lines);\n+\t\t\tlines[nr_lines++] = buf.buf;\n+\t\t\tstrbuf_detach(&buf, NULL);\n+\t\t}\n+\n+\t\tpack_indexes = lines;\n+\t\tnr_packs = nr_lines;\n+\t}\n+\n+\tgraph_hash = write_commit_graph(opts.pack_dir,\n+\t\t\t\t\tpack_indexes,\n+\t\t\t\t\tnr_packs);\n \n \tif (opts.update_head)\n \t\tupdate_head_file(opts.pack_dir, graph_hash);\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d711a2cd81..27a34f5eda 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -655,7 +655,9 @@ static void close_reachable(struct packed_oid_list *oids)\n \t}\n }\n \n-struct object_id *write_commit_graph(const char *pack_dir)\n+struct object_id *write_commit_graph(const char *pack_dir,\n+\t\t\t\t const char **pack_indexes,\n+\t\t\t\t int nr_packs)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n@@ -690,7 +692,26 @@ struct object_id *write_commit_graph(const char *pack_dir)\n \t\toids.nr = commit_graph->num_commits;\n \t}\n \n-\tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n+\tif (pack_indexes) {\n+\t\tint pack_dir_len = strlen(pack_dir) + 1;\n+\t\tstruct strbuf packname = STRBUF_INIT;\n+\t\tstrbuf_add(&packname, pack_dir, pack_dir_len - 1);\n+\t\tstrbuf_addch(&packname, '/');\n+\t\tfor (i = 0; i < nr_packs; i++) {\n+\t\t\tstruct packed_git *p;\n+\t\t\tstrbuf_setlen(&packname, pack_dir_len);\n+\t\t\tstrbuf_addstr(&packname, pack_indexes[i]);\n+\t\t\tp = add_packed_git(packname.buf, packname.len, 1);\n+\t\t\tif (!p)\n+\t\t\t\tdie(\"error adding pack %s\", packname.buf);\n+\t\t\tif (open_pack_index(p))\n+\t\t\t\tdie(\"error opening index for %s\", packname.buf);\n+\t\t\tfor_each_object_in_pack(p, if_packed_commit_add_to_list, &oids);\n+\t\t\tclose_pack(p);\n+\t\t}\n+\t}\n+\telse\n+\t\tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n \n \tclose_reachable(&oids);\n \tQSORT(oids.list, oids.nr, commit_compare);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 7c4c9c38ab..918b34dd2b 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -50,7 +50,9 @@ extern struct object_id *get_nth_commit_oid(struct commit_graph *g,\n \t\t\t\t\t uint32_t n,\n \t\t\t\t\t struct object_id *oid);\n \n-extern struct object_id *write_commit_graph(const char *pack_dir);\n+extern struct object_id *write_commit_graph(const char *pack_dir,\n+\t\t\t\t\t const char **pack_indexes,\n+\t\t\t\t\t int nr_packs);\n \n #endif\n \ndiff --git a/packfile.c b/packfile.c\nindex 29f5dc2398..e4b1dc02bc 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -299,7 +299,7 @@ void close_pack_index(struct packed_git *p)\n \t}\n }\n \n-static void close_pack(struct packed_git *p)\n+void close_pack(struct packed_git *p)\n {\n \tclose_pack_windows(p);\n \tclose_pack_fd(p);\n@@ -1840,7 +1840,7 @@ int has_pack_index(const unsigned char *sha1)\n \treturn 1;\n }\n \n-static int for_each_object_in_pack(struct packed_git *p, each_packed_object_fn cb, void *data)\n+int for_each_object_in_pack(struct packed_git *p, each_packed_object_fn cb, void *data)\n {\n \tuint32_t i;\n \tint r = 0;\ndiff --git a/packfile.h b/packfile.h\nindex 0cdeb54dcd..9281e909d5 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -61,6 +61,7 @@ extern void close_pack_index(struct packed_git *);\n \n extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n extern void close_pack_windows(struct packed_git *);\n+extern void close_pack(struct packed_git *);\n extern void close_all_packs(void);\n extern void unuse_pack(struct pack_window **);\n extern void clear_delta_base_cache(void);\n@@ -133,6 +134,7 @@ typedef int each_packed_object_fn(const struct object_id *oid,\n \t\t\t\t struct packed_git *pack,\n \t\t\t\t uint32_t pos,\n \t\t\t\t void *data);\n+extern int for_each_object_in_pack(struct packed_git *p, each_packed_object_fn, void *data);\n extern int for_each_packed_object(each_packed_object_fn, void *, unsigned flags);\n \n #endif\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 1e3fe59d70..e3546e6844 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -169,6 +169,19 @@ test_expect_success 'clear graph' '\n graph_git_behavior 'cleared graph, commit 8 vs merge 1' commits/8 merge/1\n graph_git_behavior 'cleared graph, commit 8 vs merge 2' commits/8 merge/2\n \n+test_expect_success 'build graph from latest pack with closure' '\n+\tgraph5=$(cat new-idx | git commit-graph write --update-head --delete-expired --stdin-packs) &&\n+\ttest_path_is_file $packdir/graph-$graph5.graph &&\n+\ttest_path_is_file $packdir/graph-head &&\n+\tprintf $graph5 >expect &&\n+\ttest_cmp expect $packdir/graph-head &&\n+\tgit commit-graph read --graph-hash=$graph5 >output &&\n+\tgraph_read_expect \"9\" \"$packdir\" &&\n+\ttest_cmp expect output'\n+\n+graph_git_behavior 'graph from pack, commit 8 vs merge 1' commits/8 merge/1\n+graph_git_behavior 'graph from pack, commit 8 vs merge 2' commits/8 merge/2\n+\n test_expect_success 'setup bare repo' '\n \tcd .. &&\n \tgit clone --bare --no-local full bare &&\n-- \n2.15.1.45.g9b7079f\n\n"},{"id":"338768","messageId":"1518122258-157281-15-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1518122258-157281-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v3 14/14] commit-graph: build graph from starting commits","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-08T20:37:38Z","receivedAt":"2018-02-08T20:38:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach git-commit-graph to read commits from stdin when the\n--stdin-commits flag is specified. Commits reachable from these\ncommits are added to the graph. This is a much faster way to construct\nthe graph than inspecting all packed objects, but is restricted to\nknown tips.\n\nFor the Linux repository, 700,000+ commits were added to the graph\nfile starting from 'master' in 7-9 seconds, depending on the number\nof packfiles in the repo (1, 24, or 120).\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 15 ++++++++++++++-\n builtin/commit-graph.c | 26 +++++++++++++++++++++-----\n commit-graph.c | 26 ++++++++++++++++++++++++--\n commit-graph.h | 4 +++-\n t/t5318-commit-graph.sh | 19 +++++++++++++++++++\n 5 files changed, 81 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 727d5d70bb..bd1c54025a 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -45,7 +45,12 @@ graph-head file if it is updated by the `--update-head` option.\n +\n With the `--stdin-packs` option, generate the new commit graph by\n walking objects only in the specified packfiles and any commits in\n-the existing graph-head.\n+the existing graph-head. (Cannot be combined with --stdin-commits.)\n++\n+With the `--stdin-commits` option, generate the new commit graph by\n+walking commits starting at the commits specified in stdin as a list\n+of OIDs in hex, one OID per line. (Cannot be combined with\n+--stdin-packs.)\n \n 'read'::\n \n@@ -83,6 +88,14 @@ $ git commit-graph write --update-head --delete-expired\n $ echo | git commit-graph write --update-head --delete-expired --stdin-packs\n ------------------------------------------------\n \n+* Write a graph file, extending the current graph file using all\n+* commits reachable from refs/heads/*, update graph-head, and delete\n+* stale graph files.\n++\n+------------------------------------------------\n+$ git show-ref --heads -s | git commit-graph write --update-head --delete-expired --stdin-commits\n+------------------------------------------------\n+\n * Read basic information from a graph file.\n +\n ------------------------------------------------\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex fe5f00551c..28d043b5a8 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -9,7 +9,7 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--pack-dir ]\"),\n \tN_(\"git commit-graph clear [--pack-dir ]\"),\n \tN_(\"git commit-graph read [--graph-hash=]\"),\n-\tN_(\"git commit-graph write [--pack-dir ] [--update-head] [--delete-expired] [--stdin-packs]\"),\n+\tN_(\"git commit-graph write [--pack-dir ] [--update-head] [--delete-expired] [--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -24,7 +24,7 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--pack-dir ] [--update-head] [--delete-expired] [--stdin-packs]\"),\n+\tN_(\"git commit-graph write [--pack-dir ] [--update-head] [--delete-expired] [--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -34,6 +34,7 @@ static struct opts_commit_graph {\n \tint update_head;\n \tint delete_expired;\n \tint stdin_packs;\n+\tint stdin_commits;\n } opts;\n \n static int graph_clear(int argc, const char **argv)\n@@ -219,6 +220,8 @@ static int graph_write(int argc, const char **argv)\n \tint has_existing;\n \tconst char **pack_indexes = NULL;\n \tint nr_packs = 0;\n+\tconst char **commit_hex = NULL;\n+\tint nr_commits = 0;\n \tconst char **lines = NULL;\n \tint nr_lines = 0;\n \tint alloc_lines = 0;\n@@ -233,6 +236,8 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"delete expired head graph file\")),\n \t\tOPT_BOOL('s', \"stdin-packs\", &opts.stdin_packs,\n \t\t\tN_(\"only scan packfiles listed by stdin\")),\n+\t\tOPT_BOOL('C', \"stdin-commits\", &opts.stdin_commits,\n+\t\t\tN_(\"start walk at commits listed by stdin\")),\n \t\tOPT_END(),\n \t};\n \n@@ -240,6 +245,9 @@ static int graph_write(int argc, const char **argv)\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\n \n+\tif (opts.stdin_packs && opts.stdin_commits)\n+\t\tdie(_(\"cannot use both --stdin-commits and --stdin-packs\"));\n+\n \tif (!opts.pack_dir) {\n \t\tstruct strbuf path = STRBUF_INIT;\n \t\tstrbuf_addstr(&path, get_object_directory());\n@@ -261,13 +269,21 @@ static int graph_write(int argc, const char **argv)\n \t\t\tstrbuf_detach(&buf, NULL);\n \t\t}\n \n-\t\tpack_indexes = lines;\n-\t\tnr_packs = nr_lines;\n+\t\tif (opts.stdin_packs) {\n+\t\t\tpack_indexes = lines;\n+\t\t\tnr_packs = nr_lines;\n+\t\t}\n+\t\tif (opts.stdin_commits) {\n+\t\t\tcommit_hex = lines;\n+\t\t\tnr_commits = nr_lines;\n+\t\t}\n \t}\n \n \tgraph_hash = write_commit_graph(opts.pack_dir,\n \t\t\t\t\tpack_indexes,\n-\t\t\t\t\tnr_packs);\n+\t\t\t\t\tnr_packs,\n+\t\t\t\t\tcommit_hex,\n+\t\t\t\t\tnr_commits);\n \n \tif (opts.update_head)\n \t\tupdate_head_file(opts.pack_dir, graph_hash);\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 27a34f5eda..3ff3ab03ca 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -657,7 +657,9 @@ static void close_reachable(struct packed_oid_list *oids)\n \n struct object_id *write_commit_graph(const char *pack_dir,\n \t\t\t\t const char **pack_indexes,\n-\t\t\t\t int nr_packs)\n+\t\t\t\t int nr_packs,\n+\t\t\t\t const char **commit_hex,\n+\t\t\t\t int nr_commits)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n@@ -710,7 +712,27 @@ struct object_id *write_commit_graph(const char *pack_dir,\n \t\t\tclose_pack(p);\n \t\t}\n \t}\n-\telse\n+\n+\tif (commit_hex) {\n+\t\tfor (i = 0; i < nr_commits; i++) {\n+\t\t\tconst char *end;\n+\t\t\tstruct object_id oid;\n+\t\t\tstruct commit *result;\n+\n+\t\t\tif (commit_hex[i] && parse_oid_hex(commit_hex[i], &oid, &end))\n+\t\t\t\tcontinue;\n+\n+\t\t\tresult = lookup_commit_reference_gently(&oid, 1);\n+\n+\t\t\tif (result) {\n+\t\t\t\tALLOC_GROW(oids.list, oids.nr + 1, oids.alloc);\n+\t\t\t\toids.list[oids.nr] = &(result->object.oid);\n+\t\t\t\toids.nr++;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif (!pack_indexes && !commit_hex)\n \t\tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n \n \tclose_reachable(&oids);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 918b34dd2b..c412f76707 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -52,7 +52,9 @@ extern struct object_id *get_nth_commit_oid(struct commit_graph *g,\n \n extern struct object_id *write_commit_graph(const char *pack_dir,\n \t\t\t\t\t const char **pack_indexes,\n-\t\t\t\t\t int nr_packs);\n+\t\t\t\t\t int nr_packs,\n+\t\t\t\t\t const char **commit_hex,\n+\t\t\t\t\t int nr_commits);\n \n #endif\n \ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex e3546e6844..d803c12afd 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -182,6 +182,25 @@ test_expect_success 'build graph from latest pack with closure' '\n graph_git_behavior 'graph from pack, commit 8 vs merge 1' commits/8 merge/1\n graph_git_behavior 'graph from pack, commit 8 vs merge 2' commits/8 merge/2\n \n+test_expect_success 'build graph from commits with closure' '\n+\tgit tag -a -m \"merge\" tag/merge merge/3 &&\n+\tgit rev-parse tag/merge >commits-in &&\n+\tgit rev-parse commits/8 >>commits-in &&\n+\tgit rev-parse merge/1 >>commits-in &&\n+\tgit rev-parse merge/2 >>commits-in &&\n+\tgraph6=$(cat commits-in | git commit-graph write --update-head --delete-expired --stdin-commits) &&\n+\ttest_path_is_file $packdir/graph-$graph6.graph &&\n+\ttest_path_is_file $packdir/graph-$graph5.graph &&\n+\ttest_path_is_file $packdir/graph-head &&\n+\tprintf $graph6 >expect &&\n+\ttest_cmp expect $packdir/graph-head &&\n+\tgit commit-graph read --graph-hash=$graph6 >output &&\n+\tgraph_read_expect \"11\" \"$packdir\" &&\n+\ttest_cmp expect output'\n+\n+graph_git_behavior 'graph from commits, commit 8 vs merge 1' commits/8 merge/1\n+graph_git_behavior 'graph from commits, commit 8 vs merge 2' commits/8 merge/2\n+\n test_expect_success 'setup bare repo' '\n \tcd .. &&\n \tgit clone --bare --no-local full bare &&\n-- \n2.15.1.45.g9b7079f\n\n"},{"id":"338769","messageId":"1518122258-157281-7-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1518122258-157281-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v3 06/14] commit-graph: implement 'git-commit-graph read'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-08T20:37:30Z","receivedAt":"2018-02-08T20:38:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach git-commit-graph to read commit graph files and summarize their contents.\n\nUse the read subcommand to verify the contents of a commit graph file in the\ntests.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 16 ++++\n builtin/commit-graph.c | 71 ++++++++++++++++++\n commit-graph.c | 147 +++++++++++++++++++++++++++++++++++++\n commit-graph.h | 23 ++++++\n t/t5318-commit-graph.sh | 34 +++++++--\n 5 files changed, 286 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 55dfe5c3d8..67e107f06a 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -9,6 +9,7 @@ git-commit-graph - Write and verify Git commit graphs (.graph files)\n SYNOPSIS\n --------\n [verse]\n+'git commit-graph read' [--pack-dir ]\n 'git commit-graph write' [--pack-dir ]\n \n \n@@ -34,6 +35,15 @@ Includes all commits from the existing commit graph file. Outputs the\n checksum hash of the written file.\n \n \n+'read'::\n+\n+Read a graph file given by the graph-head file and output basic\n+details about the graph file.\n++\n+With `--graph-hash=` option, consider the graph file\n+graph-.graph in the pack directory.\n+\n+\n EXAMPLES\n --------\n \n@@ -43,6 +53,12 @@ EXAMPLES\n $ git commit-graph write\n ------------------------------------------------\n \n+* Read basic information from a graph file.\n++\n+------------------------------------------------\n+$ git commit-graph read --graph-hash=\n+------------------------------------------------\n+\n \n GIT\n ---\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 5dac033bfe..3ffa7ec433 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -5,10 +5,16 @@\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--pack-dir ]\"),\n+\tN_(\"git commit-graph read [--graph-hash=]\"),\n \tN_(\"git commit-graph write [--pack-dir ]\"),\n \tNULL\n };\n \n+static const char * const builtin_commit_graph_read_usage[] = {\n+\tN_(\"git commit-graph read [--pack-dir ]\"),\n+\tNULL\n+};\n+\n static const char * const builtin_commit_graph_write_usage[] = {\n \tN_(\"git commit-graph write [--pack-dir ]\"),\n \tNULL\n@@ -16,8 +22,71 @@ static const char * const builtin_commit_graph_write_usage[] = {\n \n static struct opts_commit_graph {\n \tconst char *pack_dir;\n+\tconst char *graph_hash;\n } opts;\n \n+static int graph_read(int argc, const char **argv)\n+{\n+\tstruct object_id graph_hash;\n+\tstruct commit_graph *graph = 0;\n+\tconst char *graph_file;\n+\n+\tstatic struct option builtin_commit_graph_read_options[] = {\n+\t\t{ OPTION_STRING, 'p', \"pack-dir\", &opts.pack_dir,\n+\t\t\tN_(\"dir\"),\n+\t\t\tN_(\"The pack directory to store the graph\") },\n+\t\t{ OPTION_STRING, 'H', \"graph-hash\", &opts.graph_hash,\n+\t\t\tN_(\"hash\"),\n+\t\t\tN_(\"A hash for a specific graph file in the pack-dir.\"),\n+\t\t\tPARSE_OPT_OPTARG, NULL, (intptr_t) \"\" },\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t builtin_commit_graph_read_options,\n+\t\t\t builtin_commit_graph_read_usage, 0);\n+\n+\tif (!opts.pack_dir) {\n+\t\tstruct strbuf path = STRBUF_INIT;\n+\t\tstrbuf_addstr(&path, get_object_directory());\n+\t\tstrbuf_addstr(&path, \"/pack\");\n+\t\topts.pack_dir = strbuf_detach(&path, NULL);\n+\t}\n+\n+\tif (opts.graph_hash && strlen(opts.graph_hash) == GIT_MAX_HEXSZ)\n+\t\tget_oid_hex(opts.graph_hash, &graph_hash);\n+\telse\n+\t\tdie(\"no graph hash specified\");\n+\n+\tgraph_file = get_commit_graph_filename_hash(opts.pack_dir, &graph_hash);\n+\tgraph = load_commit_graph_one(graph_file, opts.pack_dir);\n+\n+\tif (!graph)\n+\t\tdie(\"graph file %s does not exist\", graph_file);\n+\n+\tprintf(\"header: %08x %02x %02x %02x %02x\\n\",\n+\t\tntohl(*(uint32_t*)graph->data),\n+\t\t*(unsigned char*)(graph->data + 4),\n+\t\t*(unsigned char*)(graph->data + 5),\n+\t\tgraph->hash_len,\n+\t\tgraph->num_chunks);\n+\tprintf(\"num_commits: %u\\n\", graph->num_commits);\n+\tprintf(\"chunks:\");\n+\n+\tif (graph->chunk_oid_fanout)\n+\t\tprintf(\" oid_fanout\");\n+\tif (graph->chunk_oid_lookup)\n+\t\tprintf(\" oid_lookup\");\n+\tif (graph->chunk_commit_data)\n+\t\tprintf(\" commit_metadata\");\n+\tif (graph->chunk_large_edges)\n+\t\tprintf(\" large_edges\");\n+\tprintf(\"\\n\");\n+\n+\tprintf(\"pack_dir: %s\\n\", graph->pack_dir);\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tstruct object_id *graph_hash;\n@@ -70,6 +139,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n \tif (argc > 0) {\n+\t\tif (!strcmp(argv[0], \"read\"))\n+\t\t\treturn graph_read(argc, argv);\n \t\tif (!strcmp(argv[0], \"write\"))\n \t\t\treturn graph_write(argc, argv);\n \t}\ndiff --git a/commit-graph.c b/commit-graph.c\nindex cb47b68871..9a337cea4d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -51,6 +51,153 @@ char* get_commit_graph_filename_hash(const char *pack_dir,\n \treturn strbuf_detach(&path, &len);\n }\n \n+static struct commit_graph *alloc_commit_graph(int extra)\n+{\n+\tstruct commit_graph *g = xmalloc(st_add(sizeof(*g), extra));\n+\tmemset(g, 0, sizeof(*g));\n+\tg->graph_fd = -1;\n+\n+\treturn g;\n+}\n+\n+static int close_commit_graph(struct commit_graph *g)\n+{\n+\tif (g->graph_fd < 0)\n+\t\treturn 0;\n+\n+\tmunmap((void *)g->data, g->data_len);\n+\tg->data = 0;\n+\n+\tclose(g->graph_fd);\n+\tg->graph_fd = -1;\n+\n+\treturn 1;\n+}\n+\n+static void free_commit_graph(struct commit_graph **g)\n+{\n+\tif (!g || !*g)\n+\t\treturn;\n+\n+\tclose_commit_graph(*g);\n+\tFREE_AND_NULL(*g);\n+}\n+\n+struct commit_graph *load_commit_graph_one(const char *graph_file, const char *pack_dir)\n+{\n+\tvoid *graph_map;\n+\tconst unsigned char *data, *chunk_lookup;\n+\tsize_t graph_size;\n+\tstruct stat st;\n+\tuint32_t i;\n+\tstruct commit_graph *graph;\n+\tint fd = git_open(graph_file);\n+\tuint64_t last_chunk_offset;\n+\tuint32_t last_chunk_id;\n+\tuint32_t graph_signature;\n+\tunsigned char graph_version, hash_version;\n+\n+\tif (fd < 0)\n+\t\treturn 0;\n+\tif (fstat(fd, &st)) {\n+\t\tclose(fd);\n+\t\treturn 0;\n+\t}\n+\tgraph_size = xsize_t(st.st_size);\n+\n+\tif (graph_size < GRAPH_MIN_SIZE) {\n+\t\tclose(fd);\n+\t\tdie(\"graph file %s is too small\", graph_file);\n+\t}\n+\tgraph_map = xmmap(NULL, graph_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tdata = (const unsigned char *)graph_map;\n+\n+\tgraph_signature = ntohl(*(uint32_t*)data);\n+\tif (graph_signature != GRAPH_SIGNATURE) {\n+\t\tmunmap(graph_map, graph_size);\n+\t\tclose(fd);\n+\t\tdie(\"graph signature %X does not match signature %X\",\n+\t\t\tgraph_signature, GRAPH_SIGNATURE);\n+\t}\n+\n+\tgraph_version = *(unsigned char*)(data + 4);\n+\tif (graph_version != GRAPH_VERSION) {\n+\t\tmunmap(graph_map, graph_size);\n+\t\tclose(fd);\n+\t\tdie(\"graph version %X does not match version %X\",\n+\t\t\tgraph_version, GRAPH_VERSION);\n+\t}\n+\n+\thash_version = *(unsigned char*)(data + 5);\n+\tif (hash_version != GRAPH_OID_VERSION) {\n+\t\tmunmap(graph_map, graph_size);\n+\t\tclose(fd);\n+\t\tdie(\"hash version %X does not match version %X\",\n+\t\t\thash_version, GRAPH_OID_VERSION);\n+\t}\n+\n+\tgraph = alloc_commit_graph(strlen(pack_dir) + 1);\n+\n+\tgraph->hash_len = *(unsigned char*)(data + 6);\n+\tgraph->num_chunks = *(unsigned char*)(data + 7);\n+\tgraph->graph_fd = fd;\n+\tgraph->data = graph_map;\n+\tgraph->data_len = graph_size;\n+\n+\tlast_chunk_id = 0;\n+\tlast_chunk_offset = 8;\n+\tchunk_lookup = data + 8;\n+\tfor (i = 0; i < graph->num_chunks; i++) {\n+\t\tuint32_t chunk_id = get_be32(chunk_lookup + 0);\n+\t\tuint64_t chunk_offset1 = get_be32(chunk_lookup + 4);\n+\t\tuint32_t chunk_offset2 = get_be32(chunk_lookup + 8);\n+\t\tuint64_t chunk_offset = (chunk_offset1 << 32) | chunk_offset2;\n+\n+\t\tchunk_lookup += GRAPH_CHUNKLOOKUP_WIDTH;\n+\n+\t\tif (chunk_offset > graph_size - GIT_MAX_RAWSZ)\n+\t\t\tdie(\"improper chunk offset %08x%08x\", (uint32_t)(chunk_offset >> 32),\n+\t\t\t (uint32_t)chunk_offset);\n+\n+\t\tswitch (chunk_id) {\n+\t\t\tcase GRAPH_CHUNKID_OIDFANOUT:\n+\t\t\t\tgraph->chunk_oid_fanout = data + chunk_offset;\n+\t\t\t\tbreak;\n+\n+\t\t\tcase GRAPH_CHUNKID_OIDLOOKUP:\n+\t\t\t\tgraph->chunk_oid_lookup = data + chunk_offset;\n+\t\t\t\tbreak;\n+\n+\t\t\tcase GRAPH_CHUNKID_DATA:\n+\t\t\t\tgraph->chunk_commit_data = data + chunk_offset;\n+\t\t\t\tbreak;\n+\n+\t\t\tcase GRAPH_CHUNKID_LARGEEDGES:\n+\t\t\t\tgraph->chunk_large_edges = data + chunk_offset;\n+\t\t\t\tbreak;\n+\n+\t\t\tcase 0:\n+\t\t\t\tbreak;\n+\n+\t\t\tdefault:\n+\t\t\t\tfree_commit_graph(&graph);\n+\t\t\t\tdie(\"unrecognized graph chunk id: %08x\", chunk_id);\n+\t\t}\n+\n+\t\tif (last_chunk_id == GRAPH_CHUNKID_OIDLOOKUP)\n+\t\t{\n+\t\t\tgraph->num_commits = (chunk_offset - last_chunk_offset)\n+\t\t\t\t\t / graph->hash_len;\n+\t\t}\n+\n+\t\tlast_chunk_id = chunk_id;\n+\t\tlast_chunk_offset = chunk_offset;\n+\t}\n+\n+\tstrcpy(graph->pack_dir, pack_dir);\n+\treturn graph;\n+}\n+\n static void write_graph_chunk_fanout(struct sha1file *f,\n \t\t\t\t struct commit **commits,\n \t\t\t\t int nr_commits)\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 4756f6ba5b..c1608976b3 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -7,6 +7,29 @@\n extern char* get_commit_graph_filename_hash(const char *pack_dir,\n \t\t\t\t\t struct object_id *hash);\n \n+struct commit_graph {\n+\tint graph_fd;\n+\n+\tconst unsigned char *data;\n+\tsize_t data_len;\n+\n+\tunsigned char hash_len;\n+\tunsigned char num_chunks;\n+\tuint32_t num_commits;\n+\tstruct object_id oid;\n+\n+\tconst unsigned char *chunk_oid_fanout;\n+\tconst unsigned char *chunk_oid_lookup;\n+\tconst unsigned char *chunk_commit_data;\n+\tconst unsigned char *chunk_large_edges;\n+\n+\t/* something like \".git/objects/pack\" */\n+\tchar pack_dir[FLEX_ARRAY]; /* more */\n+};\n+\n+extern struct commit_graph *load_commit_graph_one(const char *graph_file,\n+\t\t\t\t\t\t const char *pack_dir);\n+\n extern struct object_id *write_commit_graph(const char *pack_dir);\n \n #endif\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex b762587595..ad1d0e621d 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -21,9 +21,21 @@ test_expect_success 'create commits and repack' '\n \tdone &&\n \tgit repack'\n \n+graph_read_expect() {\n+\tcat >expect <<- EOF\n+\theader: 43475048 01 01 14 04\n+\tnum_commits: $1\n+\tchunks: oid_fanout oid_lookup commit_metadata large_edges\n+\tpack_dir: $2\n+\tEOF\n+}\n+\n test_expect_success 'write graph' '\n \tgraph1=$(git commit-graph write) &&\n-\ttest_path_is_file $packdir/graph-$graph1.graph'\n+\ttest_path_is_file $packdir/graph-$graph1.graph &&\n+\tgit commit-graph read --graph-hash=$graph1 >output &&\n+\tgraph_read_expect \"3\" \"$packdir\" &&\n+\ttest_cmp expect output'\n \n test_expect_success 'Add more commits' '\n \tgit reset --hard commits/1 &&\n@@ -62,7 +74,10 @@ test_expect_success 'Add more commits' '\n \n test_expect_success 'write graph with merges' '\n \tgraph2=$(git commit-graph write)&&\n-\ttest_path_is_file $packdir/graph-$graph2.graph'\n+\ttest_path_is_file $packdir/graph-$graph2.graph &&\n+\tgit commit-graph read --graph-hash=$graph2 >output &&\n+\tgraph_read_expect \"10\" \"$packdir\" &&\n+\ttest_cmp expect output'\n \n test_expect_success 'Add one more commit' '\n \ttest_commit 8 &&\n@@ -85,14 +100,20 @@ test_expect_success 'Add one more commit' '\n \n test_expect_success 'write graph with new commit' '\n \tgraph3=$(git commit-graph write) &&\n-\ttest_path_is_file $packdir/graph-$graph3.graph'\n-\n+\ttest_path_is_file $packdir/graph-$graph3.graph &&\n+\ttest_path_is_file $packdir/graph-$graph3.graph &&\n+\tgit commit-graph read --graph-hash=$graph3 >output &&\n+\tgraph_read_expect \"11\" \"$packdir\" &&\n+\ttest_cmp expect output'\n \n test_expect_success 'write graph with nothing new' '\n \tgraph4=$(git commit-graph write) &&\n \ttest_path_is_file $packdir/graph-$graph4.graph &&\n \tprintf $graph3 >expect &&\n \tprintf $graph4 >output &&\n+\ttest_cmp expect output &&\n+\tgit commit-graph read --graph-hash=$graph4 >output &&\n+\tgraph_read_expect \"11\" \"$packdir\" &&\n \ttest_cmp expect output'\n \n test_expect_success 'setup bare repo' '\n@@ -103,7 +124,10 @@ test_expect_success 'setup bare repo' '\n \n test_expect_success 'write graph in bare repo' '\n \tgraphbare=$(git commit-graph write) &&\n-\ttest_path_is_file $baredir/graph-$graphbare.graph'\n+\ttest_path_is_file $baredir/graph-$graphbare.graph &&\n+\tgit commit-graph read --graph-hash=$graphbare >output &&\n+\tgraph_read_expect \"11\" \"$baredir\" &&\n+\ttest_cmp expect output'\n \n test_done\n \n-- \n2.15.1.45.g9b7079f\n\n"},{"id":"338770","messageId":"1518122258-157281-3-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1518122258-157281-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v3 02/14] graph: add commit graph design document","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-08T20:37:26Z","receivedAt":"2018-02-08T20:38:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Add Documentation/technical/commit-graph.txt with details of the planned\ncommit graph feature, including future plans.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 189 +++++++++++++++++++++++++++++++\n 1 file changed, 189 insertions(+)\n create mode 100644 Documentation/technical/commit-graph.txt\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nnew file mode 100644\nindex 0000000000..fc86b06041\n--- /dev/null\n+++ b/Documentation/technical/commit-graph.txt\n@@ -0,0 +1,189 @@\n+Git Commit Graph Design Notes\n+=============================\n+\n+Git walks the commit graph for many reasons, including:\n+\n+1. Listing and filtering commit history.\n+2. Computing merge bases.\n+\n+These operations can become slow as the commit count grows. The merge\n+base calculation shows up in many user-facing commands, such as 'merge-base'\n+or 'status' and can take minutes to compute depending on history shape.\n+\n+There are two main costs here:\n+\n+1. Decompressing and parsing commits.\n+2. Walking the entire graph to avoid topological order mistakes.\n+\n+The commit graph file is a supplemental data structure that accelerates\n+commit graph walks. If a user downgrades or disables the 'core.commitGraph'\n+config setting, then the existing ODB is sufficient. The file is stored\n+next to packfiles either in the .git/objects/pack directory or in the pack\n+directory of an alternate.\n+\n+The commit graph file stores the commit graph structure along with some\n+extra metadata to speed up graph walks. By listing commit OIDs in lexi-\n+cographic order, we can identify an integer position for each commit and\n+refer to the parents of a commit using those integer positions. We use\n+binary search to find initial commits and then use the integer positions\n+for fast lookups during the walk.\n+\n+A consumer may load the following info for a commit from the graph:\n+\n+1. The commit OID.\n+2. The list of parents, along with their integer position.\n+3. The commit date.\n+4. The root tree OID.\n+5. The generation number (see definition below).\n+\n+Values 1-4 satisfy the requirements of parse_commit_gently().\n+\n+Define the \"generation number\" of a commit recursively as follows:\n+\n+ * A commit with no parents (a root commit) has generation number one.\n+\n+ * A commit with at least one parent has generation number one more than\n+ the largest generation number among its parents.\n+\n+Equivalently, the generation number of a commit A is one more than the\n+length of a longest path from A to a root commit. The recursive definition\n+is easier to use for computation and observing the following property:\n+\n+ If A and B are commits with generation numbers N and M, respectively,\n+ and N <= M, then A cannot reach B. That is, we know without searching\n+ that B is not an ancestor of A because it is further from a root commit\n+ than A.\n+\n+ Conversely, when checking if A is an ancestor of B, then we only need\n+ to walk commits until all commits on the walk boundary have generation\n+ number at most N. If we walk commits using a priority queue seeded by\n+ generation numbers, then we always expand the boundary commit with highest\n+ generation number and can easily detect the stopping condition.\n+\n+This property can be used to significantly reduce the time it takes to\n+walk commits and determine topological relationships. Without generation\n+numbers, the general heuristic is the following:\n+\n+ If A and B are commits with commit time X and Y, respectively, and\n+ X < Y, then A _probably_ cannot reach B.\n+\n+This heuristic is currently used whenever the computation can make\n+mistakes with topological orders (such as \"git log\" with default order),\n+but is not used when the topological order is required (such as merge\n+base calculations, \"git log --graph\").\n+\n+In practice, we expect some commits to be created recently and not stored\n+in the commit graph. We can treat these commits as having \"infinite\"\n+generation number and walk until reaching commits with known generation\n+number.\n+\n+Design Details\n+--------------\n+\n+- A graph file is stored in a file named 'graph-.graph' in the pack\n+ directory. This could be stored in an alternate.\n+\n+- The most-recent graph file hash is stored in a 'graph-head' file for\n+ immediate access and storing backup graphs. This could be stored in an\n+ alternate, and refers to a 'graph-.graph' file in the same pack\n+ directory.\n+\n+- The core.commitGraph config setting must be on to consume graph files.\n+\n+- The file format includes parameters for the object id length and hash\n+ algorithm, so a future change of hash algorithm does not require a change\n+ in format.\n+\n+Current Limitations\n+-------------------\n+\n+- Only one graph file is used at one time. This allows the integer position\n+ to seek into the single graph file. It is possible to extend the model\n+ for multiple graph files, but that is currently not part of the design.\n+\n+- .graph files are managed only by the 'commit-graph' builtin. These are not\n+ updated automatically during clone, fetch, repack, or creating new commits.\n+\n+- There is no '--verify' option for the 'commit-graph' builtin to verify the\n+ contents of the graph file agree with the contents in the ODB.\n+\n+- When rewriting the graph, we do not check for a commit still existing\n+ in the ODB, so garbage collection may remove commits.\n+\n+- Generation numbers are not computed in the current version. The file\n+ format supports storing them, along with a mechanism to upgrade from\n+ a file without generation numbers to one that uses them.\n+\n+Future Work\n+-----------\n+\n+- The file format includes room for precomputed generation numbers. These\n+ are not currently computed, so all generation numbers will be marked as\n+ 0 (or \"uncomputed\"). A later patch will include this calculation.\n+\n+- The commit graph is currently incompatible with commit grafts. This can be\n+ remedied by duplicating or refactoring the current graft logic.\n+\n+- After computing and storing generation numbers, we must make graph\n+ walks aware of generation numbers to gain the performance benefits they\n+ enable. This will mostly be accomplished by swapping a commit-date-ordered\n+ priority queue with one ordered by generation number. The following\n+ operations are important candidates:\n+\n+ - paint_down_to_common()\n+ - 'log --topo-order'\n+\n+- The graph currently only adds commits to a previously existing graph.\n+ When writing a new graph, we could check that the ODB still contains\n+ the commits and choose to remove the commits that are deleted from the\n+ ODB. For performance reasons, this check should remain optional.\n+\n+- Currently, parse_commit_gently() requires filling in the root tree\n+ object for a commit. This passes through lookup_tree() and consequently\n+ lookup_object(). Also, it calls lookup_commit() when loading the parents.\n+ These method calls check the ODB for object existence, even if the\n+ consumer does not need the content. For example, we do not need the\n+ tree contents when computing merge bases. Now that commit parsing is\n+ removed from the computation time, these lookup operations are the\n+ slowest operations keeping graph walks from being fast. Consider\n+ loading these objects without verifying their existence in the ODB and\n+ only loading them fully when consumers need them. Consider a method\n+ such as \"ensure_tree_loaded(commit)\" that fully loads a tree before\n+ using commit->tree.\n+\n+- The current design uses the 'commit-graph' builtin to generate the graph.\n+ When this feature stabilizes enough to recommend to most users, we should\n+ add automatic graph writes to common operations that create many commits.\n+ For example, one coulde compute a graph on 'clone', 'fetch', or 'repack'\n+ commands.\n+\n+- A server could provide a commit graph file as part of the network protocol\n+ to avoid extra calculations by clients.\n+\n+Related Links\n+-------------\n+[0] https://bugs.chromium.org/p/git/issues/detail?id=8\n+ Chromium work item for: Serialized Commit Graph\n+\n+[1] https://public-inbox.org/git/20110713070517.GC18566@sigill.intra.peff.net/\n+ An abandoned patch that introduced generation numbers.\n+\n+[2] https://public-inbox.org/git/20170908033403.q7e6dj7benasrjes@sigill.intra.peff.net/\n+ Discussion about generation numbers on commits and how they interact\n+ with fsck.\n+\n+[3] https://public-inbox.org/git/20170907094718.b6kuzp2uhvkmwcso@sigill.intra.peff.net/t/#m7a2ea7b355aeda962e6b86404bcbadc648abfbba\n+ More discussion about generation numbers and not storing them inside\n+ commit objects. A valuable quote:\n+\n+ \"I think we should be moving more in the direction of keeping\n+ repo-local caches for optimizations. Reachability bitmaps have been\n+ a big performance win. I think we should be doing the same with our\n+ properties of commits. Not just generation numbers, but making it\n+ cheap to access the graph structure without zlib-inflating whole\n+ commit objects (i.e., packv4 or something like the \"metapacks\" I\n+ proposed a few years ago).\"\n+\n+[4] https://public-inbox.org/git/20180108154822.54829-1-git@jeffhostetler.com/T/#u\n+ A patch to remove the ahead-behind calculation from 'status'.\n+\n-- \n2.15.1.45.g9b7079f\n\n"},{"id":"338771","messageId":"1518122258-157281-5-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1518122258-157281-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v3 04/14] commit-graph: implement write_commit_graph()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-08T20:37:28Z","receivedAt":"2018-02-08T20:38:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach Git to write a commit graph file by checking all packed objects\nto see if they are commits, then store the file in the given pack\ndirectory.\n\nSigned-off-by: Derrick Stolee \n---\n Makefile | 1 +\n commit-graph.c | 368 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n commit-graph.h | 13 ++\n 3 files changed, 382 insertions(+)\n create mode 100644 commit-graph.c\n create mode 100644 commit-graph.h\n\ndiff --git a/Makefile b/Makefile\nindex fc40b816dc..eeaeb6a745 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -761,6 +761,7 @@ LIB_OBJS += color.o\n LIB_OBJS += column.o\n LIB_OBJS += combine-diff.o\n LIB_OBJS += commit.o\n+LIB_OBJS += commit-graph.o\n LIB_OBJS += compat/obstack.o\n LIB_OBJS += compat/terminal.o\n LIB_OBJS += config.o\ndiff --git a/commit-graph.c b/commit-graph.c\nnew file mode 100644\nindex 0000000000..cb47b68871\n--- /dev/null\n+++ b/commit-graph.c\n@@ -0,0 +1,368 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"git-compat-util.h\"\n+#include \"pack.h\"\n+#include \"packfile.h\"\n+#include \"commit.h\"\n+#include \"object.h\"\n+#include \"revision.h\"\n+#include \"sha1-lookup.h\"\n+#include \"commit-graph.h\"\n+\n+#define GRAPH_SIGNATURE 0x43475048 /* \"CGPH\" */\n+#define GRAPH_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n+#define GRAPH_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n+#define GRAPH_CHUNKID_DATA 0x43444154 /* \"CDAT\" */\n+#define GRAPH_CHUNKID_LARGEEDGES 0x45444745 /* \"EDGE\" */\n+\n+#define GRAPH_DATA_WIDTH 36\n+\n+#define GRAPH_VERSION_1 0x1\n+#define GRAPH_VERSION GRAPH_VERSION_1\n+\n+#define GRAPH_OID_VERSION_SHA1 1\n+#define GRAPH_OID_LEN_SHA1 20\n+#define GRAPH_OID_VERSION GRAPH_OID_VERSION_SHA1\n+#define GRAPH_OID_LEN GRAPH_OID_LEN_SHA1\n+\n+#define GRAPH_LARGE_EDGES_NEEDED 0x80000000\n+#define GRAPH_PARENT_MISSING 0x7fffffff\n+#define GRAPH_EDGE_LAST_MASK 0x7fffffff\n+#define GRAPH_PARENT_NONE 0x70000000\n+\n+#define GRAPH_LAST_EDGE 0x80000000\n+\n+#define GRAPH_FANOUT_SIZE (4 * 256)\n+#define GRAPH_CHUNKLOOKUP_WIDTH 12\n+#define GRAPH_CHUNKLOOKUP_SIZE (5 * GRAPH_CHUNKLOOKUP_WIDTH)\n+#define GRAPH_MIN_SIZE (GRAPH_CHUNKLOOKUP_SIZE + GRAPH_FANOUT_SIZE + \\\n+\t\t\tGRAPH_OID_LEN + 8)\n+\n+char* get_commit_graph_filename_hash(const char *pack_dir,\n+\t\t\t\t struct object_id *hash)\n+{\n+\tsize_t len;\n+\tstruct strbuf path = STRBUF_INIT;\n+\tstrbuf_addstr(&path, pack_dir);\n+\tstrbuf_addstr(&path, \"/graph-\");\n+\tstrbuf_addstr(&path, oid_to_hex(hash));\n+\tstrbuf_addstr(&path, \".graph\");\n+\n+\treturn strbuf_detach(&path, &len);\n+}\n+\n+static void write_graph_chunk_fanout(struct sha1file *f,\n+\t\t\t\t struct commit **commits,\n+\t\t\t\t int nr_commits)\n+{\n+\tuint32_t i, count = 0;\n+\tstruct commit **list = commits;\n+\tstruct commit **last = commits + nr_commits;\n+\n+\t/*\n+\t * Write the first-level table (the list is sorted,\n+\t * but we use a 256-entry lookup to be able to avoid\n+\t * having to do eight extra binary search iterations).\n+\t */\n+\tfor (i = 0; i < 256; i++) {\n+\t\twhile (list < last) {\n+\t\t\tif ((*list)->object.oid.hash[0] != i)\n+\t\t\t\tbreak;\n+\t\t\tcount++;\n+\t\t\tlist++;\n+\t\t}\n+\n+\t\tsha1write_be32(f, count);\n+\t}\n+}\n+\n+static void write_graph_chunk_oids(struct sha1file *f, int hash_len,\n+\t\t\t\t struct commit **commits, int nr_commits)\n+{\n+\tstruct commit **list, **last = commits + nr_commits;\n+\tfor (list = commits; list < last; list++)\n+\t\tsha1write(f, (*list)->object.oid.hash, (int)hash_len);\n+}\n+\n+static int commit_pos(struct commit **commits, int nr_commits,\n+\t\t const struct object_id *oid, uint32_t *pos)\n+{\n+\tuint32_t first = 0, last = nr_commits;\n+\n+\twhile (first < last) {\n+\t\tuint32_t mid = first + (last - first) / 2;\n+\t\tstruct object_id *current;\n+\t\tint cmp;\n+\n+\t\tcurrent = &(commits[mid]->object.oid);\n+\t\tcmp = oidcmp(oid, current);\n+\t\tif (!cmp) {\n+\t\t\t*pos = mid;\n+\t\t\treturn 1;\n+\t\t}\n+\t\tif (cmp > 0) {\n+\t\t\tfirst = mid + 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tlast = mid;\n+\t}\n+\n+\t*pos = first;\n+\treturn 0;\n+}\n+\n+static void write_graph_chunk_data(struct sha1file *f, int hash_len,\n+\t\t\t\t struct commit **commits, int nr_commits)\n+{\n+\tstruct commit **list = commits;\n+\tstruct commit **last = commits + nr_commits;\n+\tuint32_t num_large_edges = 0;\n+\n+\twhile (list < last) {\n+\t\tstruct commit_list *parent;\n+\t\tuint32_t int_id;\n+\t\tuint32_t packedDate[2];\n+\n+\t\tparse_commit(*list);\n+\t\tsha1write(f, (*list)->tree->object.oid.hash, hash_len);\n+\n+\t\tparent = (*list)->parents;\n+\n+\t\tif (!parent)\n+\t\t\tint_id = GRAPH_PARENT_NONE;\n+\t\telse if (!commit_pos(commits, nr_commits,\n+\t\t\t\t &(parent->item->object.oid), &int_id))\n+\t\t\tint_id = GRAPH_PARENT_MISSING;\n+\n+\t\tsha1write_be32(f, int_id);\n+\n+\t\tif (parent)\n+\t\t\tparent = parent->next;\n+\n+\t\tif (!parent)\n+\t\t\tint_id = GRAPH_PARENT_NONE;\n+\t\telse if (parent->next)\n+\t\t\tint_id = GRAPH_LARGE_EDGES_NEEDED | num_large_edges;\n+\t\telse if (!commit_pos(commits, nr_commits,\n+\t\t\t\t &(parent->item->object.oid), &int_id))\n+\t\t\tint_id = GRAPH_PARENT_MISSING;\n+\n+\t\tsha1write_be32(f, int_id);\n+\n+\t\tif (parent && parent->next) {\n+\t\t\tdo {\n+\t\t\t\tnum_large_edges++;\n+\t\t\t\tparent = parent->next;\n+\t\t\t} while (parent);\n+\t\t}\n+\n+\t\tif (sizeof((*list)->date) > 4)\n+\t\t\tpackedDate[0] = htonl(((*list)->date >> 32) & 0x3);\n+\t\telse\n+\t\t\tpackedDate[0] = 0;\n+\n+\t\tpackedDate[1] = htonl((*list)->date);\n+\t\tsha1write(f, packedDate, 8);\n+\n+\t\tlist++;\n+\t}\n+}\n+\n+static void write_graph_chunk_large_edges(struct sha1file *f,\n+\t\t\t\t\t struct commit **commits,\n+\t\t\t\t\t int nr_commits)\n+{\n+\tstruct commit **list = commits;\n+\tstruct commit **last = commits + nr_commits;\n+\tstruct commit_list *parent;\n+\n+\twhile (list < last) {\n+\t\tint num_parents = 0;\n+\t\tfor (parent = (*list)->parents; num_parents < 3 && parent;\n+\t\t parent = parent->next)\n+\t\t\tnum_parents++;\n+\n+\t\tif (num_parents <= 2) {\n+\t\t\tlist++;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/* Since num_parents > 2, this initializer is safe. */\n+\t\tfor (parent = (*list)->parents->next; parent; parent = parent->next) {\n+\t\t\tuint32_t int_id, swap_int_id;\n+\t\t\tuint32_t last_edge = 0;\n+\t\t\tif (!parent->next)\n+\t\t\t\tlast_edge |= GRAPH_LAST_EDGE;\n+\n+\t\t\tif (commit_pos(commits, nr_commits,\n+\t\t\t\t &(parent->item->object.oid),\n+\t\t\t\t &int_id))\n+\t\t\t\tswap_int_id = htonl(int_id | last_edge);\n+\t\t\telse\n+\t\t\t\tswap_int_id = htonl(GRAPH_PARENT_MISSING | last_edge);\n+\n+\t\t\tsha1write(f, &swap_int_id, 4);\n+\t\t}\n+\n+\t\tlist++;\n+\t}\n+}\n+\n+static int commit_compare(const void *_a, const void *_b)\n+{\n+\tstruct object_id *a = *(struct object_id **)_a;\n+\tstruct object_id *b = *(struct object_id **)_b;\n+\treturn oidcmp(a, b);\n+}\n+\n+struct packed_commit_list {\n+\tstruct commit **list;\n+\tint nr;\n+\tint alloc;\n+};\n+\n+struct packed_oid_list {\n+\tstruct object_id **list;\n+\tint nr;\n+\tint alloc;\n+};\n+\n+static int if_packed_commit_add_to_list(const struct object_id *oid,\n+\t\t\t\t\tstruct packed_git *pack,\n+\t\t\t\t\tuint32_t pos,\n+\t\t\t\t\tvoid *data)\n+{\n+\tstruct packed_oid_list *list = (struct packed_oid_list*)data;\n+\tenum object_type type;\n+\tunsigned long size;\n+\tvoid *inner_data;\n+\toff_t offset = nth_packed_object_offset(pack, pos);\n+\tinner_data = unpack_entry(pack, offset, &type, &size);\n+\n+\tif (inner_data)\n+\t\tfree(inner_data);\n+\n+\tif (type != OBJ_COMMIT)\n+\t\treturn 0;\n+\n+\tALLOC_GROW(list->list, list->nr + 1, list->alloc);\n+\tlist->list[list->nr] = xmalloc(sizeof(struct object_id));\n+\toidcpy(list->list[list->nr], oid);\n+\t(list->nr)++;\n+\n+\treturn 0;\n+}\n+\n+struct object_id *write_commit_graph(const char *pack_dir)\n+{\n+\tstruct packed_oid_list oids;\n+\tstruct packed_commit_list commits;\n+\tstruct sha1file *f;\n+\tint i, count_distinct = 0;\n+\tstruct strbuf tmp_file = STRBUF_INIT;\n+\tunsigned char final_hash[GIT_MAX_RAWSZ];\n+\tchar *graph_name;\n+\tint fd;\n+\tuint32_t chunk_ids[5];\n+\tuint64_t chunk_offsets[5];\n+\tint num_long_edges;\n+\tstruct object_id *f_hash;\n+\tchar *fname;\n+\tstruct commit_list *parent;\n+\n+\toids.nr = 0;\n+\toids.alloc = 1024;\n+\tALLOC_ARRAY(oids.list, oids.alloc);\n+\n+\tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n+\n+\tQSORT(oids.list, oids.nr, commit_compare);\n+\n+\tcount_distinct = 1;\n+\tfor (i = 1; i < oids.nr; i++) {\n+\t\tif (oidcmp(oids.list[i-1], oids.list[i]))\n+\t\t\tcount_distinct++;\n+\t}\n+\n+\tcommits.nr = 0;\n+\tcommits.alloc = count_distinct;\n+\tALLOC_ARRAY(commits.list, commits.alloc);\n+\n+\tnum_long_edges = 0;\n+\tfor (i = 0; i < oids.nr; i++) {\n+\t\tint num_parents = 0;\n+\t\tif (i > 0 && !oidcmp(oids.list[i-1], oids.list[i]))\n+\t\t\tcontinue;\n+\n+\t\tcommits.list[commits.nr] = lookup_commit(oids.list[i]);\n+\t\tparse_commit(commits.list[commits.nr]);\n+\n+\t\tfor (parent = commits.list[commits.nr]->parents;\n+\t\t parent; parent = parent->next)\n+\t\t\tnum_parents++;\n+\n+\t\tif (num_parents > 2)\n+\t\t\tnum_long_edges += num_parents - 1;\n+\n+\t\tcommits.nr++;\n+\t}\n+\n+\tstrbuf_addstr(&tmp_file, pack_dir);\n+\tstrbuf_addstr(&tmp_file, \"/tmp_graph_XXXXXX\");\n+\n+\tfd = git_mkstemp_mode(tmp_file.buf, 0444);\n+\tif (fd < 0)\n+\t\tdie_errno(\"unable to create '%s'\", tmp_file.buf);\n+\n+\tgraph_name = strbuf_detach(&tmp_file, NULL);\n+\tf = sha1fd(fd, graph_name);\n+\n+\tsha1write_be32(f, GRAPH_SIGNATURE);\n+\n+\tsha1write_u8(f, GRAPH_VERSION);\n+\tsha1write_u8(f, GRAPH_OID_VERSION);\n+\tsha1write_u8(f, GRAPH_OID_LEN);\n+\tsha1write_u8(f, 4); /* number of chunks */\n+\n+\tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n+\tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n+\tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n+\tchunk_ids[3] = GRAPH_CHUNKID_LARGEEDGES;\n+\tchunk_ids[4] = 0;\n+\n+\tchunk_offsets[0] = 8 + GRAPH_CHUNKLOOKUP_SIZE;\n+\tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n+\tchunk_offsets[2] = chunk_offsets[1] + GRAPH_OID_LEN * commits.nr;\n+\tchunk_offsets[3] = chunk_offsets[2] + (GRAPH_OID_LEN + 16) * commits.nr;\n+\tchunk_offsets[4] = chunk_offsets[3] + 4 * num_long_edges;\n+\n+\tfor (i = 0; i <= 4; i++) {\n+\t\tuint32_t chunk_write[3];\n+\n+\t\tchunk_write[0] = htonl(chunk_ids[i]);\n+\t\tchunk_write[1] = htonl(chunk_offsets[i] >> 32);\n+\t\tchunk_write[2] = htonl(chunk_offsets[i] & 0xffffffff);\n+\t\tsha1write(f, chunk_write, 12);\n+\t}\n+\n+\twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n+\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n+\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n+\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n+\n+\tsha1close(f, final_hash, CSUM_CLOSE | CSUM_FSYNC);\n+\n+\tf_hash = xmalloc(sizeof(struct object_id));\n+\thashcpy(f_hash->hash, final_hash);\n+\tfname = get_commit_graph_filename_hash(pack_dir, f_hash);\n+\n+\tif (rename(graph_name, fname))\n+\t\tdie(\"failed to rename %s to %s\", graph_name, fname);\n+\n+\tfree(oids.list);\n+\toids.alloc = 0;\n+\toids.nr = 0;\n+\n+\treturn f_hash;\n+}\n+\ndiff --git a/commit-graph.h b/commit-graph.h\nnew file mode 100644\nindex 0000000000..4756f6ba5b\n--- /dev/null\n+++ b/commit-graph.h\n@@ -0,0 +1,13 @@\n+#ifndef COMMIT_GRAPH_H\n+#define COMMIT_GRAPH_H\n+\n+#include \"git-compat-util.h\"\n+#include \"commit.h\"\n+\n+extern char* get_commit_graph_filename_hash(const char *pack_dir,\n+\t\t\t\t\t struct object_id *hash);\n+\n+extern struct object_id *write_commit_graph(const char *pack_dir);\n+\n+#endif\n+\n-- \n2.15.1.45.g9b7079f\n\n"},{"id":"338777","messageId":"xmqq8tc3qitf.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"1518122258-157281-2-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v3 01/14] commit-graph: add format document","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-08T21:21:00Z","receivedAt":"2018-02-08T21:21:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> Add document specifying the binary format for commit graphs. This\n> format allows for:\n>\n> * New versions.\n> * New hash functions and hash lengths.\n\nIt still is unclear, at least to me, why OID and OID length are\nstored as if they can be independent. If a reader does not\nunderstand a new Object Id hash, is there anything the reader can\nstill do by knowing how long the hash (which it cannot recompute to\nvalidate) is? And if a reader does know what OID hashing scheme is\nused to refer to the objects, it certainly would know how long the\nOIDs are.\n\nGiving length may make sense only when a reader can treat these OIDs\nas completely opaque identifiers, without having to (re)hash from\nthe contents, but if that is the case, then there is no point saying\nwhat exact hash function is used to compute OID.\n\nSo I'd understand storing only either one or the other, but not\nboth. Am I missing something?\n\n> +The Git commit graph stores a list of commit OIDs and some associated\n> +metadata, including:\n> +\n> +- The generation number of the commit. Commits with no parents have\n> + generation number 1; commits with parents have generation number\n> + one more than the maximum generation number of its parents. We\n> + reserve zero as special, and can be used to mark a generation\n> + number invalid or as \"not computed\".\n\nThis \"most natural\" definition of generation number is stricter than\nabsolutely necessary (a looser definition that is sufficient is\n\"gennum of a child is larger than all of its parents'\"). While I\npersonally think that is OK, some people who floated different ideas\nin previous discussions on generation numbers may want to articulate\ntheir ideas again. One idea that I found clever was to use the\ntotal number of commits that are ancestors of a commit instead (it\nis far more expensive to compute than the most natural gennum, but\ndoing so may help other topology math, like \"describe\").\n\n> +CHUNK LOOKUP:\n> +\n> + (C + 1) * 12 bytes listing the table of contents for the chunks:\n> + First 4 bytes describe chunk id. Value 0 is a terminating label.\n> + Other 8 bytes provide offset in current file for chunk to start.\n> + (Chunks are ordered contiguously in the file, so you can infer\n> + the length using the next chunk position if necessary.)\n\nAren't chunks numbered contiguously, starting from #1, thereby\nmaking it unnecessary to store the 4-byte?\n\nHow does a reader obtain the length of the last chunk? Ahh, that is\nwhy there are C+1 entries in this table, not just C, so that the\nreader knows where to stop while reading the last one. Does that\nmean that this table looks like this?\n \n { 1, offset_1 },\n { 2, offset_2 },\n ...\n { C, offset_C },\n { 0, offset_C+1 },\n\nwhere where (offset_N+1 - offset_N) gives the length of chunk #N?\n\n> + The remaining data in the body is described one chunk at a time, and\n> + these chunks may be given in any order. Chunks are required unless\n> + otherwise specified.\n> +\n> +CHUNK DATA:\n> +\n> + OID Fanout (ID: {'O', 'I', 'D', 'F'}) (256 * 4 bytes)\n> + The ith entry, F[i], stores the number of OIDs with first\n> + byte at most i. Thus F[255] stores the total\n> + number of commits (N).\n> +\n> + OID Lookup (ID: {'O', 'I', 'D', 'L'}) (N * H bytes)\n> + The OIDs for all commits in the graph, sorted in ascending order.\n> +\n> + Commit Data (ID: {'C', 'G', 'E', 'T' }) (N * (H + 16) bytes)\n> + * The first H bytes are for the OID of the root tree.\n> + * The next 8 bytes are for the int-ids of the first two parents\n> + of the ith commit. Stores value 0xffffffff if no parent in that\n> + position. If there are more than two parents, the second value\n> + has its most-significant bit on and the other bits store an array\n> + position into the Large Edge List chunk.\n> + * The next 8 bytes store the generation number of the commit and\n> + the commit time in seconds since EPOCH. The generation number\n> + uses the higher 30 bits of the first 4 bytes, while the commit\n> + time uses the 32 bits of the second 4 bytes, along with the lowest\n> + 2 bits of the lowest byte, storing the 33rd and 34th bit of the\n> + commit time.\n> +\n> + Large Edge List (ID: {'E', 'D', 'G', 'E'})\n> + This list of 4-byte values store the second through nth parents for\n> + all octopus merges. The second parent value in the commit data stores\n> + an array position within this list along with the most-significant bit\n> + on. Starting at that array position, iterate through this list of int-ids\n> + for the parents until reaching a value with the most-significant bit on.\n> + The other bits correspond to the int-id of the last parent. This chunk\n> + should always be present, but may be empty.\n\nI am not convinced about the value of these 4-byte section IDs. \n\nThey are useless unless you define what should happen when a reader\nsees a block of data with an unknown ID; presence of these IDs given\nan impression that you allow a reimplementation to reorder the\nsections unnecessarily, especially when all of these are required\nanyway, making the canonical reader implementation unnecessarily\ncomplex, no?\n\n"},{"id":"338778","messageId":"xmqqzi4jp3y5.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"1518122258-157281-4-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v3 03/14] commit-graph: create git-commit-graph builtin","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-08T21:27:30Z","receivedAt":"2018-02-08T21:27:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> Teach git the 'commit-graph' builtin that will be used for writing and\n> reading packed graph files. The current implementation is mostly\n> empty, except for a '--pack-dir' option.\n\nWhy do we want to use \"pack\" dir, when this is specifically designed\nnot tied to packfile? .git/objects/pack/ certainly is a possibility\nin the sense that anywhere inside .git/objects/ would make sense,\nbut using the \"pack\" dir smells like signalling a wrong message to\nusers.\n\n"},{"id":"338779","messageId":"68287af0-a4ec-d2ab-bbcc-84326871120b@gmail.com","threadId":"47724","inReplyTo":"xmqq8tc3qitf.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3 01/14] commit-graph: add format document","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-08T21:33:21Z","receivedAt":"2018-02-08T21:33:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/8/2018 4:21 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> Add document specifying the binary format for commit graphs. This\n>> format allows for:\n>>\n>> * New versions.\n>> * New hash functions and hash lengths.\n> It still is unclear, at least to me, why OID and OID length are\n> stored as if they can be independent. If a reader does not\n> understand a new Object Id hash, is there anything the reader can\n> still do by knowing how long the hash (which it cannot recompute to\n> validate) is? And if a reader does know what OID hashing scheme is\n> used to refer to the objects, it certainly would know how long the\n> OIDs are.\n>\n> Giving length may make sense only when a reader can treat these OIDs\n> as completely opaque identifiers, without having to (re)hash from\n> the contents, but if that is the case, then there is no point saying\n> what exact hash function is used to compute OID.\n>\n> So I'd understand storing only either one or the other, but not\n> both. Am I missing something?\n\nYou're right that this data is redundant. It is easy to describe the \nwidth of the tables using the OID length, so it is convenient to have \nthat part of the format. Also, it is good to have 4-byte alignment here, \nso we are not wasting space.\n\nThere isn't a strong reason to put that here, but I don't have a great \nreason to remove it, either.\n\nPerhaps leave a byte blank for possible future use?\n\n>\n>> +The Git commit graph stores a list of commit OIDs and some associated\n>> +metadata, including:\n>> +\n>> +- The generation number of the commit. Commits with no parents have\n>> + generation number 1; commits with parents have generation number\n>> + one more than the maximum generation number of its parents. We\n>> + reserve zero as special, and can be used to mark a generation\n>> + number invalid or as \"not computed\".\n> This \"most natural\" definition of generation number is stricter than\n> absolutely necessary (a looser definition that is sufficient is\n> \"gennum of a child is larger than all of its parents'\"). While I\n> personally think that is OK, some people who floated different ideas\n> in previous discussions on generation numbers may want to articulate\n> their ideas again. One idea that I found clever was to use the\n> total number of commits that are ancestors of a commit instead (it\n> is far more expensive to compute than the most natural gennum, but\n> doing so may help other topology math, like \"describe\").\n\nIt is more difficult to compute the number of reachable commits, since \nyou cannot learn that only by looking at the parents (you need to know \nhow many commits are in the intersection of their reachable sets for a \ntwo-parent merge, or just walk all of the commits). This leads to a \nquadratic computation to discover the value for N commits.\n\nI define it this rigidly now because I will submit a patch soon after \nthis one lands that computes generation numbers and consumes them in \npaint_down_to_common(). I've got it sitting in my local repo ready for a \nrebase.\n\n>\n>> +CHUNK LOOKUP:\n>> +\n>> + (C + 1) * 12 bytes listing the table of contents for the chunks:\n>> + First 4 bytes describe chunk id. Value 0 is a terminating label.\n>> + Other 8 bytes provide offset in current file for chunk to start.\n>> + (Chunks are ordered contiguously in the file, so you can infer\n>> + the length using the next chunk position if necessary.)\n> Aren't chunks numbered contiguously, starting from #1, thereby\n> making it unnecessary to store the 4-byte?\n>\n> How does a reader obtain the length of the last chunk? Ahh, that is\n> why there are C+1 entries in this table, not just C, so that the\n> reader knows where to stop while reading the last one. Does that\n> mean that this table looks like this?\n> \n> { 1, offset_1 },\n> { 2, offset_2 },\n> ...\n> { C, offset_C },\n> { 0, offset_C+1 },\n>\n> where where (offset_N+1 - offset_N) gives the length of chunk #N?\n\nThis is correct.\n\n>\n>> + The remaining data in the body is described one chunk at a time, and\n>> + these chunks may be given in any order. Chunks are required unless\n>> + otherwise specified.\n>> +\n>> +CHUNK DATA:\n>> +\n>> + OID Fanout (ID: {'O', 'I', 'D', 'F'}) (256 * 4 bytes)\n>> + The ith entry, F[i], stores the number of OIDs with first\n>> + byte at most i. Thus F[255] stores the total\n>> + number of commits (N).\n>> +\n>> + OID Lookup (ID: {'O', 'I', 'D', 'L'}) (N * H bytes)\n>> + The OIDs for all commits in the graph, sorted in ascending order.\n>> +\n>> + Commit Data (ID: {'C', 'G', 'E', 'T' }) (N * (H + 16) bytes)\n>> + * The first H bytes are for the OID of the root tree.\n>> + * The next 8 bytes are for the int-ids of the first two parents\n>> + of the ith commit. Stores value 0xffffffff if no parent in that\n>> + position. If there are more than two parents, the second value\n>> + has its most-significant bit on and the other bits store an array\n>> + position into the Large Edge List chunk.\n>> + * The next 8 bytes store the generation number of the commit and\n>> + the commit time in seconds since EPOCH. The generation number\n>> + uses the higher 30 bits of the first 4 bytes, while the commit\n>> + time uses the 32 bits of the second 4 bytes, along with the lowest\n>> + 2 bits of the lowest byte, storing the 33rd and 34th bit of the\n>> + commit time.\n>> +\n>> + Large Edge List (ID: {'E', 'D', 'G', 'E'})\n>> + This list of 4-byte values store the second through nth parents for\n>> + all octopus merges. The second parent value in the commit data stores\n>> + an array position within this list along with the most-significant bit\n>> + on. Starting at that array position, iterate through this list of int-ids\n>> + for the parents until reaching a value with the most-significant bit on.\n>> + The other bits correspond to the int-id of the last parent. This chunk\n>> + should always be present, but may be empty.\n> I am not convinced about the value of these 4-byte section IDs.\n>\n> They are useless unless you define what should happen when a reader\n> sees a block of data with an unknown ID; presence of these IDs given\n> an impression that you allow a reimplementation to reorder the\n> sections unnecessarily, especially when all of these are required\n> anyway, making the canonical reader implementation unnecessarily\n> complex, no?\n>\n\nOne reason to have chunks is to be simple: we have a clear table of \ncontents at the beginning of the file and can immediately navigate to \nportions of the file we care about. The chunk order also doesn't matter \nfor the purpose of the format.\n\nThe true value they present is flexibility: We can extend the v1 format \nto include extra metadata in a chunk and insert it anywhere in the \nformat. If there is some extra information that would be beneficial for \ngraphs (say, a de-duplicated author list) then we could extend the \nformat to be v1-compatible.\n\nBUT: I do notice now that load_commit_graph_one() will die() if seeing a \nchunk id it doesn't recognize, so that should be fixed if we keep this \nchunk model. We should be able to read \"v1.1\" files if they have extra \nchunks (but ignore that data).\n\nThanks,\n-Stolee\n"},{"id":"338780","messageId":"9c473af6-fc67-5612-07dd-54b2a6e24ff0@gmail.com","threadId":"47724","inReplyTo":"xmqqzi4jp3y5.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3 03/14] commit-graph: create git-commit-graph builtin","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-08T21:36:20Z","receivedAt":"2018-02-08T21:36:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/8/2018 4:27 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> Teach git the 'commit-graph' builtin that will be used for writing and\n>> reading packed graph files. The current implementation is mostly\n>> empty, except for a '--pack-dir' option.\n> Why do we want to use \"pack\" dir, when this is specifically designed\n> not tied to packfile? .git/objects/pack/ certainly is a possibility\n> in the sense that anywhere inside .git/objects/ would make sense,\n> but using the \"pack\" dir smells like signalling a wrong message to\n> users.\n>\n\nI wanted to have the smallest footprint as possible in the objects \ndirectory, and the .git/objects directory currently only holds folders.\n\nI suppose this feature, along with the multi-pack-index (MIDX), extends \nthe concept of the pack directory to be a \"compressed data\" directory, \nbut keeps the \"pack\" name to be compatible with earlier versions.\n\nAnother option is to create a .git/objects/graph directory instead, but \nthen we need to worry about that directory being present.\n\nThanks,\n-Stolee\n"},{"id":"338785","messageId":"xmqqtvurp1s2.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"1518122258-157281-5-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v3 04/14] commit-graph: implement write_commit_graph()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-08T22:14:21Z","receivedAt":"2018-02-08T22:14:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> +char* get_commit_graph_filename_hash(const char *pack_dir,\n\nAsterisk sticks to the identifier, not type, in our codebase.\n\n> +\t\t\t\t struct object_id *hash)\n> +{\n> +\tsize_t len;\n> +\tstruct strbuf path = STRBUF_INIT;\n> +\tstrbuf_addstr(&path, pack_dir);\n> +\tstrbuf_addstr(&path, \"/graph-\");\n> +\tstrbuf_addstr(&path, oid_to_hex(hash));\n> +\tstrbuf_addstr(&path, \".graph\");\n\nUse of strbuf_addf() would make it easier to read and maintain, no?\n\n> +\n> +\treturn strbuf_detach(&path, &len);\n> +}\n> +\n> +static void write_graph_chunk_fanout(struct sha1file *f,\n> +\t\t\t\t struct commit **commits,\n> +\t\t\t\t int nr_commits)\n> +{\n> +\tuint32_t i, count = 0;\n> +\tstruct commit **list = commits;\n> +\tstruct commit **last = commits + nr_commits;\n> +\n> +\t/*\n> +\t * Write the first-level table (the list is sorted,\n> +\t * but we use a 256-entry lookup to be able to avoid\n> +\t * having to do eight extra binary search iterations).\n> +\t */\n> +\tfor (i = 0; i < 256; i++) {\n> +\t\twhile (list < last) {\n> +\t\t\tif ((*list)->object.oid.hash[0] != i)\n> +\t\t\t\tbreak;\n> +\t\t\tcount++;\n> +\t\t\tlist++;\n> +\t\t}\n\nIf count and list are always incremented in unison, perhaps you do\nnot need an extra variable \"last\". If typeof(nr_commits) is wider\nthan typeof(count), this loop and the next write-be32 is screwed\nanyway ;-)\n\nThis comment probably applies equally to some other uses of the same\n\"compute last pointer to compare with running pointer for\ntermination\" pattern in this patch.\n\n> +\t\tsha1write_be32(f, count);\n> +\t}\n> +}\n\n> +static int if_packed_commit_add_to_list(const struct object_id *oid,\n\nThat is a strange name. \"collect packed commits\", perhaps?\n\n> +\t\t\t\t\tstruct packed_git *pack,\n> +\t\t\t\t\tuint32_t pos,\n> +\t\t\t\t\tvoid *data)\n> +{\n> +\tstruct packed_oid_list *list = (struct packed_oid_list*)data;\n> +\tenum object_type type;\n> +\tunsigned long size;\n> +\tvoid *inner_data;\n> +\toff_t offset = nth_packed_object_offset(pack, pos);\n> +\tinner_data = unpack_entry(pack, offset, &type, &size);\n> +\n> +\tif (inner_data)\n> +\t\tfree(inner_data);\n> +\n> +\tif (type != OBJ_COMMIT)\n> +\t\treturn 0;\n> +\n> +\tALLOC_GROW(list->list, list->nr + 1, list->alloc);\n\nThis probably will become inefficient in large repositories. You\nknow you'll be walking all the pack files, and total number of\nobjects in a packfile can be read cheaply, so it may make sense to\nmake a rough guestimate of the number of commits (e.g. 15-25% of the\ntotal number of objects) in the repository and allocate the list\nupfront, instead of a hardcoded 1024.\n\n"},{"id":"338792","messageId":"xmqq8tc36pj9.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"68287af0-a4ec-d2ab-bbcc-84326871120b@gmail.com","subject":"Re: [PATCH v3 01/14] commit-graph: add format document","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-08T23:16:10Z","receivedAt":"2018-02-08T23:16:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> You're right that this data is redundant. It is easy to describe the\n> width of the tables using the OID length, so it is convenient to have\n> that part of the format. Also, it is good to have 4-byte alignment\n> here, so we are not wasting space.\n>\n> There isn't a strong reason to put that here, but I don't have a great\n> reason to remove it, either.\n\nRedundant information that can go out of sync is a great enough\nreason not to have it in the first place.\n"},{"id":"338793","messageId":"xmqq4lmr6paj.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"9c473af6-fc67-5612-07dd-54b2a6e24ff0@gmail.com","subject":"Re: [PATCH v3 03/14] commit-graph: create git-commit-graph builtin","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-08T23:21:24Z","receivedAt":"2018-02-08T23:21:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> I wanted to have the smallest footprint as possible in the objects\n> directory, and the .git/objects directory currently only holds\n> folders.\n\nWhen we cull stale files from pack directory, we rely on the related\nfiles to share pack-.* pattern. It is better not to contaminate\nthe directory with unrelated cruft.\n\nAs this is purely optional auxiliary information used for optimization,\nperhaps .git/objects/info is a better place? I dunno.\n\nIn any case, even if its default position ends up in .git/objects/pack/,\nif this thing conceptually does not have any ties with packs\n(i.e. it is not a corruption if the graph file also described\ntopologies including loose objects, and it is not a corruption if\nthe graph file did not cover objects in all packs), then the end\nuser visible option name shouldn't say \"--pack-dir\". \"--graph-dir\"\nthat defaults to .git/objects/pack/ might be acceptable but it still\nfeels wrong.\n\n\n"},{"id":"338794","messageId":"xmqqy3k359xw.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"1518122258-157281-7-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v3 06/14] commit-graph: implement 'git-commit-graph read'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-08T23:38:19Z","receivedAt":"2018-02-08T23:38:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> +'read'::\n> +\n> +Read a graph file given by the graph-head file and output basic\n> +details about the graph file.\n\n\"a graph file\", assuming that there must be only one in the\nspecified place? Or if there are more than one, read all of them?\nOr is it an error to have more than one?\n\n Do not answer questions in a message that is a response to _me_;\n the purpose of a review is not to educate reviewers---it is to\n improve the patch.\n\n> +With `--graph-hash=` option, consider the graph file\n> +graph-.graph in the pack directory.\n\nI think it is more in line with how plumbing works to just let the\nfull pathname be specifiable (e.g. learn from how pack-objects takes\n\"pack-\" prefix from the command line, even though in practice names\nof all packs you see in any repos start from \"pack-\").\n\n> +struct commit_graph *load_commit_graph_one(const char *graph_file, const char *pack_dir)\n\nThis somehow smells like a screwed up API. It gets a filename to\nread from that is directly passed to git_open(). Why does an\ninstance of graph has to know and remember the path to the directory\n(i.e. pack_dir) that was given when it was constructed? \"I am an\ninstance that holds commit topology learned from this object\ndatabase\" is something it might want to know and remember, but \"I am\ntold that I'll eventually be written back to there when I was\ncreated\" does not sound like a useful thing to have.\n\n"},{"id":"338866","messageId":"CAM0VKjk-JvCNebTiHVm16vtwKpLF+N-gfGNL=7sYR=OVTb979w@mail.gmail.com","threadId":"47724","inReplyTo":"1518122258-157281-15-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v3 14/14] commit-graph: build graph from starting commits","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-02-09T13:02:14Z","receivedAt":"2018-02-09T13:02:24Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Feb 8, 2018 at 9:37 PM, Derrick Stolee wrote:\n> Teach git-commit-graph to read commits from stdin when the\n> --stdin-commits flag is specified. Commits reachable from these\n> commits are added to the graph. This is a much faster way to construct\n> the graph than inspecting all packed objects, but is restricted to\n> known tips.\n>\n> For the Linux repository, 700,000+ commits were added to the graph\n> file starting from 'master' in 7-9 seconds, depending on the number\n> of packfiles in the repo (1, 24, or 120).\n\nIt seems something went wrong with '--stdin-commits' in v3, look:\n\n ~/src/git (commit-graph-v2 %)$ time { git rev-parse HEAD | ./git\ncommit-graph --write --update-head --stdin-commits ; }\n ee3223fe116bf7031a6c1ad6d41e0456beefa754\n\n real 0m1.199s\n user 0m1.123s\n sys 0m0.024s\n\n ~/src/git (commit-graph-v3 %)$ time { git rev-parse HEAD | ./git\ncommit-graph write --update-head --stdin-commits ; }\n ee3223fe116bf7031a6c1ad6d41e0456beefa754\n\n real 0m30.766s\n user 0m29.120s\n sys 0m0.546s\n"},{"id":"338878","messageId":"054043b5-7f2e-0b69-d770-ab23b756f82b@gmail.com","threadId":"47724","inReplyTo":"CAM0VKjk-JvCNebTiHVm16vtwKpLF+N-gfGNL=7sYR=OVTb979w@mail.gmail.com","subject":"Re: [PATCH v3 14/14] commit-graph: build graph from starting commits","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-09T13:45:55Z","receivedAt":"2018-02-09T13:53:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/9/2018 8:02 AM, SZEDER Gábor wrote:\n> On Thu, Feb 8, 2018 at 9:37 PM, Derrick Stolee wrote:\n>> Teach git-commit-graph to read commits from stdin when the\n>> --stdin-commits flag is specified. Commits reachable from these\n>> commits are added to the graph. This is a much faster way to construct\n>> the graph than inspecting all packed objects, but is restricted to\n>> known tips.\n>>\n>> For the Linux repository, 700,000+ commits were added to the graph\n>> file starting from 'master' in 7-9 seconds, depending on the number\n>> of packfiles in the repo (1, 24, or 120).\n> It seems something went wrong with '--stdin-commits' in v3, look:\n>\n> ~/src/git (commit-graph-v2 %)$ time { git rev-parse HEAD | ./git\n> commit-graph --write --update-head --stdin-commits ; }\n> ee3223fe116bf7031a6c1ad6d41e0456beefa754\n>\n> real 0m1.199s\n> user 0m1.123s\n> sys 0m0.024s\n>\n> ~/src/git (commit-graph-v3 %)$ time { git rev-parse HEAD | ./git\n> commit-graph write --update-head --stdin-commits ; }\n> ee3223fe116bf7031a6c1ad6d41e0456beefa754\n>\n> real 0m30.766s\n> user 0m29.120s\n> sys 0m0.546s\n\nThanks, Szeder. You're right. This is the diff that I forgot to apply in \nthe last commit:\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 28d043b..175b967 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -257,7 +257,7 @@ static int graph_write(int argc, const char **argv)\n\n         has_existing = !!get_graph_head_hash(opts.pack_dir, \n&old_graph_hash);\n\n-       if (opts.stdin_packs) {\n+       if (opts.stdin_packs || opts.stdin_commits) {\n                 struct strbuf buf = STRBUF_INIT;\n                 nr_lines = 0;\n                 alloc_lines = 128;\n\n\nI'll work to create a test that ensures we are only adding commits \nreachable from specific commits to prevent this regression.\n\nThanks,\n-Stolee\n"},{"id":"339091","messageId":"xmqqeflqt4st.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"1518122258-157281-8-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v3 07/14] commit-graph: update graph-head during write","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-12T18:56:50Z","receivedAt":"2018-02-12T18:56:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> It is possible to have multiple commit graph files in a pack directory,\n> but only one is important at a time. Use a 'graph_head' file to point\n> to the important file. Teach git-commit-graph to write 'graph_head' upon\n> writing a new commit graph file.\n\nWhy this design, instead of what \"repack -a\" would do, iow, if there\nalways is a singleton that is the only one that matters, shouldn't\nthe creation of that latest singleton just clear the older ones\nbefore it returns control?\n"},{"id":"339105","messageId":"xmqqwozirlky.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"xmqqeflqt4st.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3 07/14] commit-graph: update graph-head during write","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-12T20:37:17Z","receivedAt":"2018-02-12T20:37:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Derrick Stolee writes:\n>\n>> It is possible to have multiple commit graph files in a pack directory,\n>> but only one is important at a time. Use a 'graph_head' file to point\n>> to the important file. Teach git-commit-graph to write 'graph_head' upon\n>> writing a new commit graph file.\n>\n> Why this design, instead of what \"repack -a\" would do, iow, if there\n> always is a singleton that is the only one that matters, shouldn't\n> the creation of that latest singleton just clear the older ones\n> before it returns control?\n\nNote that I am not complaining---I am just curious why we want to\nexpose this \"there is one relevant one but we keep irrelevant ones\nwe usually do not look at and need to be garbage collected\" to end\nusers, and also expect readers of the series, resulting code and\ndocs would have the same puzzled feeling.\n\n\n"},{"id":"339112","messageId":"99543db0-26e4-8daa-a580-b618497e48ba@gmail.com","threadId":"47724","inReplyTo":"xmqqwozirlky.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3 07/14] commit-graph: update graph-head during write","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-12T21:24:55Z","receivedAt":"2018-02-12T21:25:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/12/2018 3:37 PM, Junio C Hamano wrote:\n> Junio C Hamano writes:\n>\n>> Derrick Stolee writes:\n>>\n>>> It is possible to have multiple commit graph files in a pack directory,\n>>> but only one is important at a time. Use a 'graph_head' file to point\n>>> to the important file. Teach git-commit-graph to write 'graph_head' upon\n>>> writing a new commit graph file.\n>> Why this design, instead of what \"repack -a\" would do, iow, if there\n>> always is a singleton that is the only one that matters, shouldn't\n>> the creation of that latest singleton just clear the older ones\n>> before it returns control?\n> Note that I am not complaining---I am just curious why we want to\n> expose this \"there is one relevant one but we keep irrelevant ones\n> we usually do not look at and need to be garbage collected\" to end\n> users, and also expect readers of the series, resulting code and\n> docs would have the same puzzled feeling.\n>\n\nAside: I forgot to mention in my cover letter that the experience around \nthe \"--delete-expired\" flag for \"git commit-graph write\" is different \nthan v2. If specified, we delete all \".graph\" files in the pack \ndirectory other than the one referenced by \"graph_head\" at the beginning \nof the process or the one written by the process. If these deletes fail, \nthen we ignore the failure (assuming that they are being used by another \nGit process). In usual cases, we will delete these expired files in the \nnext instance. I believe this matches similar behavior in gc and repack.\n\n-- Back to discussion about the value of \"graph_head\" --\n\nThe current design of using a pointer file (graph_head) is intended to \nhave these benefits:\n\n1. We do not need to rely on a directory listing and mtimes to determine \nwhich graph file to use.\n\n2. If we write a new graph file while another git process is reading the \nexisting graph file, we can update the graph_head pointer without \ndeleting the file that is currently memory-mapped. (This is why we \ncannot just rely on a canonical file name, such as \"the_graph\", to store \nthe data.)\n\n3. We can atomically change the 'graph_head' file without interrupting \nconcurrent git processes. I think this is different from the \"repack\" \nsituation because a concurrent process would load all packfiles in the \npack directory and possibly have open handles when the repack is trying \nto delete them.\n\n4. We remain open to making the graph file incremental (as the MIDX \nfeature is designed to do; see [1]). It is less crucial to have an \nincremental graph file structure (the graph file for the Windows \nrepository is currently ~120MB versus a MIDX file of 1.25 GB), but the \ngraph_head pattern makes this a possibility.\n\nI tried to avoid item 1 due to personal taste, and since I am storing \nthe files in the objects/pack directory (so that listing may be very \nlarge with a lot of wasted entries). This is less important with our \npending change of moving the graph files to a different directory. This \nalso satisfies items 2 and 3, as long as we never write graph files so \nquickly that we have a collision on mtime.\n\nI cannot think of another design that satisfies item 4.\n\nAs for your end user concerns: My philosophy with this feature is that \nend users will never interact with the commit-graph builtin. 99% of \nusers will benefit from a repack or GC automatically computing a commit \ngraph (when we add that integration point). The other uses for the \nbuiltin are for users who want extreme control over their data, such as \ncode servers and build agents.\n\nPerhaps someone with experience managing large repositories with git in \na server environment could chime in with some design requirements they \nwould need.\n\nThanks,\n-Stolee\n\n[1] \nhttps://public-inbox.org/git/20180107181459.222909-2-dstolee@microsoft.com/\n     [RFC PATCH 01/18] docs: Multi-Pack Index (MIDX) Design Notes\n"},{"id":"339234","messageId":"20180213135755.85944f875fdca85b6d4db9a3@google.com","threadId":"47724","inReplyTo":"1518122258-157281-6-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v3 05/14] commit-graph: implement 'git-commit-graph write'","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-13T21:57:55Z","receivedAt":"2018-02-13T21:58:04Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 8 Feb 2018 15:37:29 -0500\nDerrick Stolee wrote:\n\n> +test_expect_success 'setup full repo' '\n> +\trm -rf .git &&\n> +\tmkdir full &&\n> +\tcd full &&\n> +\tgit init &&\n> +\tpackdir=\".git/objects/pack\"'\n\nThanks for simplifying the repo generated in the test. One more style\nnit: the final apostrophe goes onto its own line.\n"},{"id":"339237","messageId":"20180213143853.00cbd0fd2a17131beb2e08b7@google.com","threadId":"47724","inReplyTo":"1518122258-157281-8-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v3 07/14] commit-graph: update graph-head during write","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-13T22:38:53Z","receivedAt":"2018-02-13T22:39:02Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 8 Feb 2018 15:37:31 -0500\nDerrick Stolee wrote:\n\n> It is possible to have multiple commit graph files in a pack directory,\n> but only one is important at a time. Use a 'graph_head' file to point\n> to the important file. Teach git-commit-graph to write 'graph_head' upon\n> writing a new commit graph file.\n\nYou should probably include the rationale for a special \"graph_head\"\nfile that you describe here [1] in the commit message.\n\n[1] https://public-inbox.org/git/99543db0-26e4-8daa-a580-b618497e48ba@gmail.com/\n\n> +char *get_graph_head_filename(const char *pack_dir)\n> +{\n> +\tstruct strbuf fname = STRBUF_INIT;\n> +\tstrbuf_addstr(&fname, pack_dir);\n> +\tstrbuf_addstr(&fname, \"/graph-head\");\n> +\treturn strbuf_detach(&fname, 0);\n\nNULL, not 0.\n\n> +}\n"},{"id":"339239","messageId":"20180213144929.174b7e3304ab592d2f8c0aea@google.com","threadId":"47724","inReplyTo":"1518122258-157281-9-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v3 08/14] commit-graph: implement 'git-commit-graph clear'","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-13T22:49:29Z","receivedAt":"2018-02-13T22:49:36Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 8 Feb 2018 15:37:32 -0500\nDerrick Stolee wrote:\n\n> Teach Git to delete the current 'graph_head' file and the commit graph\n> it references. This is a good safety valve if somehow the file is\n> corrupted and needs to be recalculated. Since the commit graph is a\n> summary of contents already in the ODB, it can be regenerated.\n\nSpelling of graph-head (hyphen, not underscore).\n\nI'm not sure of the usefulness of this feature - if the graph is indeed\ncorrupt, the user can just be instructed to delete graph-head (not even\nthe commit graph it references, since when we create a new graph-head,\n--delete-expired will take care of deleting the old one).\n\n> extern char *get_graph_head_filename(const char *pack_dir);\n> +extern struct object_id *get_graph_head_hash(const char *pack_dir,\n> +\t\t\t\t\t struct object_id *hash);\n> extern char* get_commit_graph_filename_hash(const char *pack_dir,\n> \t\t\t\t\t struct object_id *hash);\n\nThis file is starting to need documentation - in particular, the\ndifference between get_graph_head_hash() and\nget_commit_graph_filename_hash() is not clear.\n"},{"id":"339245","messageId":"20180213161225.6e21291d25015be632b03fee@google.com","threadId":"47724","inReplyTo":"1518122258-157281-12-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v3 11/14] commit: integrate commit graph with commit parsing","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-02-14T00:12:25Z","receivedAt":"2018-02-14T00:12:33Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 8 Feb 2018 15:37:35 -0500\nDerrick Stolee wrote:\n\n> | Command | Before | After | Rel % |\n> |----------------------------------|--------|--------|-------|\n> | log --oneline --topo-order -1000 | 5.9s | 0.7s | -88% |\n> | branch -vv | 0.42s | 0.27s | -35% |\n> | rev-list --all | 6.4s | 1.0s | -84% |\n> | rev-list --all --objects | 32.6s | 27.6s | -15% |\n\nCould we have a performance test (in t/perf) demonstrating this?\n\n> +static int check_commit_parents(struct commit *item, struct commit_graph *g,\n> +\t\t\t\tuint32_t pos, const unsigned char *commit_data)\n\nDocument what this function does? Also, this function probably needs a\nbetter name.\n\n> +/*\n> + * Given a commit struct, try to fill the commit struct info, including:\n> + * 1. tree object\n> + * 2. date\n> + * 3. parents.\n> + *\n> + * Returns 1 if and only if the commit was found in the commit graph.\n> + *\n> + * See parse_commit_buffer() for the fallback after this call.\n> + */\n> +int parse_commit_in_graph(struct commit *item)\n> +{\n\nThe documentation above duplicates what's in the header file, so we can\nprobably omit it.\n\n> +extern struct object_id *get_nth_commit_oid(struct commit_graph *g,\n> +\t\t\t\t\t uint32_t n,\n> +\t\t\t\t\t struct object_id *oid);\n\nThis doesn't seem to be used elsewhere - do you plan for a future patch\nto use it?\n"},{"id":"339302","messageId":"5e53ef79-e196-e524-194c-741851db0755@gmail.com","threadId":"47724","inReplyTo":"20180213161225.6e21291d25015be632b03fee@google.com","subject":"Re: [PATCH v3 11/14] commit: integrate commit graph with commit parsing","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-14T18:08:53Z","receivedAt":"2018-02-14T18:09:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/13/2018 7:12 PM, Jonathan Tan wrote:\n> On Thu, 8 Feb 2018 15:37:35 -0500\n> Derrick Stolee wrote:\n>\n>> | Command | Before | After | Rel % |\n>> |----------------------------------|--------|--------|-------|\n>> | log --oneline --topo-order -1000 | 5.9s | 0.7s | -88% |\n>> | branch -vv | 0.42s | 0.27s | -35% |\n>> | rev-list --all | 6.4s | 1.0s | -84% |\n>> | rev-list --all --objects | 32.6s | 27.6s | -15% |\n> Could we have a performance test (in t/perf) demonstrating this?\n\nThe rev-list perf tests are found in t/perf/p0001-rev-list.sh\n\nThe \"log --oneline --topo-order -1000\" test would be good to add to \nt/perf/p4211-line-log.sh\n\nThe \"branch -vv\" test is pretty uninteresting unless you set up your \nrepo to have local branches significantly behind the remote branches. It \ndepends a lot more on the data shape than the others which only need a \nlarge number of reachable objects.\n\nOne reason I did not use the builtin perf test scripts is that they seem \nto ignore all local config options, and hence do not inherit the \ncore.commitGraph=true setting from the repos pointed at by GIT_PERF_REPO.\n\n>\n>> +static int check_commit_parents(struct commit *item, struct commit_graph *g,\n>> +\t\t\t\tuint32_t pos, const unsigned char *commit_data)\n> Document what this function does? Also, this function probably needs a\n> better name.\n>\n>> +/*\n>> + * Given a commit struct, try to fill the commit struct info, including:\n>> + * 1. tree object\n>> + * 2. date\n>> + * 3. parents.\n>> + *\n>> + * Returns 1 if and only if the commit was found in the commit graph.\n>> + *\n>> + * See parse_commit_buffer() for the fallback after this call.\n>> + */\n>> +int parse_commit_in_graph(struct commit *item)\n>> +{\n> The documentation above duplicates what's in the header file, so we can\n> probably omit it.\n>\n>> +extern struct object_id *get_nth_commit_oid(struct commit_graph *g,\n>> +\t\t\t\t\t uint32_t n,\n>> +\t\t\t\t\t struct object_id *oid);\n> This doesn't seem to be used elsewhere - do you plan for a future patch\n> to use it?\n\n"},{"id":"339306","messageId":"4d1ee202-7d79-d73c-6e05-d0fc85db943c@gmail.com","threadId":"47724","inReplyTo":"1518122258-157281-1-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v3 00/14] Serialized Git Commit Graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-14T18:15:12Z","receivedAt":"2018-02-14T18:15:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"There has been a lot of interesting discussion on this topic. Some of \nthat involves some decently significant changes from v3, so I wanted to \nsummarize my understanding of the feedback and seek out more feedback \nfrom reviewers before rolling v4.\n\nIf we have consensus on these topics, then I'll re-roll on Friday, Feb \n16th. Please let me know if you are planning on reviewing v3 and need \nmore time than that.\n\n\n* Graph Storage:\n\n     - Move the graph files to a different directory than the \"pack\" \ndirectory. Currently considering \".git/objects/info\"\n\n     - Change the \"--pack-dir\" command-line arguments to \"--object-dir\" \narguments.\n\n     - Keep a \"graph_head\" file, but expand on the reasons (as discussed \n[1]) in the commit message.\n\n     - Adjust \"graph_head\" and the \"--graph-id\" argument to use a full \nfilename (assuming based in {object-dir}/info/).\n\n     - Remove \"pack_dir\" from struct commit_graph and \nload_commit_graph_one().\n\n     - Drop \"git commit-graph clear\" subcommand.\n\n\n* Graph format:\n\n     - remove redundant hash type & length bytes in favor of a combined \ntype/length enum byte.\n\n     - emphasize the fact that the file can contain chunk ids unknown to \nGit and will be ignored on read. Also fix the read code to not die() on \nunknown chunk ids.\n\n     - Don't write the large-edge chunk if it is going to be empty. \nModify tests to verify this.\n\n\n* Tests:\n\n     - Format (last apostrophe on new line)\n\n     - Bug check (--stdin-commits should limit by reachability)\n\n\n* Other style fixes.\n\n\n[1] \nhttps://public-inbox.org/git/99543db0-26e4-8daa-a580-b618497e48ba@gmail.com/\n"},{"id":"339309","messageId":"CAGZ79kbMURmVhnaP4rdho9TpzZ=K+ySkQGgW8TD0x+KgJkqirA@mail.gmail.com","threadId":"47724","inReplyTo":"4d1ee202-7d79-d73c-6e05-d0fc85db943c@gmail.com","subject":"Re: [PATCH v3 00/14] Serialized Git Commit Graph","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-14T18:27:43Z","receivedAt":"2018-02-14T18:27:51Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Feb 14, 2018 at 10:15 AM, Derrick Stolee wrote:\n> There has been a lot of interesting discussion on this topic. Some of that\n> involves some decently significant changes from v3, so I wanted to summarize\n> my understanding of the feedback and seek out more feedback from reviewers\n> before rolling v4.\n>\n> If we have consensus on these topics, then I'll re-roll on Friday, Feb 16th.\n> Please let me know if you are planning on reviewing v3 and need more time\n> than that.\n>\n>\n> * Graph Storage:\n>\n> - Move the graph files to a different directory than the \"pack\"\n> directory. Currently considering \".git/objects/info\"\n\nIn my copy of git there is already a file\n\n $ cat .git/objects/info/packs\n P pack-8fdfd126aa8c2a868baf1f89788b07b79a4d365b.pack\n\nwhich seems to be in line with the information provided in\n'man gitrepository-layout':\n objects/info\n Additional information about the object store is\n recorded in this directory.\n\nThe commit graph files are not exactly \"additional info about the\nobject store\" but rather \"about the objects\". Close enough IMO.\n\nStefan\n"},{"id":"339386","messageId":"84ad10eb-4db1-4a3e-9030-6a53283d9877@gmail.com","threadId":"47724","inReplyTo":"CAGZ79kbMURmVhnaP4rdho9TpzZ=K+ySkQGgW8TD0x+KgJkqirA@mail.gmail.com","subject":"Re: [PATCH v3 00/14] Serialized Git Commit Graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-14T19:11:19Z","receivedAt":"2018-02-14T19:11:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"\n\nOn 2/14/2018 1:27 PM, Stefan Beller wrote:\n> On Wed, Feb 14, 2018 at 10:15 AM, Derrick Stolee wrote:\n>> There has been a lot of interesting discussion on this topic. Some of that\n>> involves some decently significant changes from v3, so I wanted to summarize\n>> my understanding of the feedback and seek out more feedback from reviewers\n>> before rolling v4.\n>>\n>> If we have consensus on these topics, then I'll re-roll on Friday, Feb 16th.\n>> Please let me know if you are planning on reviewing v3 and need more time\n>> than that.\n>>\n>>\n>> * Graph Storage:\n>>\n>> - Move the graph files to a different directory than the \"pack\"\n>> directory. Currently considering \".git/objects/info\"\n> In my copy of git there is already a file\n>\n> $ cat .git/objects/info/packs\n> P pack-8fdfd126aa8c2a868baf1f89788b07b79a4d365b.pack\n>\n> which seems to be in line with the information provided in\n> 'man gitrepository-layout':\n> objects/info\n> Additional information about the object store is\n> recorded in this directory.\n>\n> The commit graph files are not exactly \"additional info about the\n> object store\" but rather \"about the objects\". Close enough IMO.\n>\n> Stefan\n\nThanks for the tip [1]. I was unfamiliar with it because it doesn't \nexist in repos that don't repack.\n\n[1] \nhttps://git-scm.com/docs/gitrepository-layout/2.12.0#gitrepository-layout-objectsinfopacks\n"},{"id":"339488","messageId":"xmqqinay9kv4.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"1518122258-157281-5-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v3 04/14] commit-graph: implement write_commit_graph()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-15T18:19:11Z","receivedAt":"2018-02-15T18:19:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> +struct packed_oid_list {\n> +\tstruct object_id **list;\n> +\tint nr;\n> +\tint alloc;\n> +};\n\nWhat is the typical access pattern for this data structure? If it\nis pretty much \"allocate and grow as we find more\", then a dynamic\narray of struct (rather than a dynamic array of pointers to struct)\nwould be a lot more appropriate. IOW\n\n\tstruct packed_oid_list {\n\t\tstruct object_id *list;\n\t\tint nr, alloc;\n\t};\n\nThe version in the posted patch has to pay malloc overhead plus an\nextra pointer for each object id in the list; unless you often\nreplace elements in the list randomly and/or you borrow object ID\nfield in other existing data structure whose lifetime is longer than\nthis list by pointing at it, I do not see how the extra indirection\nis worth it.\n\n\n"},{"id":"339489","messageId":"91e19f5b-ac4b-5843-b3b0-8eba37aaee68@gmail.com","threadId":"47724","inReplyTo":"xmqqinay9kv4.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3 04/14] commit-graph: implement write_commit_graph()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-15T18:23:12Z","receivedAt":"2018-02-15T18:23:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/15/2018 1:19 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> +struct packed_oid_list {\n>> +\tstruct object_id **list;\n>> +\tint nr;\n>> +\tint alloc;\n>> +};\n> What is the typical access pattern for this data structure? If it\n> is pretty much \"allocate and grow as we find more\", then a dynamic\n> array of struct (rather than a dynamic array of pointers to struct)\n> would be a lot more appropriate. IOW\n>\n> \tstruct packed_oid_list {\n> \t\tstruct object_id *list;\n> \t\tint nr, alloc;\n> \t};\n>\n> The version in the posted patch has to pay malloc overhead plus an\n> extra pointer for each object id in the list; unless you often\n> replace elements in the list randomly and/or you borrow object ID\n> field in other existing data structure whose lifetime is longer than\n> this list by pointing at it, I do not see how the extra indirection\n> is worth it.\n>\n\nThe pattern used in write_commit_graph() is to append OIDs to the list \nas we discover them and then sort in lexicographic order. The sort then \nonly swaps pointers.\n\nI can switch this to sort the 'struct object_id' elements themselves, if \nthat is a better pattern.\n\nThanks,\n-Stolee\n"},{"id":"339490","messageId":"xmqqd1169kku.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"1518122258-157281-12-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v3 11/14] commit: integrate commit graph with commit parsing","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-15T18:25:21Z","receivedAt":"2018-02-15T18:25:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> +struct object_id *get_nth_commit_oid(struct commit_graph *g,\n> +\t\t\t\t uint32_t n,\n> +\t\t\t\t struct object_id *oid)\n> +{\n> +\thashcpy(oid->hash, g->chunk_oid_lookup + g->hash_len * n);\n> +\treturn oid;\n> +}\n\nThis looks like a rather klunky API to me. \n\nIt seems that many current callers in this series (not limited to\nthis step but in later patches in the series) discard the returned\nvalue.\n\nI would understand the API a lot better if the function returned\n\"const struct object_id *\" that points into the copy of the oid the\ngraph structure keeps (and the caller can do hashcpy() if it wants\nto).\n\nThat would allow the API to later check for errors when the caller\ngives 'n' that is too large by returning a NULL, for example.\n\n> +static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n> +\t\t\t\t\t int pos,\n> +\t\t\t\t\t struct commit_list **pptr)\n> +{\n> +\tstruct commit *c;\n> +\tstruct object_id oid;\n> +\tget_nth_commit_oid(g, pos, &oid);\n> +\tc = lookup_commit(&oid);\n"},{"id":"339621","messageId":"1519066406-81663-1-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"4d1ee202-7d79-d73c-6e05-d0fc85db943c@gmail.com","subject":"[PATCH v4 00/13] Serialized Git Commit Graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-19T18:53:13Z","receivedAt":"2018-02-19T18:53:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Thanks for all of the feedback. I've learned a lot working on this patch.\n\nAs discussed [0], this version changes several fundamental structures\nand operations, including:\n\n* Graph files are stored in .git/objects/info\n\n* The \"graph-head\" file is now called \"graph-latest\" to avoid confusion\n with HEAD. We use \"--set-latest\" argument instead of \"--update-head\".\n\n* The graph format no longer stores the hash width, but expects the hash\n version to imply a length. This frees up one byte for future use while\n preserving 4-byte alignment.\n\n* The graph format is more explicit about optional chunks and no longer\n dies when seeing an unknown chunk. The large edges chunk is now optional.\n\n* There is no longer a \"clear\" subcommand to git-commit-graph.\n\n* Fixed the bug related to \"--stdin-commits\" and check that the command\n only includes commits reachable from the input OIDs.\n\n* The struct packed_oid_list type is now an array of struct object_id\n instead of an array of pointers. In my testing, I saw no performance\n difference between these two options, so I switched to the simpler\n pattern.\n\n* My patch is based on jt/binsearch-with-fanout (b4e00f730), with the\n newer method prototype since v3.\n\nThanks,\n-Stolee\n\n[0] https://public-inbox.org/git/1517348383-112294-1-git-send-email-dstolee@microsoft.com/T/#m22bfdb7cf7b3d6e5f380b8bf0eec957e2cfd2dd7\n\n-- >8 --\n\nAs promised [1], this patch contains a way to serialize the commit graph.\nThe current implementation defines a new file format to store the graph\nstructure (parent relationships) and basic commit metadata (commit date,\nroot tree OID) in order to prevent parsing raw commits while performing\nbasic graph walks. For example, we do not need to parse the full commit\nwhen performing these walks:\n\n* 'git log --topo-order -1000' walks all reachable commits to avoid\n incorrect topological orders, but only needs the commit message for\n the top 1000 commits.\n\n* 'git merge-base ' may walk many commits to find the correct\n boundary between the commits reachable from A and those reachable\n from B. No commit messages are needed.\n\n* 'git branch -vv' checks ahead/behind status for all local branches\n compared to their upstream remote branches. This is essentially as\n hard as computing merge bases for each.\n\nThe current patch speeds up these calculations by injecting a check in\nparse_commit_gently() to check if there is a graph file and using that\nto provide the required metadata to the struct commit.\n\nThe file format has room to store generation numbers, which will be\nprovided as a patch after this framework is merged. Generation numbers\nare referenced by the design document but not implemented in order to\nmake the current patch focus on the graph construction process. Once\nthat is stable, it will be easier to add generation numbers and make\ngraph walks aware of generation numbers one-by-one.\n\nHere are some performance results for a copy of the Linux repository\nwhere 'master' has 704,766 reachable commits and is behind 'origin/master'\nby 19,610 commits.\n\n| Command | Before | After | Rel % |\n|----------------------------------|--------|--------|-------|\n| log --oneline --topo-order -1000 | 5.9s | 0.7s | -88% |\n| branch -vv | 0.42s | 0.27s | -35% |\n| rev-list --all | 6.4s | 1.0s | -84% |\n| rev-list --all --objects | 32.6s | 27.6s | -15% |\n\nTo test this yourself, run the following on your repo:\n\n git config core.commitGraph true\n git show-ref -s | git commit-graph write --set-latest --stdin-commits\n\nThe second command writes a commit graph file containing every commit\nreachable from your refs. Now, all git commands that walk commits will\ncheck your graph first before consulting the ODB. You can run your own\nperformance comparisions by toggling the 'core.commitgraph' setting.\n\n[1] https://public-inbox.org/git/d154319e-bb9e-b300-7c37-27b1dcd2a2ce@jeffhostetler.com/\n Re: What's cooking in git.git (Jan 2018, #03; Tue, 23)\n\n[2] https://github.com/derrickstolee/git/pull/2\n A GitHub pull request containing the latest version of this patch.\n\nDerrick Stolee (13):\n commit-graph: add format document\n graph: add commit graph design document\n commit-graph: create git-commit-graph builtin\n commit-graph: implement write_commit_graph()\n commit-graph: implement 'git-commit-graph write'\n commit-graph: implement git commit-graph read\n commit-graph: implement --set-latest\n commit-graph: implement --delete-expired\n commit-graph: add core.commitGraph setting\n commit-graph: close under reachability\n commit: integrate commit graph with commit parsing\n commit-graph: read only from specific pack-indexes\n commit-graph: build graph from starting commits\n\n .gitignore | 1 +\n Documentation/config.txt | 3 +\n Documentation/git-commit-graph.txt | 105 ++++\n Documentation/technical/commit-graph-format.txt | 90 +++\n Documentation/technical/commit-graph.txt | 185 ++++++\n Makefile | 2 +\n alloc.c | 1 +\n builtin.h | 1 +\n builtin/commit-graph.c | 261 +++++++++\n cache.h | 1 +\n command-list.txt | 1 +\n commit-graph.c | 724 ++++++++++++++++++++++++\n commit-graph.h | 49 ++\n commit.c | 3 +\n commit.h | 3 +\n config.c | 5 +\n environment.c | 1 +\n git.c | 1 +\n log-tree.c | 3 +-\n packfile.c | 4 +-\n packfile.h | 2 +\n t/t5318-commit-graph.sh | 244 ++++++++\n 22 files changed, 1686 insertions(+), 4 deletions(-)\n create mode 100644 Documentation/git-commit-graph.txt\n create mode 100644 Documentation/technical/commit-graph-format.txt\n create mode 100644 Documentation/technical/commit-graph.txt\n create mode 100644 builtin/commit-graph.c\n create mode 100644 commit-graph.c\n create mode 100644 commit-graph.h\n create mode 100755 t/t5318-commit-graph.sh\n\n-- \n2.15.1.44.g453ed2b\n\n"},{"id":"339622","messageId":"1519066406-81663-2-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1519066406-81663-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v4 01/13] commit-graph: add format document","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-19T18:53:14Z","receivedAt":"2018-02-19T18:53:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Add document specifying the binary format for commit graphs. This\nformat allows for:\n\n* New versions.\n* New hash functions and hash lengths.\n* Optional extensions.\n\nBasic header information is followed by a binary table of contents\ninto \"chunks\" that include:\n\n* An ordered list of commit object IDs.\n* A 256-entry fanout into that list of OIDs.\n* A list of metadata for the commits.\n* A list of \"large edges\" to enable octopus merges.\n\nThe format automatically includes two parent positions for every\ncommit. This favors speed over space, since using only one position\nper commit would cause an extra level of indirection for every merge\ncommit. (Octopus merges suffer from this indirection, but they are\nvery rare.)\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph-format.txt | 90 +++++++++++++++++++++++++\n 1 file changed, 90 insertions(+)\n create mode 100644 Documentation/technical/commit-graph-format.txt\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nnew file mode 100644\nindex 0000000..11b18b5\n--- /dev/null\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -0,0 +1,90 @@\n+Git commit graph format\n+=======================\n+\n+The Git commit graph stores a list of commit OIDs and some associated\n+metadata, including:\n+\n+- The generation number of the commit. Commits with no parents have\n+ generation number 1; commits with parents have generation number\n+ one more than the maximum generation number of its parents. We\n+ reserve zero as special, and can be used to mark a generation\n+ number invalid or as \"not computed\".\n+\n+- The root tree OID.\n+\n+- The commit date.\n+\n+- The parents of the commit, stored using positional references within\n+ the graph file.\n+\n+== graph-*.graph files have the following format:\n+\n+In order to allow extensions that add extra data to the graph, we organize\n+the body into \"chunks\" and provide a binary lookup table at the beginning\n+of the body. The header includes certain values, such as number of chunks,\n+hash lengths and types.\n+\n+All 4-byte numbers are in network order.\n+\n+HEADER:\n+\n+ 4-byte signature:\n+ The signature is: {'C', 'G', 'P', 'H'}\n+\n+ 1-byte version number:\n+ Currently, the only valid version is 1.\n+\n+ 1-byte Object Id Version (1 = SHA-1)\n+\n+ 1-byte number (C) of \"chunks\"\n+\n+ 1-byte (reserved for later use)\n+\n+CHUNK LOOKUP:\n+\n+ (C + 1) * 12 bytes listing the table of contents for the chunks:\n+ First 4 bytes describe chunk id. Value 0 is a terminating label.\n+ Other 8 bytes provide offset in current file for chunk to start.\n+ (Chunks are ordered contiguously in the file, so you can infer\n+ the length using the next chunk position if necessary.)\n+\n+ The remaining data in the body is described one chunk at a time, and\n+ these chunks may be given in any order. Chunks are required unless\n+ otherwise specified.\n+\n+CHUNK DATA:\n+\n+ OID Fanout (ID: {'O', 'I', 'D', 'F'}) (256 * 4 bytes)\n+ The ith entry, F[i], stores the number of OIDs with first\n+ byte at most i. Thus F[255] stores the total\n+ number of commits (N).\n+\n+ OID Lookup (ID: {'O', 'I', 'D', 'L'}) (N * H bytes)\n+ The OIDs for all commits in the graph, sorted in ascending order.\n+\n+ Commit Data (ID: {'C', 'G', 'E', 'T' }) (N * (H + 16) bytes)\n+ * The first H bytes are for the OID of the root tree.\n+ * The next 8 bytes are for the int-ids of the first two parents\n+ of the ith commit. Stores value 0xffffffff if no parent in that\n+ position. If there are more than two parents, the second value\n+ has its most-significant bit on and the other bits store an array\n+ position into the Large Edge List chunk.\n+ * The next 8 bytes store the generation number of the commit and\n+ the commit time in seconds since EPOCH. The generation number\n+ uses the higher 30 bits of the first 4 bytes, while the commit\n+ time uses the 32 bits of the second 4 bytes, along with the lowest\n+ 2 bits of the lowest byte, storing the 33rd and 34th bit of the\n+ commit time.\n+\n+ Large Edge List (ID: {'E', 'D', 'G', 'E'}) [Optional]\n+ This list of 4-byte values store the second through nth parents for\n+ all octopus merges. The second parent value in the commit data stores\n+ an array position within this list along with the most-significant bit\n+ on. Starting at that array position, iterate through this list of int-ids\n+ for the parents until reaching a value with the most-significant bit on.\n+ The other bits correspond to the int-id of the last parent.\n+\n+TRAILER:\n+\n+\tH-byte HASH-checksum of all of the above.\n+\n-- \n2.7.4\n\n"},{"id":"339623","messageId":"1519066406-81663-4-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1519066406-81663-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v4 03/13] commit-graph: create git-commit-graph builtin","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-19T18:53:16Z","receivedAt":"2018-02-19T18:53:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach git the 'commit-graph' builtin that will be used for writing and\nreading packed graph files. The current implementation is mostly\nempty, except for an '--object-dir' option.\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-commit-graph.txt | 11 +++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/commit-graph.c | 37 +++++++++++++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n 7 files changed, 53 insertions(+)\n create mode 100644 Documentation/git-commit-graph.txt\n create mode 100644 builtin/commit-graph.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 833ef3b..e82f901 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -34,6 +34,7 @@\n /git-clone\n /git-column\n /git-commit\n+/git-commit-graph\n /git-commit-tree\n /git-config\n /git-count-objects\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nnew file mode 100644\nindex 0000000..e1c3078\n--- /dev/null\n+++ b/Documentation/git-commit-graph.txt\n@@ -0,0 +1,11 @@\n+git-commit-graph(1)\n+===================\n+\n+NAME\n+----\n+git-commit-graph - Write and verify Git commit graphs (.graph files)\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\n+\ndiff --git a/Makefile b/Makefile\nindex ee9d5eb..fc40b81 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -932,6 +932,7 @@ BUILTIN_OBJS += builtin/clone.o\n BUILTIN_OBJS += builtin/column.o\n BUILTIN_OBJS += builtin/commit-tree.o\n BUILTIN_OBJS += builtin/commit.o\n+BUILTIN_OBJS += builtin/commit-graph.o\n BUILTIN_OBJS += builtin/config.o\n BUILTIN_OBJS += builtin/count-objects.o\n BUILTIN_OBJS += builtin/credential.o\ndiff --git a/builtin.h b/builtin.h\nindex 42378f3..079855b 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -149,6 +149,7 @@ extern int cmd_clone(int argc, const char **argv, const char *prefix);\n extern int cmd_clean(int argc, const char **argv, const char *prefix);\n extern int cmd_column(int argc, const char **argv, const char *prefix);\n extern int cmd_commit(int argc, const char **argv, const char *prefix);\n+extern int cmd_commit_graph(int argc, const char **argv, const char *prefix);\n extern int cmd_commit_tree(int argc, const char **argv, const char *prefix);\n extern int cmd_config(int argc, const char **argv, const char *prefix);\n extern int cmd_count_objects(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nnew file mode 100644\nindex 0000000..98110bb\n--- /dev/null\n+++ b/builtin/commit-graph.c\n@@ -0,0 +1,37 @@\n+#include \"builtin.h\"\n+#include \"config.h\"\n+#include \"parse-options.h\"\n+\n+static char const * const builtin_commit_graph_usage[] = {\n+\tN_(\"git commit-graph [--object-dir ]\"),\n+\tNULL\n+};\n+\n+static struct opts_commit_graph {\n+\tconst char *obj_dir;\n+} opts;\n+\n+\n+int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic struct option builtin_commit_graph_options[] = {\n+\t\t{ OPTION_STRING, 'p', \"object-dir\", &opts.obj_dir,\n+\t\t\tN_(\"dir\"),\n+\t\t\tN_(\"The object directory to store the graph\") },\n+\t\tOPT_END(),\n+\t};\n+\n+\tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n+\t\tusage_with_options(builtin_commit_graph_usage,\n+\t\t\t\t builtin_commit_graph_options);\n+\n+\tgit_config(git_default_config, NULL);\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t builtin_commit_graph_options,\n+\t\t\t builtin_commit_graph_usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n+\n+\tusage_with_options(builtin_commit_graph_usage,\n+\t\t\t builtin_commit_graph_options);\n+}\n+\ndiff --git a/command-list.txt b/command-list.txt\nindex a1fad28..835c589 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -34,6 +34,7 @@ git-clean mainporcelain\n git-clone mainporcelain init\n git-column purehelpers\n git-commit mainporcelain history\n+git-commit-graph plumbingmanipulators\n git-commit-tree plumbingmanipulators\n git-config ancillarymanipulators\n git-count-objects ancillaryinterrogators\ndiff --git a/git.c b/git.c\nindex 9e96dd4..d4832c1 100644\n--- a/git.c\n+++ b/git.c\n@@ -388,6 +388,7 @@ static struct cmd_struct commands[] = {\n \t{ \"clone\", cmd_clone },\n \t{ \"column\", cmd_column, RUN_SETUP_GENTLY },\n \t{ \"commit\", cmd_commit, RUN_SETUP | NEED_WORK_TREE },\n+\t{ \"commit-graph\", cmd_commit_graph, RUN_SETUP },\n \t{ \"commit-tree\", cmd_commit_tree, RUN_SETUP },\n \t{ \"config\", cmd_config, RUN_SETUP_GENTLY },\n \t{ \"count-objects\", cmd_count_objects, RUN_SETUP },\n-- \n2.7.4\n\n"},{"id":"339624","messageId":"1519066406-81663-7-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1519066406-81663-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v4 06/13] commit-graph: implement git commit-graph read","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-19T18:53:19Z","receivedAt":"2018-02-19T18:53:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach git-commit-graph to read commit graph files and summarize their contents.\n\nUse the read subcommand to verify the contents of a commit graph file in the\ntests.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 15 +++++\n builtin/commit-graph.c | 63 ++++++++++++++++++++\n commit-graph.c | 116 +++++++++++++++++++++++++++++++++++++\n commit-graph.h | 21 +++++++\n t/t5318-commit-graph.sh | 38 ++++++++++--\n 5 files changed, 249 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex c3f222f..6d26e56 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -9,6 +9,7 @@ git-commit-graph - Write and verify Git commit graphs (.graph files)\n SYNOPSIS\n --------\n [verse]\n+'git commit-graph read' [--object-dir ]\n 'git commit-graph write' [--object-dir ]\n \n \n@@ -34,6 +35,14 @@ Write a commit graph file based on the commits found in packfiles.\n Includes all commits from the existing commit graph file. Outputs the\n resulting filename.\n \n+'read'::\n+\n+Read a graph file given by the graph-head file and output basic\n+details about the graph file.\n++\n+With `--file=` option, consider the graph stored in the file at\n+the path /info/.\n+\n \n EXAMPLES\n --------\n@@ -44,6 +53,12 @@ EXAMPLES\n $ git commit-graph write\n ------------------------------------------------\n \n+* Read basic information from a graph file.\n++\n+------------------------------------------------\n+$ git commit-graph read --file=\n+------------------------------------------------\n+\n \n GIT\n ---\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex a51d87b..28cd097 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -7,10 +7,16 @@\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n+\tN_(\"git commit-graph read [--object-dir ] [--file=]\"),\n \tN_(\"git commit-graph write [--object-dir ]\"),\n \tNULL\n };\n \n+static const char * const builtin_commit_graph_read_usage[] = {\n+\tN_(\"git commit-graph read [--object-dir ] [--file=]\"),\n+\tNULL\n+};\n+\n static const char * const builtin_commit_graph_write_usage[] = {\n \tN_(\"git commit-graph write [--object-dir ]\"),\n \tNULL\n@@ -18,8 +24,63 @@ static const char * const builtin_commit_graph_write_usage[] = {\n \n static struct opts_commit_graph {\n \tconst char *obj_dir;\n+\tconst char *graph_file;\n } opts;\n \n+static int graph_read(int argc, const char **argv)\n+{\n+\tstruct commit_graph *graph = 0;\n+\tstruct strbuf full_path = STRBUF_INIT;\n+\n+\tstatic struct option builtin_commit_graph_read_options[] = {\n+\t\t{ OPTION_STRING, 'o', \"object-dir\", &opts.obj_dir,\n+\t\t\tN_(\"dir\"),\n+\t\t\tN_(\"The object directory to store the graph\") },\n+\t\t{ OPTION_STRING, 'H', \"file\", &opts.graph_file,\n+\t\t\tN_(\"file\"),\n+\t\t\tN_(\"The filename for a specific commit graph file in the object directory.\"),\n+\t\t\tPARSE_OPT_OPTARG, NULL, (intptr_t) \"\" },\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t builtin_commit_graph_read_options,\n+\t\t\t builtin_commit_graph_read_usage, 0);\n+\n+\tif (!opts.obj_dir)\n+\t\topts.obj_dir = get_object_directory();\n+\n+\tif (!opts.graph_file)\n+\t\tdie(\"no graph hash specified\");\n+\n+\tstrbuf_addf(&full_path, \"%s/info/%s\", opts.obj_dir, opts.graph_file);\n+\tgraph = load_commit_graph_one(full_path.buf);\n+\n+\tif (!graph)\n+\t\tdie(\"graph file %s does not exist\", full_path.buf);\n+\n+\tprintf(\"header: %08x %d %d %d %d\\n\",\n+\t\tntohl(*(uint32_t*)graph->data),\n+\t\t*(unsigned char*)(graph->data + 4),\n+\t\t*(unsigned char*)(graph->data + 5),\n+\t\t*(unsigned char*)(graph->data + 6),\n+\t\t*(unsigned char*)(graph->data + 7));\n+\tprintf(\"num_commits: %u\\n\", graph->num_commits);\n+\tprintf(\"chunks:\");\n+\n+\tif (graph->chunk_oid_fanout)\n+\t\tprintf(\" oid_fanout\");\n+\tif (graph->chunk_oid_lookup)\n+\t\tprintf(\" oid_lookup\");\n+\tif (graph->chunk_commit_data)\n+\t\tprintf(\" commit_metadata\");\n+\tif (graph->chunk_large_edges)\n+\t\tprintf(\" large_edges\");\n+\tprintf(\"\\n\");\n+\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tchar *graph_name;\n@@ -68,6 +129,8 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n \tif (argc > 0) {\n+\t\tif (!strcmp(argv[0], \"read\"))\n+\t\t\treturn graph_read(argc, argv);\n \t\tif (!strcmp(argv[0], \"write\"))\n \t\t\treturn graph_write(argc, argv);\n \t}\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f9e39b0..2a8594f 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -38,6 +38,122 @@\n #define GRAPH_MIN_SIZE (GRAPH_CHUNKLOOKUP_SIZE + GRAPH_FANOUT_SIZE + \\\n \t\t\tGRAPH_OID_LEN + 8)\n \n+static struct commit_graph *alloc_commit_graph(void)\n+{\n+\tstruct commit_graph *g = xmalloc(sizeof(*g));\n+\tmemset(g, 0, sizeof(*g));\n+\tg->graph_fd = -1;\n+\n+\treturn g;\n+}\n+\n+struct commit_graph *load_commit_graph_one(const char *graph_file)\n+{\n+\tvoid *graph_map;\n+\tconst unsigned char *data, *chunk_lookup;\n+\tsize_t graph_size;\n+\tstruct stat st;\n+\tuint32_t i;\n+\tstruct commit_graph *graph;\n+\tint fd = git_open(graph_file);\n+\tuint64_t last_chunk_offset;\n+\tuint32_t last_chunk_id;\n+\tuint32_t graph_signature;\n+\tunsigned char graph_version, hash_version;\n+\n+\tif (fd < 0)\n+\t\treturn 0;\n+\tif (fstat(fd, &st)) {\n+\t\tclose(fd);\n+\t\treturn 0;\n+\t}\n+\tgraph_size = xsize_t(st.st_size);\n+\n+\tif (graph_size < GRAPH_MIN_SIZE) {\n+\t\tclose(fd);\n+\t\tdie(\"graph file %s is too small\", graph_file);\n+\t}\n+\tgraph_map = xmmap(NULL, graph_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tdata = (const unsigned char *)graph_map;\n+\n+\tgraph_signature = ntohl(*(uint32_t*)data);\n+\tif (graph_signature != GRAPH_SIGNATURE) {\n+\t\tmunmap(graph_map, graph_size);\n+\t\tclose(fd);\n+\t\tdie(\"graph signature %X does not match signature %X\",\n+\t\t\tgraph_signature, GRAPH_SIGNATURE);\n+\t}\n+\n+\tgraph_version = *(unsigned char*)(data + 4);\n+\tif (graph_version != GRAPH_VERSION) {\n+\t\tmunmap(graph_map, graph_size);\n+\t\tclose(fd);\n+\t\tdie(\"graph version %X does not match version %X\",\n+\t\t\tgraph_version, GRAPH_VERSION);\n+\t}\n+\n+\thash_version = *(unsigned char*)(data + 5);\n+\tif (hash_version != GRAPH_OID_VERSION) {\n+\t\tmunmap(graph_map, graph_size);\n+\t\tclose(fd);\n+\t\tdie(\"hash version %X does not match version %X\",\n+\t\t\thash_version, GRAPH_OID_VERSION);\n+\t}\n+\n+\tgraph = alloc_commit_graph();\n+\n+\tgraph->hash_len = GRAPH_OID_LEN;\n+\tgraph->num_chunks = *(unsigned char*)(data + 6);\n+\tgraph->graph_fd = fd;\n+\tgraph->data = graph_map;\n+\tgraph->data_len = graph_size;\n+\n+\tlast_chunk_id = 0;\n+\tlast_chunk_offset = 8;\n+\tchunk_lookup = data + 8;\n+\tfor (i = 0; i < graph->num_chunks; i++) {\n+\t\tuint32_t chunk_id = get_be32(chunk_lookup + 0);\n+\t\tuint64_t chunk_offset1 = get_be32(chunk_lookup + 4);\n+\t\tuint32_t chunk_offset2 = get_be32(chunk_lookup + 8);\n+\t\tuint64_t chunk_offset = (chunk_offset1 << 32) | chunk_offset2;\n+\n+\t\tchunk_lookup += GRAPH_CHUNKLOOKUP_WIDTH;\n+\n+\t\tif (chunk_offset > graph_size - GIT_MAX_RAWSZ)\n+\t\t\tdie(\"improper chunk offset %08x%08x\", (uint32_t)(chunk_offset >> 32),\n+\t\t\t (uint32_t)chunk_offset);\n+\n+\t\tswitch (chunk_id) {\n+\t\t\tcase GRAPH_CHUNKID_OIDFANOUT:\n+\t\t\t\tgraph->chunk_oid_fanout = (uint32_t*)(data + chunk_offset);\n+\t\t\t\tbreak;\n+\n+\t\t\tcase GRAPH_CHUNKID_OIDLOOKUP:\n+\t\t\t\tgraph->chunk_oid_lookup = data + chunk_offset;\n+\t\t\t\tbreak;\n+\n+\t\t\tcase GRAPH_CHUNKID_DATA:\n+\t\t\t\tgraph->chunk_commit_data = data + chunk_offset;\n+\t\t\t\tbreak;\n+\n+\t\t\tcase GRAPH_CHUNKID_LARGEEDGES:\n+\t\t\t\tgraph->chunk_large_edges = data + chunk_offset;\n+\t\t\t\tbreak;\n+\t\t}\n+\n+\t\tif (last_chunk_id == GRAPH_CHUNKID_OIDLOOKUP)\n+\t\t{\n+\t\t\tgraph->num_commits = (chunk_offset - last_chunk_offset)\n+\t\t\t\t\t / graph->hash_len;\n+\t\t}\n+\n+\t\tlast_chunk_id = chunk_id;\n+\t\tlast_chunk_offset = chunk_offset;\n+\t}\n+\n+\treturn graph;\n+}\n+\n static void write_graph_chunk_fanout(struct sha1file *f,\n \t\t\t\t struct commit **commits,\n \t\t\t\t int nr_commits)\ndiff --git a/commit-graph.h b/commit-graph.h\nindex dc8c73a..9093b97 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -1,6 +1,27 @@\n #ifndef COMMIT_GRAPH_H\n #define COMMIT_GRAPH_H\n \n+#include \"git-compat-util.h\"\n+\n+struct commit_graph {\n+\tint graph_fd;\n+\n+\tconst unsigned char *data;\n+\tsize_t data_len;\n+\n+\tunsigned char hash_len;\n+\tunsigned char num_chunks;\n+\tuint32_t num_commits;\n+\tstruct object_id oid;\n+\n+\tconst uint32_t *chunk_oid_fanout;\n+\tconst unsigned char *chunk_oid_lookup;\n+\tconst unsigned char *chunk_commit_data;\n+\tconst unsigned char *chunk_large_edges;\n+};\n+\n+extern struct commit_graph *load_commit_graph_one(const char *graph_file);\n+\n extern char *write_commit_graph(const char *obj_dir);\n \n #endif\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 6a5e93c..893fa24 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -24,9 +24,27 @@ test_expect_success 'create commits and repack' '\n \tgit repack\n '\n \n+graph_read_expect() {\n+\tOPTIONAL=\"\"\n+\tNUM_CHUNKS=3\n+\tif [ ! -z $2 ]\n+\tthen\n+\t\tOPTIONAL=\" $2\"\n+\t\tNUM_CHUNKS=$((3 + $(echo \"$2\" | wc -w)))\n+\tfi\n+\tcat >expect <<- EOF\n+\theader: 43475048 1 1 $NUM_CHUNKS 0\n+\tnum_commits: $1\n+\tchunks: oid_fanout oid_lookup commit_metadata$OPTIONAL\n+\tEOF\n+}\n+\n test_expect_success 'write graph' '\n \tgraph1=$(git commit-graph write) &&\n-\ttest_path_is_file $objdir/info/$graph1\n+\ttest_path_is_file $objdir/info/$graph1 &&\n+\tgit commit-graph read --file=$graph1 >output &&\n+\tgraph_read_expect \"3\" &&\n+\ttest_cmp expect output\n '\n \n test_expect_success 'Add more commits' '\n@@ -67,7 +85,10 @@ test_expect_success 'Add more commits' '\n \n test_expect_success 'write graph with merges' '\n \tgraph2=$(git commit-graph write)&&\n-\ttest_path_is_file $objdir/info/$graph2\n+\ttest_path_is_file $objdir/info/$graph2 &&\n+\tgit commit-graph read --file=$graph2 >output &&\n+\tgraph_read_expect \"10\" \"large_edges\" &&\n+\ttest_cmp expect output\n '\n \n test_expect_success 'Add one more commit' '\n@@ -92,7 +113,10 @@ test_expect_success 'Add one more commit' '\n \n test_expect_success 'write graph with new commit' '\n \tgraph3=$(git commit-graph write) &&\n-\ttest_path_is_file $objdir/info/$graph3\n+\ttest_path_is_file $objdir/info/$graph3 &&\n+\tgit commit-graph read --file=$graph3 >output &&\n+\tgraph_read_expect \"11\" \"large_edges\" &&\n+\ttest_cmp expect output\n '\n \n test_expect_success 'write graph with nothing new' '\n@@ -100,6 +124,9 @@ test_expect_success 'write graph with nothing new' '\n \ttest_path_is_file $objdir/info/$graph4 &&\n \tprintf $graph3 >expect &&\n \tprintf $graph4 >output &&\n+\ttest_cmp expect output &&\n+\tgit commit-graph read --file=$graph4 >output &&\n+\tgraph_read_expect \"11\" \"large_edges\" &&\n \ttest_cmp expect output\n '\n \n@@ -112,7 +139,10 @@ test_expect_success 'setup bare repo' '\n \n test_expect_success 'write graph in bare repo' '\n \tgraphbare=$(git commit-graph write) &&\n-\ttest_path_is_file $baredir/info/$graphbare\n+\ttest_path_is_file $baredir/info/$graphbare &&\n+\tgit commit-graph read --file=$graphbare >output &&\n+\tgraph_read_expect \"11\" \"large_edges\" &&\n+\ttest_cmp expect output\n '\n \n test_done\n-- \n2.7.4\n\n"},{"id":"339625","messageId":"1519066406-81663-9-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1519066406-81663-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v4 08/13] commit-graph: implement --delete-expired","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-19T18:53:21Z","receivedAt":"2018-02-19T18:53:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach git-commit-graph to delete the .graph files that are siblings of a\nnewly-written graph file, except for the file referenced by 'graph-latest'\nat the beginning of the process and the newly-written file. If we fail to\ndelete a graph file, only report a warning because another git process may\nbe using that file. In a multi-process environment, we expect the previoius\ngraph file to be used by a concurrent process, so we do not delete it to\navoid race conditions.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 11 +++++--\n builtin/commit-graph.c | 61 ++++++++++++++++++++++++++++++++++++--\n commit-graph.c | 23 ++++++++++++++\n commit-graph.h | 1 +\n t/t5318-commit-graph.sh | 7 +++--\n 5 files changed, 96 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex dc948c5..b9b4031 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -37,6 +37,11 @@ resulting filename.\n +\n With `--set-latest` option, update the graph-latest file to point\n to the written graph file.\n++\n+With the `--delete-expired` option, delete the graph files in the pack\n+directory that are not referred to by the graph-latest file. To avoid race\n+conditions, do not delete the file previously referred to by the\n+graph-latest file if it is updated by the `--set-latest` option.\n \n 'read'::\n \n@@ -56,11 +61,11 @@ EXAMPLES\n $ git commit-graph write\n ------------------------------------------------\n \n-* Write a graph file for the packed commits in your local .git folder\n-* and update graph-latest.\n+* Write a graph file for the packed commits in your local .git folder,\n+* update graph-latest, and delete stale graph files.\n +\n ------------------------------------------------\n-$ git commit-graph write --set-latest\n+$ git commit-graph write --set-latest --delete-expired\n ------------------------------------------------\n \n * Read basic information from a graph file.\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex bf86172..fd99169 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -8,7 +8,7 @@\n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ] [--file=]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--set-latest]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--set-latest] [--delete-expired]\"),\n \tNULL\n };\n \n@@ -18,7 +18,7 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--set-latest]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--set-latest] [--delete-expired]\"),\n \tNULL\n };\n \n@@ -26,6 +26,7 @@ static struct opts_commit_graph {\n \tconst char *obj_dir;\n \tconst char *graph_file;\n \tint set_latest;\n+\tint delete_expired;\n } opts;\n \n static int graph_read(int argc, const char **argv)\n@@ -98,9 +99,56 @@ static void set_latest_file(const char *obj_dir, const char *graph_file)\n \tcommit_lock_file(&lk);\n }\n \n+/*\n+ * To avoid race conditions and deleting graph files that are being\n+ * used by other processes, look inside a pack directory for all files\n+ * of the form \"graph-.graph\" that do not match the old or new\n+ * graph hashes and delete them.\n+ */\n+static void do_delete_expired(const char *obj_dir,\n+\t\t\t const char *old_graph_name,\n+\t\t\t const char *new_graph_name)\n+{\n+\tDIR *dir;\n+\tstruct dirent *de;\n+\tint dirnamelen;\n+\tstruct strbuf path = STRBUF_INIT;\n+\n+\tstrbuf_addf(&path, \"%s/info\", obj_dir);\n+\tdir = opendir(path.buf);\n+\tif (!dir) {\n+\t\tif (errno != ENOENT)\n+\t\t\terror_errno(\"unable to open object pack directory: %s\",\n+\t\t\t\t obj_dir);\n+\t\treturn;\n+\t}\n+\n+\tstrbuf_addch(&path, '/');\n+\tdirnamelen = path.len;\n+\twhile ((de = readdir(dir)) != NULL) {\n+\t\tsize_t base_len;\n+\n+\t\tif (is_dot_or_dotdot(de->d_name))\n+\t\t\tcontinue;\n+\n+\t\tstrbuf_setlen(&path, dirnamelen);\n+\t\tstrbuf_addstr(&path, de->d_name);\n+\n+\t\tbase_len = path.len;\n+\t\tif (strip_suffix_mem(path.buf, &base_len, \".graph\") &&\n+\t\t strcmp(new_graph_name, de->d_name) &&\n+\t\t (!old_graph_name || strcmp(old_graph_name, de->d_name)) &&\n+\t\t remove_path(path.buf))\n+\t\t\tdie(\"failed to remove path %s\", path.buf);\n+\t}\n+\n+\tstrbuf_release(&path);\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tchar *graph_name;\n+\tchar *old_graph_name;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\t{ OPTION_STRING, 'o', \"object-dir\", &opts.obj_dir,\n@@ -108,6 +156,8 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"The object directory to store the graph\") },\n \t\tOPT_BOOL('u', \"set-latest\", &opts.set_latest,\n \t\t\tN_(\"update graph-head to written graph file\")),\n+\t\tOPT_BOOL('d', \"delete-expired\", &opts.delete_expired,\n+\t\t\tN_(\"delete expired head graph file\")),\n \t\tOPT_END(),\n \t};\n \n@@ -118,12 +168,19 @@ static int graph_write(int argc, const char **argv)\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n \n+\told_graph_name = get_graph_latest_contents(opts.obj_dir);\n+\n \tgraph_name = write_commit_graph(opts.obj_dir);\n \n \tif (graph_name) {\n \t\tif (opts.set_latest)\n \t\t\tset_latest_file(opts.obj_dir, graph_name);\n \n+\t\tif (opts.delete_expired)\n+\t\t\tdo_delete_expired(opts.obj_dir,\n+\t\t\t\t\t old_graph_name,\n+\t\t\t\t\t graph_name);\n+\n \t\tprintf(\"%s\\n\", graph_name);\n \t\tFREE_AND_NULL(graph_name);\n \t}\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 5ee0805..c8fb38f 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -45,6 +45,29 @@ char *get_graph_latest_filename(const char *obj_dir)\n \treturn strbuf_detach(&fname, 0);\n }\n \n+char *get_graph_latest_contents(const char *obj_dir)\n+{\n+\tstruct strbuf graph_file = STRBUF_INIT;\n+\tchar *fname;\n+\tFILE *f;\n+\tchar buf[64];\n+\n+\tfname = get_graph_latest_filename(obj_dir);\n+\tf = fopen(fname, \"r\");\n+\tFREE_AND_NULL(fname);\n+\n+\tif (!f)\n+\t\treturn 0;\n+\n+\twhile (!feof(f)) {\n+\t\tif (fgets(buf, sizeof(buf), f))\n+\t\t\tstrbuf_addstr(&graph_file, buf);\n+\t}\n+\n+\tfclose(f);\n+\treturn strbuf_detach(&graph_file, NULL);\n+}\n+\n static struct commit_graph *alloc_commit_graph(void)\n {\n \tstruct commit_graph *g = xmalloc(sizeof(*g));\ndiff --git a/commit-graph.h b/commit-graph.h\nindex ae24b3a..56215ad 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -4,6 +4,7 @@\n #include \"git-compat-util.h\"\n \n extern char *get_graph_latest_filename(const char *obj_dir);\n+extern char *get_graph_latest_contents(const char *obj_dir);\n \n struct commit_graph {\n \tint graph_fd;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex cad9d90..1d5ec7d 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -117,8 +117,10 @@ test_expect_success 'Add one more commit' '\n # 1\n \n test_expect_success 'write graph with new commit' '\n-\tgraph3=$(git commit-graph write --set-latest) &&\n+\tgraph3=$(git commit-graph write --set-latest --delete-expired) &&\n \ttest_path_is_file $objdir/info/$graph3 &&\n+\ttest_path_is_file $objdir/info/$graph2 &&\n+\ttest_path_is_missing $objdir/info/$graph1 &&\n \ttest_path_is_file $objdir/info/graph-latest &&\n \tprintf $graph3 >expect &&\n \ttest_cmp expect $objdir/info/graph-latest &&\n@@ -128,8 +130,9 @@ test_expect_success 'write graph with new commit' '\n '\n \n test_expect_success 'write graph with nothing new' '\n-\tgraph4=$(git commit-graph write --set-latest) &&\n+\tgraph4=$(git commit-graph write --set-latest --delete-expired) &&\n \ttest_path_is_file $objdir/info/$graph4 &&\n+\ttest_path_is_missing $objdir/info/$graph2 &&\n \tprintf $graph3 >expect &&\n \tprintf $graph4 >output &&\n \ttest_cmp expect output &&\n-- \n2.7.4\n\n"},{"id":"339626","messageId":"1519066406-81663-10-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1519066406-81663-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v4 09/13] commit-graph: add core.commitGraph setting","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-19T18:53:22Z","receivedAt":"2018-02-19T18:53:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"The commit graph feature is controlled by the new core.commitGraph config\nsetting. This defaults to 0, so the feature is opt-in.\n\nThe intention of core.commitGraph is that a user can always stop checking\nfor or parsing commit graph files if core.commitGraph=0.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config.txt | 3 +++\n cache.h | 1 +\n config.c | 5 +++++\n environment.c | 1 +\n 4 files changed, 10 insertions(+)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 9593bfa..e90d0d1 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -883,6 +883,9 @@ core.notesRef::\n This setting defaults to \"refs/notes/commits\", and it can be overridden by\n the `GIT_NOTES_REF` environment variable. See linkgit:git-notes[1].\n \n+core.commitGraph::\n+\tEnable git commit graph feature. Allows reading from .graph files.\n+\n core.sparseCheckout::\n \tEnable \"sparse checkout\" feature. See section \"Sparse checkout\" in\n \tlinkgit:git-read-tree[1] for more information.\ndiff --git a/cache.h b/cache.h\nindex 6440e2b..1063873 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -771,6 +771,7 @@ extern char *git_replace_ref_base;\n \n extern int fsync_object_files;\n extern int core_preload_index;\n+extern int core_commit_graph;\n extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n extern int protect_hfs;\ndiff --git a/config.c b/config.c\nindex 41862d4..614cf59 100644\n--- a/config.c\n+++ b/config.c\n@@ -1213,6 +1213,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.commitgraph\")) {\n+\t\tcore_commit_graph = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \tif (!strcmp(var, \"core.sparsecheckout\")) {\n \t\tcore_apply_sparse_checkout = git_config_bool(var, value);\n \t\treturn 0;\ndiff --git a/environment.c b/environment.c\nindex 8289c25..81fed83 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -60,6 +60,7 @@ enum push_default_type push_default = PUSH_DEFAULT_UNSPECIFIED;\n enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE;\n char *notes_ref_name;\n int grafts_replace_parents = 1;\n+int core_commit_graph;\n int core_apply_sparse_checkout;\n int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n-- \n2.7.4\n\n"},{"id":"339627","messageId":"1519066406-81663-11-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1519066406-81663-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v4 10/13] commit-graph: close under reachability","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-19T18:53:23Z","receivedAt":"2018-02-19T18:53:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach write_commit_graph() to walk all parents from the commits\ndiscovered in packfiles. This prevents gaps given by loose objects or\npreviously-missed packfiles.\n\nAlso automatically add commits from the existing graph file, if it\nexists.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 23 +++++++++++++++++++++++\n 1 file changed, 23 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex c8fb38f..00bd73a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -385,6 +385,28 @@ static int if_packed_commit_add_to_list(const struct object_id *oid,\n \treturn 0;\n }\n \n+static void close_reachable(struct packed_oid_list *oids)\n+{\n+\tint i;\n+\tstruct rev_info revs;\n+\tstruct commit *commit;\n+\tinit_revisions(&revs, NULL);\n+\tfor (i = 0; i < oids->nr; i++) {\n+\t\tcommit = lookup_commit(&oids->list[i]);\n+\t\tif (commit && !parse_commit(commit))\n+\t\t\trevs.commits = commit_list_insert(commit, &revs.commits);\n+\t}\n+\n+\tif (prepare_revision_walk(&revs))\n+\t\tdie(_(\"revision walk setup failed\"));\n+\n+\twhile ((commit = get_revision(&revs)) != NULL) {\n+\t\tALLOC_GROW(oids->list, oids->nr + 1, oids->alloc);\n+\t\toidcpy(&oids->list[oids->nr], &(commit->object.oid));\n+\t\t(oids->nr)++;\n+\t}\n+}\n+\n char *write_commit_graph(const char *obj_dir)\n {\n \tstruct packed_oid_list oids;\n@@ -411,6 +433,7 @@ char *write_commit_graph(const char *obj_dir)\n \tALLOC_ARRAY(oids.list, oids.alloc);\n \n \tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n+\tclose_reachable(&oids);\n \n \tQSORT(oids.list, oids.nr, commit_compare);\n \n-- \n2.7.4\n\n"},{"id":"339628","messageId":"1519066406-81663-12-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1519066406-81663-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v4 11/13] commit: integrate commit graph with commit parsing","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-19T18:53:24Z","receivedAt":"2018-02-19T18:53:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach Git to inspect a commit graph file to supply the contents of a\nstruct commit when calling parse_commit_gently(). This implementation\nsatisfies all post-conditions on the struct commit, including loading\nparents, the root tree, and the commit date. The only loosely-expected\ncondition is that the commit buffer is loaded into the cache. This\nwas checked in log-tree.c:show_log(), but the \"return;\" on failure\nproduced unexpected results (i.e. the message line was never terminated).\nThe new behavior of loading the buffer when needed prevents the\nunexpected behavior.\n\nIf core.commitGraph is false, then do not check graph files.\n\nIn test script t5318-commit-graph.sh, add output-matching conditions on\nread-only graph operations.\n\nBy loading commits from the graph instead of parsing commit buffers, we\nsave a lot of time on long commit walks. Here are some performance\nresults for a copy of the Linux repository where 'master' has 704,766\nreachable commits and is behind 'origin/master' by 19,610 commits.\n\n| Command | Before | After | Rel % |\n|----------------------------------|--------|--------|-------|\n| log --oneline --topo-order -1000 | 5.9s | 0.7s | -88% |\n| branch -vv | 0.42s | 0.27s | -35% |\n| rev-list --all | 6.4s | 1.0s | -84% |\n| rev-list --all --objects | 32.6s | 27.6s | -15% |\n\nSigned-off-by: Derrick Stolee \n---\n alloc.c | 1 +\n commit-graph.c | 148 ++++++++++++++++++++++++++++++++++++++++++++++++\n commit-graph.h | 18 +++++-\n commit.c | 3 +\n commit.h | 3 +\n log-tree.c | 3 +-\n t/t5318-commit-graph.sh | 45 ++++++++++++++-\n 7 files changed, 216 insertions(+), 5 deletions(-)\n\ndiff --git a/alloc.c b/alloc.c\nindex 12afadf..cf4f8b6 100644\n--- a/alloc.c\n+++ b/alloc.c\n@@ -93,6 +93,7 @@ void *alloc_commit_node(void)\n \tstruct commit *c = alloc_node(&commit_state, sizeof(struct commit));\n \tc->object.type = OBJ_COMMIT;\n \tc->index = alloc_commit_index();\n+\tc->graph_pos = COMMIT_NOT_FROM_GRAPH;\n \treturn c;\n }\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 00bd73a..ea07b47 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -38,6 +38,9 @@\n #define GRAPH_MIN_SIZE (GRAPH_CHUNKLOOKUP_SIZE + GRAPH_FANOUT_SIZE + \\\n \t\t\tGRAPH_OID_LEN + 8)\n \n+/* global storage */\n+struct commit_graph *commit_graph = NULL;\n+\n char *get_graph_latest_filename(const char *obj_dir)\n {\n \tstruct strbuf fname = STRBUF_INIT;\n@@ -184,6 +187,150 @@ struct commit_graph *load_commit_graph_one(const char *graph_file)\n \treturn graph;\n }\n \n+static void prepare_commit_graph_one(const char *obj_dir)\n+{\n+\tstruct strbuf graph_file = STRBUF_INIT;\n+\tchar *graph_name;\n+\n+\tif (commit_graph)\n+\t\treturn;\n+\n+\tgraph_name = get_graph_latest_contents(obj_dir);\n+\n+\tif (!graph_name)\n+\t\treturn;\n+\n+\tstrbuf_addf(&graph_file, \"%s/info/%s\", obj_dir, graph_name);\n+\n+\tcommit_graph = load_commit_graph_one(graph_file.buf);\n+\n+\tFREE_AND_NULL(graph_name);\n+\tstrbuf_release(&graph_file);\n+}\n+\n+static int prepare_commit_graph_run_once = 0;\n+void prepare_commit_graph(void)\n+{\n+\tstruct alternate_object_database *alt;\n+\tchar *obj_dir;\n+\n+\tif (prepare_commit_graph_run_once)\n+\t\treturn;\n+\tprepare_commit_graph_run_once = 1;\n+\n+\tobj_dir = get_object_directory();\n+\tprepare_commit_graph_one(obj_dir);\n+\tprepare_alt_odb();\n+\tfor (alt = alt_odb_list; !commit_graph && alt; alt = alt->next)\n+\t\tprepare_commit_graph_one(alt->path);\n+}\n+\n+static void close_commit_graph(void)\n+{\n+\tif (!commit_graph)\n+\t\treturn;\n+\n+\tif (commit_graph->graph_fd >= 0) {\n+\t\tmunmap((void *)commit_graph->data, commit_graph->data_len);\n+\t\tcommit_graph->data = NULL;\n+\t\tclose(commit_graph->graph_fd);\n+\t}\n+\n+\tFREE_AND_NULL(commit_graph);\n+}\n+\n+static int bsearch_graph(struct commit_graph *g, struct object_id *oid, uint32_t *pos)\n+{\n+\treturn bsearch_hash(oid->hash, g->chunk_oid_fanout,\n+\t\t\t g->chunk_oid_lookup, g->hash_len, pos);\n+}\n+\n+static struct commit_list **insert_parent_or_die(struct commit_graph *g,\n+\t\t\t\t\t\t uint64_t pos,\n+\t\t\t\t\t\t struct commit_list **pptr)\n+{\n+\tstruct commit *c;\n+\tstruct object_id oid;\n+\thashcpy(oid.hash, g->chunk_oid_lookup + g->hash_len * pos);\n+\tc = lookup_commit(&oid);\n+\tif (!c)\n+\t\tdie(\"could not find commit %s\", oid_to_hex(&oid));\n+\tc->graph_pos = pos;\n+\treturn &commit_list_insert(c, pptr)->next;\n+}\n+\n+static int fill_commit_in_graph(struct commit *item, struct commit_graph *g, uint32_t pos)\n+{\n+\tstruct object_id oid;\n+\tuint32_t new_parent_pos;\n+\tuint32_t *parent_data_ptr;\n+\tuint64_t date_low, date_high;\n+\tstruct commit_list **pptr;\n+\tconst unsigned char *commit_data = g->chunk_commit_data + (g->hash_len + 16) * pos;\n+\n+\titem->object.parsed = 1;\n+\titem->graph_pos = pos;\n+\n+\thashcpy(oid.hash, commit_data);\n+\titem->tree = lookup_tree(&oid);\n+\n+\tdate_high = ntohl(*(uint32_t*)(commit_data + g->hash_len + 8)) & 0x3;\n+\tdate_low = ntohl(*(uint32_t*)(commit_data + g->hash_len + 12));\n+\titem->date = (timestamp_t)((date_high << 32) | date_low);\n+\n+\tpptr = &item->parents;\n+\n+\tnew_parent_pos = ntohl(*(uint32_t*)(commit_data + g->hash_len));\n+\tif (new_parent_pos == GRAPH_PARENT_NONE)\n+\t\treturn 1;\n+\tpptr = insert_parent_or_die(g, new_parent_pos, pptr);\n+\n+\tnew_parent_pos = ntohl(*(uint32_t*)(commit_data + g->hash_len + 4));\n+\tif (new_parent_pos == GRAPH_PARENT_NONE)\n+\t\treturn 1;\n+\tif (!(new_parent_pos & GRAPH_LARGE_EDGES_NEEDED)) {\n+\t\tpptr = insert_parent_or_die(g, new_parent_pos, pptr);\n+\t\treturn 1;\n+\t}\n+\n+\tparent_data_ptr = (uint32_t*)(g->chunk_large_edges +\n+\t\t\t 4 * (uint64_t)(new_parent_pos & GRAPH_EDGE_LAST_MASK));\n+\tdo {\n+\t\tnew_parent_pos = ntohl(*parent_data_ptr);\n+\t\tpptr = insert_parent_or_die(g,\n+\t\t\t\t\t new_parent_pos & GRAPH_EDGE_LAST_MASK,\n+\t\t\t\t\t pptr);\n+\t\tparent_data_ptr++;\n+\t} while (!(new_parent_pos & GRAPH_LAST_EDGE));\n+\n+\treturn 1;\n+}\n+\n+int parse_commit_in_graph(struct commit *item)\n+{\n+\tif (!core_commit_graph)\n+\t\treturn 0;\n+\tif (item->object.parsed)\n+\t\treturn 1;\n+\n+\tprepare_commit_graph();\n+\tif (commit_graph) {\n+\t\tuint32_t pos;\n+\t\tint found;\n+\t\tif (item->graph_pos != COMMIT_NOT_FROM_GRAPH) {\n+\t\t\tpos = item->graph_pos;\n+\t\t\tfound = 1;\n+\t\t} else {\n+\t\t\tfound = bsearch_graph(commit_graph, &(item->object.oid), &pos);\n+\t\t}\n+\n+\t\tif (found)\n+\t\t\treturn fill_commit_in_graph(item, commit_graph, pos);\n+\t}\n+\n+\treturn 0;\n+}\n+\n static void write_graph_chunk_fanout(struct sha1file *f,\n \t\t\t\t struct commit **commits,\n \t\t\t\t int nr_commits)\n@@ -525,6 +672,7 @@ char *write_commit_graph(const char *obj_dir)\n \tgraph_name = strbuf_detach(&graph_file, NULL);\n \tstrbuf_addf(&graph_file, \"%s/info/%s\", obj_dir, graph_name);\n \n+\tclose_commit_graph();\n \tif (rename(tmp_file.buf, graph_file.buf))\n \t\tdie(\"failed to rename %s to %s\", tmp_file.buf, graph_file.buf);\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex 56215ad..4818838 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -6,7 +6,19 @@\n extern char *get_graph_latest_filename(const char *obj_dir);\n extern char *get_graph_latest_contents(const char *obj_dir);\n \n-struct commit_graph {\n+/*\n+ * Given a commit struct, try to fill the commit struct info, including:\n+ * 1. tree object\n+ * 2. date\n+ * 3. parents.\n+ *\n+ * Returns 1 if and only if the commit was found in the packed graph.\n+ *\n+ * See parse_commit_buffer() for the fallback after this call.\n+ */\n+extern int parse_commit_in_graph(struct commit *item);\n+\n+extern struct commit_graph {\n \tint graph_fd;\n \n \tconst unsigned char *data;\n@@ -21,10 +33,12 @@ struct commit_graph {\n \tconst unsigned char *chunk_oid_lookup;\n \tconst unsigned char *chunk_commit_data;\n \tconst unsigned char *chunk_large_edges;\n-};\n+} *commit_graph;\n \n extern struct commit_graph *load_commit_graph_one(const char *graph_file);\n \n+extern void prepare_commit_graph(void);\n+\n extern char *write_commit_graph(const char *obj_dir);\n \n #endif\ndiff --git a/commit.c b/commit.c\nindex cab8d44..a8b464d 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"tag.h\"\n #include \"commit.h\"\n+#include \"commit-graph.h\"\n #include \"pkt-line.h\"\n #include \"utf8.h\"\n #include \"diff.h\"\n@@ -385,6 +386,8 @@ int parse_commit_gently(struct commit *item, int quiet_on_missing)\n \t\treturn -1;\n \tif (item->object.parsed)\n \t\treturn 0;\n+\tif (parse_commit_in_graph(item))\n+\t\treturn 0;\n \tbuffer = read_sha1_file(item->object.oid.hash, &type, &size);\n \tif (!buffer)\n \t\treturn quiet_on_missing ? -1 :\ndiff --git a/commit.h b/commit.h\nindex 99a3fea..57963d8 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -8,6 +8,8 @@\n #include \"gpg-interface.h\"\n #include \"string-list.h\"\n \n+#define COMMIT_NOT_FROM_GRAPH 0xFFFFFFFF\n+\n struct commit_list {\n \tstruct commit *item;\n \tstruct commit_list *next;\n@@ -20,6 +22,7 @@ struct commit {\n \ttimestamp_t date;\n \tstruct commit_list *parents;\n \tstruct tree *tree;\n+\tuint32_t graph_pos;\n };\n \n extern int save_commit_buffer;\ndiff --git a/log-tree.c b/log-tree.c\nindex 580b3a9..14735d4 100644\n--- a/log-tree.c\n+++ b/log-tree.c\n@@ -647,8 +647,7 @@ void show_log(struct rev_info *opt)\n \t\tshow_mergetag(opt, commit);\n \t}\n \n-\tif (!get_cached_commit_buffer(commit, NULL))\n-\t\treturn;\n+\tget_commit_buffer(commit, NULL);\n \n \tif (opt->show_notes) {\n \t\tint raw;\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 1d5ec7d..8c6b510 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -8,6 +8,7 @@ test_expect_success 'setup full repo' '\n \tmkdir full &&\n \tcd full &&\n \tgit init &&\n+\tgit config core.commitGraph true &&\n \tobjdir=\".git/objects\"\n '\n \n@@ -25,6 +26,27 @@ test_expect_success 'create commits and repack' '\n \tgit repack\n '\n \n+graph_git_two_modes() {\n+\tgit -c core.graph=true $1 >output\n+\tgit -c core.graph=false $1 >expect\n+\ttest_cmp output expect\n+}\n+\n+graph_git_behavior() {\n+\tMSG=$1\n+\tBRANCH=$2\n+\tCOMPARE=$3\n+\ttest_expect_success \"check normal git operations: $MSG\" '\n+\t\tgraph_git_two_modes \"log --oneline $BRANCH\" &&\n+\t\tgraph_git_two_modes \"log --topo-order $BRANCH\" &&\n+\t\tgraph_git_two_modes \"log --graph $COMPARE..$BRANCH\" &&\n+\t\tgraph_git_two_modes \"branch -vv\" &&\n+\t\tgraph_git_two_modes \"merge-base -a $BRANCH $COMPARE\"\n+\t'\n+}\n+\n+graph_git_behavior 'no graph' commits/3 commits/1\n+\n graph_read_expect() {\n \tOPTIONAL=\"\"\n \tNUM_CHUNKS=3\n@@ -49,6 +71,8 @@ test_expect_success 'write graph' '\n \ttest_cmp expect output\n '\n \n+graph_git_behavior 'graph exists, no head' commits/3 commits/1\n+\n test_expect_success 'Add more commits' '\n \tgit reset --hard commits/1 &&\n \tfor i in $(test_seq 4 5)\n@@ -84,7 +108,6 @@ test_expect_success 'Add more commits' '\n # |___/____/\n # 1\n \n-\n test_expect_success 'write graph with merges' '\n \tgraph2=$(git commit-graph write --set-latest)&&\n \ttest_path_is_file $objdir/info/$graph2 &&\n@@ -96,6 +119,10 @@ test_expect_success 'write graph with merges' '\n \ttest_cmp expect output\n '\n \n+graph_git_behavior 'merge 1 vs 2' merge/1 merge/2\n+graph_git_behavior 'merge 1 vs 3' merge/1 merge/3\n+graph_git_behavior 'merge 2 vs 3' merge/2 merge/3\n+\n test_expect_success 'Add one more commit' '\n \ttest_commit 8 &&\n \tgit branch commits/8 &&\n@@ -116,6 +143,9 @@ test_expect_success 'Add one more commit' '\n # |___/____/\n # 1\n \n+graph_git_behavior 'mixed mode, commit 8 vs merge 1' commits/8 merge/1\n+graph_git_behavior 'mixed mode, commit 8 vs merge 2' commits/8 merge/2\n+\n test_expect_success 'write graph with new commit' '\n \tgraph3=$(git commit-graph write --set-latest --delete-expired) &&\n \ttest_path_is_file $objdir/info/$graph3 &&\n@@ -129,6 +159,9 @@ test_expect_success 'write graph with new commit' '\n \ttest_cmp expect output\n '\n \n+graph_git_behavior 'full graph, commit 8 vs merge 1' commits/8 merge/1\n+graph_git_behavior 'full graph, commit 8 vs merge 2' commits/8 merge/2\n+\n test_expect_success 'write graph with nothing new' '\n \tgraph4=$(git commit-graph write --set-latest --delete-expired) &&\n \ttest_path_is_file $objdir/info/$graph4 &&\n@@ -144,13 +177,20 @@ test_expect_success 'write graph with nothing new' '\n \ttest_cmp expect output\n '\n \n+graph_git_behavior 'cleared graph, commit 8 vs merge 1' commits/8 merge/1\n+graph_git_behavior 'cleared graph, commit 8 vs merge 2' commits/8 merge/2\n+\n test_expect_success 'setup bare repo' '\n \tcd .. &&\n \tgit clone --bare --no-local full bare &&\n \tcd bare &&\n+\tgit config core.commitGraph true &&\n \tbaredir=\"./objects\"\n '\n \n+graph_git_behavior 'bare repo, commit 8 vs merge 1' commits/8 merge/1\n+graph_git_behavior 'bare repo, commit 8 vs merge 2' commits/8 merge/2\n+\n test_expect_success 'write graph in bare repo' '\n \tgraphbare=$(git commit-graph write --set-latest) &&\n \ttest_path_is_file $baredir/info/$graphbare &&\n@@ -162,5 +202,8 @@ test_expect_success 'write graph in bare repo' '\n \ttest_cmp expect output\n '\n \n+graph_git_behavior 'bare repo with graph, commit 8 vs merge 1' commits/8 merge/1\n+graph_git_behavior 'bare repo with graph, commit 8 vs merge 2' commits/8 merge/2\n+\n test_done\n \n-- \n2.7.4\n\n"},{"id":"339629","messageId":"1519066406-81663-13-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1519066406-81663-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v4 12/13] commit-graph: read only from specific pack-indexes","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-19T18:53:25Z","receivedAt":"2018-02-19T18:53:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach git-commit-graph to inspect the objects only in a certain list\nof pack-indexes within the given pack directory. This allows updating\nthe commit graph iteratively, since we add all commits stored in a\nprevious commit graph.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 11 +++++++++++\n builtin/commit-graph.c | 32 +++++++++++++++++++++++++++++---\n commit-graph.c | 26 ++++++++++++++++++++++++--\n commit-graph.h | 4 +++-\n packfile.c | 4 ++--\n packfile.h | 2 ++\n t/t5318-commit-graph.sh | 16 ++++++++++++++++\n 7 files changed, 87 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex b9b4031..93d50d1 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -42,6 +42,10 @@ With the `--delete-expired` option, delete the graph files in the pack\n directory that are not referred to by the graph-latest file. To avoid race\n conditions, do not delete the file previously referred to by the\n graph-latest file if it is updated by the `--set-latest` option.\n++\n+With the `--stdin-packs` option, generate the new commit graph by\n+walking objects only in the specified packfiles and any commits in\n+the existing graph-head.\n \n 'read'::\n \n@@ -68,6 +72,13 @@ $ git commit-graph write\n $ git commit-graph write --set-latest --delete-expired\n ------------------------------------------------\n \n+* Write a graph file, extending the current graph file using commits\n+* in , update graph-latest, and delete stale graph files.\n++\n+------------------------------------------------\n+$ echo | git commit-graph write --set-latest --delete-expired --stdin-packs\n+------------------------------------------------\n+\n * Read basic information from a graph file.\n +\n ------------------------------------------------\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex fd99169..5f08c40 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -8,7 +8,7 @@\n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ] [--file=]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--set-latest] [--delete-expired]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--set-latest] [--delete-expired] [--stdin-packs]\"),\n \tNULL\n };\n \n@@ -18,7 +18,7 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--set-latest] [--delete-expired]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--set-latest] [--delete-expired] [--stdin-packs]\"),\n \tNULL\n };\n \n@@ -27,6 +27,7 @@ static struct opts_commit_graph {\n \tconst char *graph_file;\n \tint set_latest;\n \tint delete_expired;\n+\tint stdin_packs;\n } opts;\n \n static int graph_read(int argc, const char **argv)\n@@ -149,6 +150,11 @@ static int graph_write(int argc, const char **argv)\n {\n \tchar *graph_name;\n \tchar *old_graph_name;\n+\tconst char **pack_indexes = NULL;\n+\tint nr_packs = 0;\n+\tconst char **lines = NULL;\n+\tint nr_lines = 0;\n+\tint alloc_lines = 0;\n \n \tstatic struct option builtin_commit_graph_write_options[] = {\n \t\t{ OPTION_STRING, 'o', \"object-dir\", &opts.obj_dir,\n@@ -158,6 +164,8 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"update graph-head to written graph file\")),\n \t\tOPT_BOOL('d', \"delete-expired\", &opts.delete_expired,\n \t\t\tN_(\"delete expired head graph file\")),\n+\t\tOPT_BOOL('s', \"stdin-packs\", &opts.stdin_packs,\n+\t\t\tN_(\"only scan packfiles listed by stdin\")),\n \t\tOPT_END(),\n \t};\n \n@@ -170,7 +178,25 @@ static int graph_write(int argc, const char **argv)\n \n \told_graph_name = get_graph_latest_contents(opts.obj_dir);\n \n-\tgraph_name = write_commit_graph(opts.obj_dir);\n+\tif (opts.stdin_packs) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tnr_lines = 0;\n+\t\talloc_lines = 128;\n+\t\tALLOC_ARRAY(lines, alloc_lines);\n+\n+\t\twhile (strbuf_getline(&buf, stdin) != EOF) {\n+\t\t\tALLOC_GROW(lines, nr_lines + 1, alloc_lines);\n+\t\t\tlines[nr_lines++] = buf.buf;\n+\t\t\tstrbuf_detach(&buf, NULL);\n+\t\t}\n+\n+\t\tpack_indexes = lines;\n+\t\tnr_packs = nr_lines;\n+\t}\n+\n+\tgraph_name = write_commit_graph(opts.obj_dir,\n+\t\t\t\t\tpack_indexes,\n+\t\t\t\t\tnr_packs);\n \n \tif (graph_name) {\n \t\tif (opts.set_latest)\ndiff --git a/commit-graph.c b/commit-graph.c\nindex ea07b47..943192c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -554,7 +554,9 @@ static void close_reachable(struct packed_oid_list *oids)\n \t}\n }\n \n-char *write_commit_graph(const char *obj_dir)\n+char *write_commit_graph(const char *obj_dir,\n+\t\t\t const char **pack_indexes,\n+\t\t\t int nr_packs)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n@@ -579,7 +581,27 @@ char *write_commit_graph(const char *obj_dir)\n \t\toids.alloc = 1024;\n \tALLOC_ARRAY(oids.list, oids.alloc);\n \n-\tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n+\tif (pack_indexes) {\n+\t\tstruct strbuf packname = STRBUF_INIT;\n+\t\tint dirlen;\n+\t\tstrbuf_addf(&packname, \"%s/pack/\", obj_dir);\n+\t\tdirlen = packname.len;\n+\t\tfor (i = 0; i < nr_packs; i++) {\n+\t\t\tstruct packed_git *p;\n+\t\t\tstrbuf_setlen(&packname, dirlen);\n+\t\t\tstrbuf_addstr(&packname, pack_indexes[i]);\n+\t\t\tp = add_packed_git(packname.buf, packname.len, 1);\n+\t\t\tif (!p)\n+\t\t\t\tdie(\"error adding pack %s\", packname.buf);\n+\t\t\tif (open_pack_index(p))\n+\t\t\t\tdie(\"error opening index for %s\", packname.buf);\n+\t\t\tfor_each_object_in_pack(p, if_packed_commit_add_to_list, &oids);\n+\t\t\tclose_pack(p);\n+\t\t}\n+\t}\n+\telse\n+\t\tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n+\n \tclose_reachable(&oids);\n \n \tQSORT(oids.list, oids.nr, commit_compare);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 4818838..5617842 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -39,7 +39,9 @@ extern struct commit_graph *load_commit_graph_one(const char *graph_file);\n \n extern void prepare_commit_graph(void);\n \n-extern char *write_commit_graph(const char *obj_dir);\n+extern char *write_commit_graph(const char *obj_dir,\n+\t\t\t\tconst char **pack_indexes,\n+\t\t\t\tint nr_packs);\n \n #endif\n \ndiff --git a/packfile.c b/packfile.c\nindex 59648a1..b9ad7b1 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -299,7 +299,7 @@ void close_pack_index(struct packed_git *p)\n \t}\n }\n \n-static void close_pack(struct packed_git *p)\n+void close_pack(struct packed_git *p)\n {\n \tclose_pack_windows(p);\n \tclose_pack_fd(p);\n@@ -1839,7 +1839,7 @@ int has_pack_index(const unsigned char *sha1)\n \treturn 1;\n }\n \n-static int for_each_object_in_pack(struct packed_git *p, each_packed_object_fn cb, void *data)\n+int for_each_object_in_pack(struct packed_git *p, each_packed_object_fn cb, void *data)\n {\n \tuint32_t i;\n \tint r = 0;\ndiff --git a/packfile.h b/packfile.h\nindex 0cdeb54..9281e90 100644\n--- a/packfile.h\n+++ b/packfile.h\n@@ -61,6 +61,7 @@ extern void close_pack_index(struct packed_git *);\n \n extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);\n extern void close_pack_windows(struct packed_git *);\n+extern void close_pack(struct packed_git *);\n extern void close_all_packs(void);\n extern void unuse_pack(struct pack_window **);\n extern void clear_delta_base_cache(void);\n@@ -133,6 +134,7 @@ typedef int each_packed_object_fn(const struct object_id *oid,\n \t\t\t\t struct packed_git *pack,\n \t\t\t\t uint32_t pos,\n \t\t\t\t void *data);\n+extern int for_each_object_in_pack(struct packed_git *p, each_packed_object_fn, void *data);\n extern int for_each_packed_object(each_packed_object_fn, void *, unsigned flags);\n \n #endif\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 8c6b510..5bd1f77 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -180,6 +180,22 @@ test_expect_success 'write graph with nothing new' '\n graph_git_behavior 'cleared graph, commit 8 vs merge 1' commits/8 merge/1\n graph_git_behavior 'cleared graph, commit 8 vs merge 2' commits/8 merge/2\n \n+test_expect_success 'build graph from latest pack with closure' '\n+\trm $objdir/info/graph-latest &&\n+\tgraph5=$(cat new-idx | git commit-graph write --set-latest --delete-expired --stdin-packs) &&\n+\ttest_path_is_file $objdir/info/$graph5 &&\n+\ttest_path_is_missing $objdir/info/$graph4 &&\n+\ttest_path_is_file $objdir/info/graph-latest &&\n+\tprintf $graph5 >expect &&\n+\ttest_cmp expect $objdir/info/graph-latest &&\n+\tgit commit-graph read --file=$graph5 >output &&\n+\tgraph_read_expect \"9\" \"large_edges\" &&\n+\ttest_cmp expect output\n+'\n+\n+graph_git_behavior 'graph from pack, commit 8 vs merge 1' commits/8 merge/1\n+graph_git_behavior 'graph from pack, commit 8 vs merge 2' commits/8 merge/2\n+\n test_expect_success 'setup bare repo' '\n \tcd .. &&\n \tgit clone --bare --no-local full bare &&\n-- \n2.7.4\n\n"},{"id":"339630","messageId":"1519066406-81663-14-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1519066406-81663-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v4 13/13] commit-graph: build graph from starting commits","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-19T18:53:26Z","receivedAt":"2018-02-19T18:54:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach git-commit-graph to read commits from stdin when the\n--stdin-commits flag is specified. Commits reachable from these\ncommits are added to the graph. This is a much faster way to construct\nthe graph than inspecting all packed objects, but is restricted to\nknown tips.\n\nFor the Linux repository, 700,000+ commits were added to the graph\nfile starting from 'master' in 7-9 seconds, depending on the number\nof packfiles in the repo (1, 24, or 120). If a commit graph file\nalready exists (and core.commitGraph=true), then this operation takes\nonly 1.8 seconds due to less time spent parsing commits.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 15 ++++++++++++++-\n builtin/commit-graph.c | 27 +++++++++++++++++++++------\n commit-graph.c | 26 ++++++++++++++++++++++++--\n commit-graph.h | 4 +++-\n t/t5318-commit-graph.sh | 19 +++++++++++++++++++\n 5 files changed, 81 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 93d50d1..43ac74b 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -45,7 +45,12 @@ graph-latest file if it is updated by the `--set-latest` option.\n +\n With the `--stdin-packs` option, generate the new commit graph by\n walking objects only in the specified packfiles and any commits in\n-the existing graph-head.\n+the existing graph-head. (Cannot be combined with --stdin-commits.)\n++\n+With the `--stdin-commits` option, generate the new commit graph by\n+walking commits starting at the commits specified in stdin as a list\n+of OIDs in hex, one OID per line. (Cannot be combined with\n+--stdin-packs.)\n \n 'read'::\n \n@@ -79,6 +84,14 @@ $ git commit-graph write --set-latest --delete-expired\n $ echo | git commit-graph write --set-latest --delete-expired --stdin-packs\n ------------------------------------------------\n \n+* Write a graph file, extending the current graph file using all\n+* commits reachable from refs/heads/*, update graph-latest, and delete\n+* stale graph files.\n++\n+------------------------------------------------\n+$ git show-ref -s | git commit-graph write --set-latest --delete-expired --stdin-commits\n+------------------------------------------------\n+\n * Read basic information from a graph file.\n +\n ------------------------------------------------\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 5f08c40..9b92549 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -8,7 +8,7 @@\n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ] [--file=]\"),\n-\tN_(\"git commit-graph write [--object-dir ] [--set-latest] [--delete-expired] [--stdin-packs]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--set-latest] [--delete-expired] [--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -18,7 +18,7 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ] [--set-latest] [--delete-expired] [--stdin-packs]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--set-latest] [--delete-expired] [--stdin-packs|--stdin-commits]\"),\n \tNULL\n };\n \n@@ -28,6 +28,7 @@ static struct opts_commit_graph {\n \tint set_latest;\n \tint delete_expired;\n \tint stdin_packs;\n+\tint stdin_commits;\n } opts;\n \n static int graph_read(int argc, const char **argv)\n@@ -152,6 +153,8 @@ static int graph_write(int argc, const char **argv)\n \tchar *old_graph_name;\n \tconst char **pack_indexes = NULL;\n \tint nr_packs = 0;\n+\tconst char **commit_hex = NULL;\n+\tint nr_commits = 0;\n \tconst char **lines = NULL;\n \tint nr_lines = 0;\n \tint alloc_lines = 0;\n@@ -166,6 +169,8 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"delete expired head graph file\")),\n \t\tOPT_BOOL('s', \"stdin-packs\", &opts.stdin_packs,\n \t\t\tN_(\"only scan packfiles listed by stdin\")),\n+\t\tOPT_BOOL('C', \"stdin-commits\", &opts.stdin_commits,\n+\t\t\tN_(\"start walk at commits listed by stdin\")),\n \t\tOPT_END(),\n \t};\n \n@@ -173,12 +178,14 @@ static int graph_write(int argc, const char **argv)\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\n \n+\tif (opts.stdin_packs && opts.stdin_commits)\n+\t\tdie(_(\"cannot use both --stdin-commits and --stdin-packs\"));\n \tif (!opts.obj_dir)\n \t\topts.obj_dir = get_object_directory();\n \n \told_graph_name = get_graph_latest_contents(opts.obj_dir);\n \n-\tif (opts.stdin_packs) {\n+\tif (opts.stdin_packs || opts.stdin_commits) {\n \t\tstruct strbuf buf = STRBUF_INIT;\n \t\tnr_lines = 0;\n \t\talloc_lines = 128;\n@@ -190,13 +197,21 @@ static int graph_write(int argc, const char **argv)\n \t\t\tstrbuf_detach(&buf, NULL);\n \t\t}\n \n-\t\tpack_indexes = lines;\n-\t\tnr_packs = nr_lines;\n+\t\tif (opts.stdin_packs) {\n+\t\t\tpack_indexes = lines;\n+\t\t\tnr_packs = nr_lines;\n+\t\t}\n+\t\tif (opts.stdin_commits) {\n+\t\t\tcommit_hex = lines;\n+\t\t\tnr_commits = nr_lines;\n+\t\t}\n \t}\n \n \tgraph_name = write_commit_graph(opts.obj_dir,\n \t\t\t\t\tpack_indexes,\n-\t\t\t\t\tnr_packs);\n+\t\t\t\t\tnr_packs,\n+\t\t\t\t\tcommit_hex,\n+\t\t\t\t\tnr_commits);\n \n \tif (graph_name) {\n \t\tif (opts.set_latest)\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 943192c..b9e938c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -556,7 +556,9 @@ static void close_reachable(struct packed_oid_list *oids)\n \n char *write_commit_graph(const char *obj_dir,\n \t\t\t const char **pack_indexes,\n-\t\t\t int nr_packs)\n+\t\t\t int nr_packs,\n+\t\t\t const char **commit_hex,\n+\t\t\t int nr_commits)\n {\n \tstruct packed_oid_list oids;\n \tstruct packed_commit_list commits;\n@@ -599,7 +601,27 @@ char *write_commit_graph(const char *obj_dir,\n \t\t\tclose_pack(p);\n \t\t}\n \t}\n-\telse\n+\n+\tif (commit_hex) {\n+\t\tfor (i = 0; i < nr_commits; i++) {\n+\t\t\tconst char *end;\n+\t\t\tstruct object_id oid;\n+\t\t\tstruct commit *result;\n+\n+\t\t\tif (commit_hex[i] && parse_oid_hex(commit_hex[i], &oid, &end))\n+\t\t\t\tcontinue;\n+\n+\t\t\tresult = lookup_commit_reference_gently(&oid, 1);\n+\n+\t\t\tif (result) {\n+\t\t\t\tALLOC_GROW(oids.list, oids.nr + 1, oids.alloc);\n+\t\t\t\toidcpy(&oids.list[oids.nr], &(result->object.oid));\n+\t\t\t\toids.nr++;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tif (!pack_indexes && !commit_hex)\n \t\tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n \n \tclose_reachable(&oids);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 5617842..2582a3c 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -41,7 +41,9 @@ extern void prepare_commit_graph(void);\n \n extern char *write_commit_graph(const char *obj_dir,\n \t\t\t\tconst char **pack_indexes,\n-\t\t\t\tint nr_packs);\n+\t\t\t\tint nr_packs,\n+\t\t\t\tconst char **commit_hex,\n+\t\t\t\tint nr_commits);\n \n #endif\n \ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 5bd1f77..2ed6b19 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -196,6 +196,25 @@ test_expect_success 'build graph from latest pack with closure' '\n graph_git_behavior 'graph from pack, commit 8 vs merge 1' commits/8 merge/1\n graph_git_behavior 'graph from pack, commit 8 vs merge 2' commits/8 merge/2\n \n+test_expect_success 'build graph from commits with closure' '\n+\tgit tag -a -m \"merge\" tag/merge merge/2 &&\n+\tgit rev-parse tag/merge >commits-in &&\n+\tgit rev-parse merge/1 >>commits-in &&\n+\trm $objdir/info/graph-latest &&\n+\tgraph6=$(cat commits-in | git commit-graph write --set-latest --delete-expired --stdin-commits) &&\n+\ttest_path_is_file $objdir/info/$graph6 &&\n+\ttest_path_is_missing $objdir/info/$graph5 &&\n+\ttest_path_is_file $objdir/info/graph-latest &&\n+\tprintf $graph6 >expect &&\n+\ttest_cmp expect $objdir/info/graph-latest &&\n+\tgit commit-graph read --file=$graph6 >output &&\n+\tgraph_read_expect \"6\" &&\n+\ttest_cmp expect output\n+'\n+\n+graph_git_behavior 'graph from commits, commit 8 vs merge 1' commits/8 merge/1\n+graph_git_behavior 'graph from commits, commit 8 vs merge 2' commits/8 merge/2\n+\n test_expect_success 'setup bare repo' '\n \tcd .. &&\n \tgit clone --bare --no-local full bare &&\n-- \n2.7.4\n\n"},{"id":"339631","messageId":"1519066406-81663-8-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1519066406-81663-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v4 07/13] commit-graph: implement --set-latest","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-19T18:53:20Z","receivedAt":"2018-02-19T18:54:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"It is possible to have multiple commit graph files in a directory, but\nonly one is important at a time.\n\nUse a 'graph-latest' file to point to the important file. Teach\ngit-commit-graph to write 'graph-latest' when given the \"--set-latest\"\noption. Using this 'graph-latest' file is more robust than relying on\ndirectory scanning and modified times.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 10 ++++++++++\n builtin/commit-graph.c | 26 ++++++++++++++++++++++++--\n commit-graph.c | 7 +++++++\n commit-graph.h | 2 ++\n t/t5318-commit-graph.sh | 24 +++++++++++++++++++-----\n 5 files changed, 62 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 6d26e56..dc948c5 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -34,6 +34,9 @@ COMMANDS\n Write a commit graph file based on the commits found in packfiles.\n Includes all commits from the existing commit graph file. Outputs the\n resulting filename.\n++\n+With `--set-latest` option, update the graph-latest file to point\n+to the written graph file.\n \n 'read'::\n \n@@ -53,6 +56,13 @@ EXAMPLES\n $ git commit-graph write\n ------------------------------------------------\n \n+* Write a graph file for the packed commits in your local .git folder\n+* and update graph-latest.\n++\n+------------------------------------------------\n+$ git commit-graph write --set-latest\n+------------------------------------------------\n+\n * Read basic information from a graph file.\n +\n ------------------------------------------------\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 28cd097..bf86172 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -8,7 +8,7 @@\n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n \tN_(\"git commit-graph read [--object-dir ] [--file=]\"),\n-\tN_(\"git commit-graph write [--object-dir ]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--set-latest]\"),\n \tNULL\n };\n \n@@ -18,13 +18,14 @@ static const char * const builtin_commit_graph_read_usage[] = {\n };\n \n static const char * const builtin_commit_graph_write_usage[] = {\n-\tN_(\"git commit-graph write [--object-dir ]\"),\n+\tN_(\"git commit-graph write [--object-dir ] [--set-latest]\"),\n \tNULL\n };\n \n static struct opts_commit_graph {\n \tconst char *obj_dir;\n \tconst char *graph_file;\n+\tint set_latest;\n } opts;\n \n static int graph_read(int argc, const char **argv)\n@@ -81,6 +82,22 @@ static int graph_read(int argc, const char **argv)\n \treturn 0;\n }\n \n+static void set_latest_file(const char *obj_dir, const char *graph_file)\n+{\n+\tint fd;\n+\tstruct lock_file lk = LOCK_INIT;\n+\tchar *latest_fname = get_graph_latest_filename(obj_dir);\n+\n+\tfd = hold_lock_file_for_update(&lk, latest_fname, LOCK_DIE_ON_ERROR);\n+\tFREE_AND_NULL(latest_fname);\n+\n+\tif (fd < 0)\n+\t\tdie_errno(\"unable to open graph-head\");\n+\n+\twrite_in_full(fd, graph_file, strlen(graph_file));\n+\tcommit_lock_file(&lk);\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tchar *graph_name;\n@@ -89,6 +106,8 @@ static int graph_write(int argc, const char **argv)\n \t\t{ OPTION_STRING, 'o', \"object-dir\", &opts.obj_dir,\n \t\t\tN_(\"dir\"),\n \t\t\tN_(\"The object directory to store the graph\") },\n+\t\tOPT_BOOL('u', \"set-latest\", &opts.set_latest,\n+\t\t\tN_(\"update graph-head to written graph file\")),\n \t\tOPT_END(),\n \t};\n \n@@ -102,6 +121,9 @@ static int graph_write(int argc, const char **argv)\n \tgraph_name = write_commit_graph(opts.obj_dir);\n \n \tif (graph_name) {\n+\t\tif (opts.set_latest)\n+\t\t\tset_latest_file(opts.obj_dir, graph_name);\n+\n \t\tprintf(\"%s\\n\", graph_name);\n \t\tFREE_AND_NULL(graph_name);\n \t}\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 2a8594f..5ee0805 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -38,6 +38,13 @@\n #define GRAPH_MIN_SIZE (GRAPH_CHUNKLOOKUP_SIZE + GRAPH_FANOUT_SIZE + \\\n \t\t\tGRAPH_OID_LEN + 8)\n \n+char *get_graph_latest_filename(const char *obj_dir)\n+{\n+\tstruct strbuf fname = STRBUF_INIT;\n+\tstrbuf_addf(&fname, \"%s/info/graph-latest\", obj_dir);\n+\treturn strbuf_detach(&fname, 0);\n+}\n+\n static struct commit_graph *alloc_commit_graph(void)\n {\n \tstruct commit_graph *g = xmalloc(sizeof(*g));\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 9093b97..ae24b3a 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -3,6 +3,8 @@\n \n #include \"git-compat-util.h\"\n \n+extern char *get_graph_latest_filename(const char *obj_dir);\n+\n struct commit_graph {\n \tint graph_fd;\n \ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 893fa24..cad9d90 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -12,7 +12,8 @@ test_expect_success 'setup full repo' '\n '\n \n test_expect_success 'write graph with no packs' '\n-\tgit commit-graph write --object-dir .\n+\tgit commit-graph write --object-dir . &&\n+\ttest_path_is_missing info/graph-latest\n '\n \n test_expect_success 'create commits and repack' '\n@@ -42,6 +43,7 @@ graph_read_expect() {\n test_expect_success 'write graph' '\n \tgraph1=$(git commit-graph write) &&\n \ttest_path_is_file $objdir/info/$graph1 &&\n+\ttest_path_is_missing $objdir/info/graph-latest &&\n \tgit commit-graph read --file=$graph1 >output &&\n \tgraph_read_expect \"3\" &&\n \ttest_cmp expect output\n@@ -84,8 +86,11 @@ test_expect_success 'Add more commits' '\n \n \n test_expect_success 'write graph with merges' '\n-\tgraph2=$(git commit-graph write)&&\n+\tgraph2=$(git commit-graph write --set-latest)&&\n \ttest_path_is_file $objdir/info/$graph2 &&\n+\ttest_path_is_file $objdir/info/graph-latest &&\n+\tprintf $graph2 >expect &&\n+\ttest_cmp expect $objdir/info/graph-latest &&\n \tgit commit-graph read --file=$graph2 >output &&\n \tgraph_read_expect \"10\" \"large_edges\" &&\n \ttest_cmp expect output\n@@ -112,19 +117,25 @@ test_expect_success 'Add one more commit' '\n # 1\n \n test_expect_success 'write graph with new commit' '\n-\tgraph3=$(git commit-graph write) &&\n+\tgraph3=$(git commit-graph write --set-latest) &&\n \ttest_path_is_file $objdir/info/$graph3 &&\n+\ttest_path_is_file $objdir/info/graph-latest &&\n+\tprintf $graph3 >expect &&\n+\ttest_cmp expect $objdir/info/graph-latest &&\n \tgit commit-graph read --file=$graph3 >output &&\n \tgraph_read_expect \"11\" \"large_edges\" &&\n \ttest_cmp expect output\n '\n \n test_expect_success 'write graph with nothing new' '\n-\tgraph4=$(git commit-graph write) &&\n+\tgraph4=$(git commit-graph write --set-latest) &&\n \ttest_path_is_file $objdir/info/$graph4 &&\n \tprintf $graph3 >expect &&\n \tprintf $graph4 >output &&\n \ttest_cmp expect output &&\n+\ttest_path_is_file $objdir/info/graph-latest &&\n+\tprintf $graph4 >expect &&\n+\ttest_cmp expect $objdir/info/graph-latest &&\n \tgit commit-graph read --file=$graph4 >output &&\n \tgraph_read_expect \"11\" \"large_edges\" &&\n \ttest_cmp expect output\n@@ -138,8 +149,11 @@ test_expect_success 'setup bare repo' '\n '\n \n test_expect_success 'write graph in bare repo' '\n-\tgraphbare=$(git commit-graph write) &&\n+\tgraphbare=$(git commit-graph write --set-latest) &&\n \ttest_path_is_file $baredir/info/$graphbare &&\n+\ttest_path_is_file $baredir/info/graph-latest &&\n+\tprintf $graphbare >expect &&\n+\ttest_cmp expect $baredir/info/graph-latest &&\n \tgit commit-graph read --file=$graphbare >output &&\n \tgraph_read_expect \"11\" \"large_edges\" &&\n \ttest_cmp expect output\n-- \n2.7.4\n\n"},{"id":"339632","messageId":"1519066406-81663-3-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1519066406-81663-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v4 02/13] graph: add commit graph design document","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-19T18:53:15Z","receivedAt":"2018-02-19T18:54:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Add Documentation/technical/commit-graph.txt with details of the planned\ncommit graph feature, including future plans.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/technical/commit-graph.txt | 185 +++++++++++++++++++++++++++++++\n 1 file changed, 185 insertions(+)\n create mode 100644 Documentation/technical/commit-graph.txt\n\ndiff --git a/Documentation/technical/commit-graph.txt b/Documentation/technical/commit-graph.txt\nnew file mode 100644\nindex 0000000..e52ab23\n--- /dev/null\n+++ b/Documentation/technical/commit-graph.txt\n@@ -0,0 +1,185 @@\n+Git Commit Graph Design Notes\n+=============================\n+\n+Git walks the commit graph for many reasons, including:\n+\n+1. Listing and filtering commit history.\n+2. Computing merge bases.\n+\n+These operations can become slow as the commit count grows. The merge\n+base calculation shows up in many user-facing commands, such as 'merge-base'\n+or 'status' and can take minutes to compute depending on history shape.\n+\n+There are two main costs here:\n+\n+1. Decompressing and parsing commits.\n+2. Walking the entire graph to avoid topological order mistakes.\n+\n+The commit graph file is a supplemental data structure that accelerates\n+commit graph walks. If a user downgrades or disables the 'core.commitGraph'\n+config setting, then the existing ODB is sufficient. The file is stored\n+either in the .git/objects/info directory or in the info directory of an\n+alternate.\n+\n+The commit graph file stores the commit graph structure along with some\n+extra metadata to speed up graph walks. By listing commit OIDs in lexi-\n+cographic order, we can identify an integer position for each commit and\n+refer to the parents of a commit using those integer positions. We use\n+binary search to find initial commits and then use the integer positions\n+for fast lookups during the walk.\n+\n+A consumer may load the following info for a commit from the graph:\n+\n+1. The commit OID.\n+2. The list of parents, along with their integer position.\n+3. The commit date.\n+4. The root tree OID.\n+5. The generation number (see definition below).\n+\n+Values 1-4 satisfy the requirements of parse_commit_gently().\n+\n+Define the \"generation number\" of a commit recursively as follows:\n+\n+ * A commit with no parents (a root commit) has generation number one.\n+\n+ * A commit with at least one parent has generation number one more than\n+ the largest generation number among its parents.\n+\n+Equivalently, the generation number of a commit A is one more than the\n+length of a longest path from A to a root commit. The recursive definition\n+is easier to use for computation and observing the following property:\n+\n+ If A and B are commits with generation numbers N and M, respectively,\n+ and N <= M, then A cannot reach B. That is, we know without searching\n+ that B is not an ancestor of A because it is further from a root commit\n+ than A.\n+\n+ Conversely, when checking if A is an ancestor of B, then we only need\n+ to walk commits until all commits on the walk boundary have generation\n+ number at most N. If we walk commits using a priority queue seeded by\n+ generation numbers, then we always expand the boundary commit with highest\n+ generation number and can easily detect the stopping condition.\n+\n+This property can be used to significantly reduce the time it takes to\n+walk commits and determine topological relationships. Without generation\n+numbers, the general heuristic is the following:\n+\n+ If A and B are commits with commit time X and Y, respectively, and\n+ X < Y, then A _probably_ cannot reach B.\n+\n+This heuristic is currently used whenever the computation can make\n+mistakes with topological orders (such as \"git log\" with default order),\n+but is not used when the topological order is required (such as merge\n+base calculations, \"git log --graph\").\n+\n+In practice, we expect some commits to be created recently and not stored\n+in the commit graph. We can treat these commits as having \"infinite\"\n+generation number and walk until reaching commits with known generation\n+number.\n+\n+Design Details\n+--------------\n+\n+- A graph file is stored in a file named 'graph-.graph' in the\n+ .git/objects/info directory. This could be stored in the info directory\n+ of an alternate.\n+\n+- The latest graph file name is stored in a 'graph-latest' file next to\n+ the graph files. This allows atomic swaps of latest graph files without\n+ race conditions with concurrent processes.\n+\n+- The core.commitGraph config setting must be on to consume graph files.\n+\n+- The file format includes parameters for the object ID hash function,\n+ so a future change of hash algorithm does not require a change in format.\n+\n+Current Limitations\n+-------------------\n+\n+- Only one graph file is used at one time. This allows the integer position\n+ to seek into the single graph file. It is possible to extend the model\n+ for multiple graph files, but that is currently not part of the design.\n+\n+- .graph files are managed only by the 'commit-graph' builtin. These are not\n+ updated automatically during clone, fetch, repack, or creating new commits.\n+\n+- There is no 'verify' subcommand for the 'commit-graph' builtin to verify\n+ the contents of the graph file agree with the contents in the ODB.\n+\n+- Generation numbers are not computed in the current version. The file\n+ format supports storing them, along with a mechanism to upgrade from\n+ a file without generation numbers to one that uses them.\n+\n+Future Work\n+-----------\n+\n+- The file format includes room for precomputed generation numbers. These\n+ are not currently computed, so all generation numbers will be marked as\n+ 0 (or \"uncomputed\"). A later patch will include this calculation.\n+\n+- The commit graph is currently incompatible with commit grafts. This can be\n+ remedied by duplicating or refactoring the current graft logic.\n+\n+- After computing and storing generation numbers, we must make graph\n+ walks aware of generation numbers to gain the performance benefits they\n+ enable. This will mostly be accomplished by swapping a commit-date-ordered\n+ priority queue with one ordered by generation number. The following\n+ operations are important candidates:\n+\n+ - paint_down_to_common()\n+ - 'log --topo-order'\n+\n+- The graph currently only adds commits to a previously existing graph.\n+ When writing a new graph, we could check that the ODB still contains\n+ the commits and choose to remove the commits that are deleted from the\n+ ODB. For performance reasons, this check should remain optional.\n+\n+- Currently, parse_commit_gently() requires filling in the root tree\n+ object for a commit. This passes through lookup_tree() and consequently\n+ lookup_object(). Also, it calls lookup_commit() when loading the parents.\n+ These method calls check the ODB for object existence, even if the\n+ consumer does not need the content. For example, we do not need the\n+ tree contents when computing merge bases. Now that commit parsing is\n+ removed from the computation time, these lookup operations are the\n+ slowest operations keeping graph walks from being fast. Consider\n+ loading these objects without verifying their existence in the ODB and\n+ only loading them fully when consumers need them. Consider a method\n+ such as \"ensure_tree_loaded(commit)\" that fully loads a tree before\n+ using commit->tree.\n+\n+- The current design uses the 'commit-graph' builtin to generate the graph.\n+ When this feature stabilizes enough to recommend to most users, we should\n+ add automatic graph writes to common operations that create many commits.\n+ For example, one coulde compute a graph on 'clone', 'fetch', or 'repack'\n+ commands.\n+\n+- A server could provide a commit graph file as part of the network protocol\n+ to avoid extra calculations by clients.\n+\n+Related Links\n+-------------\n+[0] https://bugs.chromium.org/p/git/issues/detail?id=8\n+ Chromium work item for: Serialized Commit Graph\n+\n+[1] https://public-inbox.org/git/20110713070517.GC18566@sigill.intra.peff.net/\n+ An abandoned patch that introduced generation numbers.\n+\n+[2] https://public-inbox.org/git/20170908033403.q7e6dj7benasrjes@sigill.intra.peff.net/\n+ Discussion about generation numbers on commits and how they interact\n+ with fsck.\n+\n+[3] https://public-inbox.org/git/20170907094718.b6kuzp2uhvkmwcso@sigill.intra.peff.net/t/#m7a2ea7b355aeda962e6b86404bcbadc648abfbba\n+ More discussion about generation numbers and not storing them inside\n+ commit objects. A valuable quote:\n+\n+ \"I think we should be moving more in the direction of keeping\n+ repo-local caches for optimizations. Reachability bitmaps have been\n+ a big performance win. I think we should be doing the same with our\n+ properties of commits. Not just generation numbers, but making it\n+ cheap to access the graph structure without zlib-inflating whole\n+ commit objects (i.e., packv4 or something like the \"metapacks\" I\n+ proposed a few years ago).\"\n+\n+[4] https://public-inbox.org/git/20180108154822.54829-1-git@jeffhostetler.com/T/#u\n+ A patch to remove the ahead-behind calculation from 'status'.\n+\n-- \n2.7.4\n\n"},{"id":"339633","messageId":"1519066406-81663-5-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1519066406-81663-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v4 04/13] commit-graph: implement write_commit_graph()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-19T18:53:17Z","receivedAt":"2018-02-19T18:54:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach Git to write a commit graph file by checking all packed objects\nto see if they are commits, then store the file in the given object\ndirectory.\n\nSigned-off-by: Derrick Stolee \n---\n Makefile | 1 +\n commit-graph.c | 370 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n commit-graph.h | 7 ++\n 3 files changed, 378 insertions(+)\n create mode 100644 commit-graph.c\n create mode 100644 commit-graph.h\n\ndiff --git a/Makefile b/Makefile\nindex fc40b81..eeaeb6a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -761,6 +761,7 @@ LIB_OBJS += color.o\n LIB_OBJS += column.o\n LIB_OBJS += combine-diff.o\n LIB_OBJS += commit.o\n+LIB_OBJS += commit-graph.o\n LIB_OBJS += compat/obstack.o\n LIB_OBJS += compat/terminal.o\n LIB_OBJS += config.o\ndiff --git a/commit-graph.c b/commit-graph.c\nnew file mode 100644\nindex 0000000..f9e39b0\n--- /dev/null\n+++ b/commit-graph.c\n@@ -0,0 +1,370 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"git-compat-util.h\"\n+#include \"pack.h\"\n+#include \"packfile.h\"\n+#include \"commit.h\"\n+#include \"object.h\"\n+#include \"revision.h\"\n+#include \"sha1-lookup.h\"\n+#include \"commit-graph.h\"\n+\n+#define GRAPH_SIGNATURE 0x43475048 /* \"CGPH\" */\n+#define GRAPH_CHUNKID_OIDFANOUT 0x4f494446 /* \"OIDF\" */\n+#define GRAPH_CHUNKID_OIDLOOKUP 0x4f49444c /* \"OIDL\" */\n+#define GRAPH_CHUNKID_DATA 0x43444154 /* \"CDAT\" */\n+#define GRAPH_CHUNKID_LARGEEDGES 0x45444745 /* \"EDGE\" */\n+\n+#define GRAPH_DATA_WIDTH 36\n+\n+#define GRAPH_VERSION_1 0x1\n+#define GRAPH_VERSION GRAPH_VERSION_1\n+\n+#define GRAPH_OID_VERSION_SHA1 1\n+#define GRAPH_OID_LEN_SHA1 20\n+#define GRAPH_OID_VERSION GRAPH_OID_VERSION_SHA1\n+#define GRAPH_OID_LEN GRAPH_OID_LEN_SHA1\n+\n+#define GRAPH_LARGE_EDGES_NEEDED 0x80000000\n+#define GRAPH_PARENT_MISSING 0x7fffffff\n+#define GRAPH_EDGE_LAST_MASK 0x7fffffff\n+#define GRAPH_PARENT_NONE 0x70000000\n+\n+#define GRAPH_LAST_EDGE 0x80000000\n+\n+#define GRAPH_FANOUT_SIZE (4 * 256)\n+#define GRAPH_CHUNKLOOKUP_WIDTH 12\n+#define GRAPH_CHUNKLOOKUP_SIZE (5 * GRAPH_CHUNKLOOKUP_WIDTH)\n+#define GRAPH_MIN_SIZE (GRAPH_CHUNKLOOKUP_SIZE + GRAPH_FANOUT_SIZE + \\\n+\t\t\tGRAPH_OID_LEN + 8)\n+\n+static void write_graph_chunk_fanout(struct sha1file *f,\n+\t\t\t\t struct commit **commits,\n+\t\t\t\t int nr_commits)\n+{\n+\tuint32_t i, count = 0;\n+\tstruct commit **list = commits;\n+\tstruct commit **last = commits + nr_commits;\n+\n+\t/*\n+\t * Write the first-level table (the list is sorted,\n+\t * but we use a 256-entry lookup to be able to avoid\n+\t * having to do eight extra binary search iterations).\n+\t */\n+\tfor (i = 0; i < 256; i++) {\n+\t\twhile (list < last) {\n+\t\t\tif ((*list)->object.oid.hash[0] != i)\n+\t\t\t\tbreak;\n+\t\t\tcount++;\n+\t\t\tlist++;\n+\t\t}\n+\n+\t\tsha1write_be32(f, count);\n+\t}\n+}\n+\n+static void write_graph_chunk_oids(struct sha1file *f, int hash_len,\n+\t\t\t\t struct commit **commits, int nr_commits)\n+{\n+\tstruct commit **list, **last = commits + nr_commits;\n+\tfor (list = commits; list < last; list++)\n+\t\tsha1write(f, (*list)->object.oid.hash, (int)hash_len);\n+}\n+\n+static int commit_pos(struct commit **commits, int nr_commits,\n+\t\t const struct object_id *oid, uint32_t *pos)\n+{\n+\tuint32_t first = 0, last = nr_commits;\n+\n+\twhile (first < last) {\n+\t\tuint32_t mid = first + (last - first) / 2;\n+\t\tstruct object_id *current;\n+\t\tint cmp;\n+\n+\t\tcurrent = &(commits[mid]->object.oid);\n+\t\tcmp = oidcmp(oid, current);\n+\t\tif (!cmp) {\n+\t\t\t*pos = mid;\n+\t\t\treturn 1;\n+\t\t}\n+\t\tif (cmp > 0) {\n+\t\t\tfirst = mid + 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tlast = mid;\n+\t}\n+\n+\t*pos = first;\n+\treturn 0;\n+}\n+\n+static void write_graph_chunk_data(struct sha1file *f, int hash_len,\n+\t\t\t\t struct commit **commits, int nr_commits)\n+{\n+\tstruct commit **list = commits;\n+\tstruct commit **last = commits + nr_commits;\n+\tuint32_t num_large_edges = 0;\n+\n+\twhile (list < last) {\n+\t\tstruct commit_list *parent;\n+\t\tuint32_t int_id;\n+\t\tuint32_t packedDate[2];\n+\n+\t\tparse_commit(*list);\n+\t\tsha1write(f, (*list)->tree->object.oid.hash, hash_len);\n+\n+\t\tparent = (*list)->parents;\n+\n+\t\tif (!parent)\n+\t\t\tint_id = GRAPH_PARENT_NONE;\n+\t\telse if (!commit_pos(commits, nr_commits,\n+\t\t\t\t &(parent->item->object.oid), &int_id))\n+\t\t\tint_id = GRAPH_PARENT_MISSING;\n+\n+\t\tsha1write_be32(f, int_id);\n+\n+\t\tif (parent)\n+\t\t\tparent = parent->next;\n+\n+\t\tif (!parent)\n+\t\t\tint_id = GRAPH_PARENT_NONE;\n+\t\telse if (parent->next)\n+\t\t\tint_id = GRAPH_LARGE_EDGES_NEEDED | num_large_edges;\n+\t\telse if (!commit_pos(commits, nr_commits,\n+\t\t\t\t &(parent->item->object.oid), &int_id))\n+\t\t\tint_id = GRAPH_PARENT_MISSING;\n+\n+\t\tsha1write_be32(f, int_id);\n+\n+\t\tif (parent && parent->next) {\n+\t\t\tdo {\n+\t\t\t\tnum_large_edges++;\n+\t\t\t\tparent = parent->next;\n+\t\t\t} while (parent);\n+\t\t}\n+\n+\t\tif (sizeof((*list)->date) > 4)\n+\t\t\tpackedDate[0] = htonl(((*list)->date >> 32) & 0x3);\n+\t\telse\n+\t\t\tpackedDate[0] = 0;\n+\n+\t\tpackedDate[1] = htonl((*list)->date);\n+\t\tsha1write(f, packedDate, 8);\n+\n+\t\tlist++;\n+\t}\n+}\n+\n+static void write_graph_chunk_large_edges(struct sha1file *f,\n+\t\t\t\t\t struct commit **commits,\n+\t\t\t\t\t int nr_commits)\n+{\n+\tstruct commit **list = commits;\n+\tstruct commit **last = commits + nr_commits;\n+\tstruct commit_list *parent;\n+\n+\twhile (list < last) {\n+\t\tint num_parents = 0;\n+\t\tfor (parent = (*list)->parents; num_parents < 3 && parent;\n+\t\t parent = parent->next)\n+\t\t\tnum_parents++;\n+\n+\t\tif (num_parents <= 2) {\n+\t\t\tlist++;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/* Since num_parents > 2, this initializer is safe. */\n+\t\tfor (parent = (*list)->parents->next; parent; parent = parent->next) {\n+\t\t\tuint32_t int_id, swap_int_id;\n+\t\t\tuint32_t last_edge = 0;\n+\t\t\tif (!parent->next)\n+\t\t\t\tlast_edge |= GRAPH_LAST_EDGE;\n+\n+\t\t\tif (commit_pos(commits, nr_commits,\n+\t\t\t\t &(parent->item->object.oid),\n+\t\t\t\t &int_id))\n+\t\t\t\tswap_int_id = htonl(int_id | last_edge);\n+\t\t\telse\n+\t\t\t\tswap_int_id = htonl(GRAPH_PARENT_MISSING | last_edge);\n+\n+\t\t\tsha1write(f, &swap_int_id, 4);\n+\t\t}\n+\n+\t\tlist++;\n+\t}\n+}\n+\n+static int commit_compare(const void *_a, const void *_b)\n+{\n+\tstruct object_id *a = (struct object_id *)_a;\n+\tstruct object_id *b = (struct object_id *)_b;\n+\treturn oidcmp(a, b);\n+}\n+\n+struct packed_commit_list {\n+\tstruct commit **list;\n+\tint nr;\n+\tint alloc;\n+};\n+\n+struct packed_oid_list {\n+\tstruct object_id *list;\n+\tint nr;\n+\tint alloc;\n+};\n+\n+static int if_packed_commit_add_to_list(const struct object_id *oid,\n+\t\t\t\t\tstruct packed_git *pack,\n+\t\t\t\t\tuint32_t pos,\n+\t\t\t\t\tvoid *data)\n+{\n+\tstruct packed_oid_list *list = (struct packed_oid_list*)data;\n+\tenum object_type type;\n+\tunsigned long size;\n+\tvoid *inner_data;\n+\toff_t offset = nth_packed_object_offset(pack, pos);\n+\tinner_data = unpack_entry(pack, offset, &type, &size);\n+\n+\tif (inner_data)\n+\t\tfree(inner_data);\n+\n+\tif (type != OBJ_COMMIT)\n+\t\treturn 0;\n+\n+\tALLOC_GROW(list->list, list->nr + 1, list->alloc);\n+\toidcpy(&(list->list[list->nr]), oid);\n+\t(list->nr)++;\n+\n+\treturn 0;\n+}\n+\n+char *write_commit_graph(const char *obj_dir)\n+{\n+\tstruct packed_oid_list oids;\n+\tstruct packed_commit_list commits;\n+\tstruct sha1file *f;\n+\tint i, count_distinct = 0;\n+\tDIR *info_dir;\n+\tstruct strbuf tmp_file = STRBUF_INIT;\n+\tstruct strbuf graph_file = STRBUF_INIT;\n+\tunsigned char final_hash[GIT_MAX_RAWSZ];\n+\tchar *graph_name;\n+\tint fd;\n+\tuint32_t chunk_ids[5];\n+\tuint64_t chunk_offsets[5];\n+\tint num_chunks;\n+\tint num_long_edges;\n+\tstruct commit_list *parent;\n+\n+\toids.nr = 0;\n+\toids.alloc = (int)(0.15 * approximate_object_count());\n+\n+\tif (oids.alloc < 1024)\n+\t\toids.alloc = 1024;\n+\tALLOC_ARRAY(oids.list, oids.alloc);\n+\n+\tfor_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n+\n+\tQSORT(oids.list, oids.nr, commit_compare);\n+\n+\tcount_distinct = 1;\n+\tfor (i = 1; i < oids.nr; i++) {\n+\t\tif (oidcmp(&oids.list[i-1], &oids.list[i]))\n+\t\t\tcount_distinct++;\n+\t}\n+\n+\tcommits.nr = 0;\n+\tcommits.alloc = count_distinct;\n+\tALLOC_ARRAY(commits.list, commits.alloc);\n+\n+\tnum_long_edges = 0;\n+\tfor (i = 0; i < oids.nr; i++) {\n+\t\tint num_parents = 0;\n+\t\tif (i > 0 && !oidcmp(&oids.list[i-1], &oids.list[i]))\n+\t\t\tcontinue;\n+\n+\t\tcommits.list[commits.nr] = lookup_commit(&oids.list[i]);\n+\t\tparse_commit(commits.list[commits.nr]);\n+\n+\t\tfor (parent = commits.list[commits.nr]->parents;\n+\t\t parent; parent = parent->next)\n+\t\t\tnum_parents++;\n+\n+\t\tif (num_parents > 2)\n+\t\t\tnum_long_edges += num_parents - 1;\n+\n+\t\tcommits.nr++;\n+\t}\n+\tnum_chunks = num_long_edges ? 4 : 3;\n+\n+\tstrbuf_addf(&tmp_file, \"%s/info\", obj_dir);\n+\tinfo_dir = opendir(tmp_file.buf);\n+\n+\tif (!info_dir && mkdir(tmp_file.buf, 0777) < 0)\n+\t\tdie_errno(_(\"cannot mkdir %s\"), tmp_file.buf);\n+\tif (info_dir)\n+\t\tclosedir(info_dir);\n+\n+\tstrbuf_addstr(&tmp_file, \"/tmp_graph_XXXXXX\");\n+\n+\tfd = git_mkstemp_mode(tmp_file.buf, 0444);\n+\tif (fd < 0)\n+\t\tdie_errno(\"unable to create '%s'\", tmp_file.buf);\n+\n+\tf = sha1fd(fd, tmp_file.buf);\n+\n+\tsha1write_be32(f, GRAPH_SIGNATURE);\n+\n+\tsha1write_u8(f, GRAPH_VERSION);\n+\tsha1write_u8(f, GRAPH_OID_VERSION);\n+\tsha1write_u8(f, num_chunks);\n+\tsha1write_u8(f, 0); /* unused padding byte */\n+\n+\tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n+\tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n+\tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n+\tif (num_long_edges)\n+\t\tchunk_ids[3] = GRAPH_CHUNKID_LARGEEDGES;\n+\telse\n+\t\tchunk_ids[3] = 0;\n+\tchunk_ids[4] = 0;\n+\n+\tchunk_offsets[0] = 8 + GRAPH_CHUNKLOOKUP_SIZE;\n+\tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n+\tchunk_offsets[2] = chunk_offsets[1] + GRAPH_OID_LEN * commits.nr;\n+\tchunk_offsets[3] = chunk_offsets[2] + (GRAPH_OID_LEN + 16) * commits.nr;\n+\tchunk_offsets[4] = chunk_offsets[3] + 4 * num_long_edges;\n+\n+\tfor (i = 0; i <= num_chunks; i++) {\n+\t\tuint32_t chunk_write[3];\n+\n+\t\tchunk_write[0] = htonl(chunk_ids[i]);\n+\t\tchunk_write[1] = htonl(chunk_offsets[i] >> 32);\n+\t\tchunk_write[2] = htonl(chunk_offsets[i] & 0xffffffff);\n+\t\tsha1write(f, chunk_write, 12);\n+\t}\n+\n+\twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n+\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n+\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n+\twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n+\n+\tsha1close(f, final_hash, CSUM_CLOSE | CSUM_FSYNC);\n+\n+\tstrbuf_addf(&graph_file, \"graph-%s.graph\", sha1_to_hex(final_hash));\n+\tgraph_name = strbuf_detach(&graph_file, NULL);\n+\tstrbuf_addf(&graph_file, \"%s/info/%s\", obj_dir, graph_name);\n+\n+\tif (rename(tmp_file.buf, graph_file.buf))\n+\t\tdie(\"failed to rename %s to %s\", tmp_file.buf, graph_file.buf);\n+\n+\tstrbuf_release(&tmp_file);\n+\tstrbuf_release(&graph_file);\n+\tfree(oids.list);\n+\toids.alloc = 0;\n+\toids.nr = 0;\n+\n+\treturn graph_name;\n+}\n+\ndiff --git a/commit-graph.h b/commit-graph.h\nnew file mode 100644\nindex 0000000..dc8c73a\n--- /dev/null\n+++ b/commit-graph.h\n@@ -0,0 +1,7 @@\n+#ifndef COMMIT_GRAPH_H\n+#define COMMIT_GRAPH_H\n+\n+extern char *write_commit_graph(const char *obj_dir);\n+\n+#endif\n+\n-- \n2.7.4\n\n"},{"id":"339634","messageId":"1519066406-81663-6-git-send-email-dstolee@microsoft.com","threadId":"47724","inReplyTo":"1519066406-81663-1-git-send-email-dstolee@microsoft.com","subject":"[PATCH v4 05/13] commit-graph: implement 'git-commit-graph write'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-19T18:53:18Z","receivedAt":"2018-02-19T18:54:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"Teach git-commit-graph to write graph files. Create new test script to verify\nthis command succeeds without failure.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 40 +++++++++++++\n builtin/commit-graph.c | 43 +++++++++++++-\n t/t5318-commit-graph.sh | 119 +++++++++++++++++++++++++++++++++++++\n 3 files changed, 201 insertions(+), 1 deletion(-)\n create mode 100755 t/t5318-commit-graph.sh\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex e1c3078..c3f222f 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -5,6 +5,46 @@ NAME\n ----\n git-commit-graph - Write and verify Git commit graphs (.graph files)\n \n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git commit-graph write' [--object-dir ]\n+\n+\n+DESCRIPTION\n+-----------\n+\n+Manage the serialized commit graph file.\n+\n+\n+OPTIONS\n+-------\n+--object-dir::\n+\tUse given directory for the location of packfiles and graph files.\n+\tThe graph files will be in /info and the packfiles are expected\n+\tto be in /pack.\n+\n+\n+COMMANDS\n+--------\n+'write'::\n+\n+Write a commit graph file based on the commits found in packfiles.\n+Includes all commits from the existing commit graph file. Outputs the\n+resulting filename.\n+\n+\n+EXAMPLES\n+--------\n+\n+* Write a commit graph file for the packed commits in your local .git folder.\n++\n+------------------------------------------------\n+$ git commit-graph write\n+------------------------------------------------\n+\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 98110bb..a51d87b 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -1,9 +1,18 @@\n #include \"builtin.h\"\n #include \"config.h\"\n+#include \"dir.h\"\n+#include \"lockfile.h\"\n #include \"parse-options.h\"\n+#include \"commit-graph.h\"\n \n static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph [--object-dir ]\"),\n+\tN_(\"git commit-graph write [--object-dir ]\"),\n+\tNULL\n+};\n+\n+static const char * const builtin_commit_graph_write_usage[] = {\n+\tN_(\"git commit-graph write [--object-dir ]\"),\n \tNULL\n };\n \n@@ -11,11 +20,38 @@ static struct opts_commit_graph {\n \tconst char *obj_dir;\n } opts;\n \n+static int graph_write(int argc, const char **argv)\n+{\n+\tchar *graph_name;\n+\n+\tstatic struct option builtin_commit_graph_write_options[] = {\n+\t\t{ OPTION_STRING, 'o', \"object-dir\", &opts.obj_dir,\n+\t\t\tN_(\"dir\"),\n+\t\t\tN_(\"The object directory to store the graph\") },\n+\t\tOPT_END(),\n+\t};\n+\n+\targc = parse_options(argc, argv, NULL,\n+\t\t\t builtin_commit_graph_write_options,\n+\t\t\t builtin_commit_graph_write_usage, 0);\n+\n+\tif (!opts.obj_dir)\n+\t\topts.obj_dir = get_object_directory();\n+\n+\tgraph_name = write_commit_graph(opts.obj_dir);\n+\n+\tif (graph_name) {\n+\t\tprintf(\"%s\\n\", graph_name);\n+\t\tFREE_AND_NULL(graph_name);\n+\t}\n+\n+\treturn 0;\n+}\n \n int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_commit_graph_options[] = {\n-\t\t{ OPTION_STRING, 'p', \"object-dir\", &opts.obj_dir,\n+\t\t{ OPTION_STRING, 'o', \"object-dir\", &opts.obj_dir,\n \t\t\tN_(\"dir\"),\n \t\t\tN_(\"The object directory to store the graph\") },\n \t\tOPT_END(),\n@@ -31,6 +67,11 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\t\t builtin_commit_graph_usage,\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n+\tif (argc > 0) {\n+\t\tif (!strcmp(argv[0], \"write\"))\n+\t\t\treturn graph_write(argc, argv);\n+\t}\n+\n \tusage_with_options(builtin_commit_graph_usage,\n \t\t\t builtin_commit_graph_options);\n }\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nnew file mode 100755\nindex 0000000..6a5e93c\n--- /dev/null\n+++ b/t/t5318-commit-graph.sh\n@@ -0,0 +1,119 @@\n+#!/bin/sh\n+\n+test_description='commit graph'\n+. ./test-lib.sh\n+\n+test_expect_success 'setup full repo' '\n+\trm -rf .git &&\n+\tmkdir full &&\n+\tcd full &&\n+\tgit init &&\n+\tobjdir=\".git/objects\"\n+'\n+\n+test_expect_success 'write graph with no packs' '\n+\tgit commit-graph write --object-dir .\n+'\n+\n+test_expect_success 'create commits and repack' '\n+\tfor i in $(test_seq 3)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i\n+\tdone &&\n+\tgit repack\n+'\n+\n+test_expect_success 'write graph' '\n+\tgraph1=$(git commit-graph write) &&\n+\ttest_path_is_file $objdir/info/$graph1\n+'\n+\n+test_expect_success 'Add more commits' '\n+\tgit reset --hard commits/1 &&\n+\tfor i in $(test_seq 4 5)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i\n+\tdone &&\n+\tgit reset --hard commits/2 &&\n+\tfor i in $(test_seq 6 7)\n+\tdo\n+\t\ttest_commit $i &&\n+\t\tgit branch commits/$i\n+\tdone &&\n+\tgit reset --hard commits/2 &&\n+\tgit merge commits/4 &&\n+\tgit branch merge/1 &&\n+\tgit reset --hard commits/4 &&\n+\tgit merge commits/6 &&\n+\tgit branch merge/2 &&\n+\tgit reset --hard commits/3 &&\n+\tgit merge commits/5 commits/7 &&\n+\tgit branch merge/3 &&\n+\tgit repack\n+'\n+\n+# Current graph structure:\n+#\n+# __M3___\n+# / | \\\n+# 3 M1 5 M2 7\n+# |/ \\|/ \\|\n+# 2 4 6\n+# |___/____/\n+# 1\n+\n+\n+test_expect_success 'write graph with merges' '\n+\tgraph2=$(git commit-graph write)&&\n+\ttest_path_is_file $objdir/info/$graph2\n+'\n+\n+test_expect_success 'Add one more commit' '\n+\ttest_commit 8 &&\n+\tgit branch commits/8 &&\n+\tls $objdir/pack | grep idx >existing-idx &&\n+\tgit repack &&\n+\tls $objdir/pack| grep idx | grep -v --file=existing-idx >new-idx\n+'\n+\n+# Current graph structure:\n+#\n+# 8\n+# |\n+# __M3___\n+# / | \\\n+# 3 M1 5 M2 7\n+# |/ \\|/ \\|\n+# 2 4 6\n+# |___/____/\n+# 1\n+\n+test_expect_success 'write graph with new commit' '\n+\tgraph3=$(git commit-graph write) &&\n+\ttest_path_is_file $objdir/info/$graph3\n+'\n+\n+test_expect_success 'write graph with nothing new' '\n+\tgraph4=$(git commit-graph write) &&\n+\ttest_path_is_file $objdir/info/$graph4 &&\n+\tprintf $graph3 >expect &&\n+\tprintf $graph4 >output &&\n+\ttest_cmp expect output\n+'\n+\n+test_expect_success 'setup bare repo' '\n+\tcd .. &&\n+\tgit clone --bare --no-local full bare &&\n+\tcd bare &&\n+\tbaredir=\"./objects\"\n+'\n+\n+test_expect_success 'write graph in bare repo' '\n+\tgraphbare=$(git commit-graph write) &&\n+\ttest_path_is_file $baredir/info/$graphbare\n+'\n+\n+test_done\n+\n-- \n2.7.4\n\n"},{"id":"339741","messageId":"xmqqa7w32xq7.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"1519066406-81663-2-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v4 01/13] commit-graph: add format document","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-20T20:49:04Z","receivedAt":"2018-02-20T20:49:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> Documentation/technical/commit-graph-format.txt | 90 +++++++++++++++++++++++++\n> 1 file changed, 90 insertions(+)\n> create mode 100644 Documentation/technical/commit-graph-format.txt\n\nHopefully just a few remaining nits. Overall I find this written\nreally clearly.\n\n> +== graph-*.graph files have the following format:\n> +\n> +In order to allow extensions that add extra data to the graph, we organize\n> +the body into \"chunks\" and provide a binary lookup table at the beginning\n> +of the body. The header includes certain values, such as number of chunks,\n> +hash lengths and types.\n\nWe no longer have lengths stored.\n\n> + ...\n> + The remaining data in the body is described one chunk at a time, and\n> + these chunks may be given in any order. Chunks are required unless\n> + otherwise specified.\n\nIt is good that this explicitly says chunks can come in any order,\nand which ones are required. It should also say which chunk can\nappear multiple times. I think all four chunk types we currently\ndefine can have at most one instance in a file.\n\n> +CHUNK DATA:\n> +\n> + OID Fanout (ID: {'O', 'I', 'D', 'F'}) (256 * 4 bytes)\n> + The ith entry, F[i], stores the number of OIDs with first\n> + byte at most i. Thus F[255] stores the total\n> + number of commits (N).\n> +\n> + OID Lookup (ID: {'O', 'I', 'D', 'L'}) (N * H bytes)\n> + The OIDs for all commits in the graph, sorted in ascending order.\n\nSomewhere in this document, we probably would want to say that this\nformat allows at most (1<<31)-1 commits recorded in the file (as\nCGET and EDGE uses 31-bit uint to index into this table, using MSB\nfor other purposes, and the all-1-bit pattern is also special), and\nwhen we refer to \"int-ids\" of a commit, it is this 31-bit number\nthat is an index into this table.\n\n> + Commit Data (ID: {'C', 'G', 'E', 'T' }) (N * (H + 16) bytes)\n> + * The first H bytes are for the OID of the root tree.\n> + * The next 8 bytes are for the int-ids of the first two parents\n> + of the ith commit. Stores value 0xffffffff if no parent in that\n> + position. If there are more than two parents, the second value\n> + has its most-significant bit on and the other bits store an array\n> + position into the Large Edge List chunk.\n> + * The next 8 bytes store the generation number of the commit and\n> + the commit time in seconds since EPOCH. The generation number\n> + uses the higher 30 bits of the first 4 bytes, while the commit\n> + time uses the 32 bits of the second 4 bytes, along with the lowest\n> + 2 bits of the lowest byte, storing the 33rd and 34th bit of the\n> + commit time.\n> +\n> + Large Edge List (ID: {'E', 'D', 'G', 'E'}) [Optional]\n> + This list of 4-byte values store the second through nth parents for\n> + all octopus merges. The second parent value in the commit data stores\n> + an array position within this list along with the most-significant bit\n> + on. Starting at that array position, iterate through this list of int-ids\n> + for the parents until reaching a value with the most-significant bit on.\n> + The other bits correspond to the int-id of the last parent.\n> +\n> +TRAILER:\n> +\n> +\tH-byte HASH-checksum of all of the above.\n> +\n"},{"id":"339747","messageId":"xmqq1shf2v8d.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"1519066406-81663-3-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v4 02/13] graph: add commit graph design document","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-20T21:42:58Z","receivedAt":"2018-02-20T21:43:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> +2. Walking the entire graph to avoid topological order mistakes.\n\nYou have at least one more mention of \"topological order mistakes\"\nbelow, but we commonly refer to this issue and blame it for \"clock\nskew\". Using the word highlights that there is no \"mistake\" in topo\norder algorithm and mistakes are in the commit timestamps.\n\n> +In practice, we expect some commits to be created recently and not stored\n> +in the commit graph. We can treat these commits as having \"infinite\"\n> +generation number and walk until reaching commits with known generation\n> +number.\n\nHmm, \"pretend infinity\" is an interesting approach---I need to think\nabout it a bit more if it is sufficient.\n\n> +- .graph files are managed only by the 'commit-graph' builtin. These are not\n> + updated automatically during clone, fetch, repack, or creating new commits.\n\nOK. s/builtin/subcommand/; it does not make much difference if it\nis a built-in or standalone command.\n\n> +- There is no 'verify' subcommand for the 'commit-graph' builtin to verify\n> + the contents of the graph file agree with the contents in the ODB.\n\nI am not entirely sure about the merit of going into this level of\ndetail. Being able to use only a single file looks like a more\nfundamental design limitation, which deserves to be decribed in this\nsection, and we could ship the subsystem with that limitation.\n\nBut the lack of verify that can be called from fsck is merely the\nmatter of not the subsystem being mature enough (to be written,\nreviewed and tested) and not a fundamental one, and we will not be\nshipping the subsystem until that limitation is lifted.\n\nSo I'd guess that we prefer this bullet item to be in the commit log\nmessage, not here, that describes the current status of the\ndevelopment (as opposed to the state of the subsystem).\n\n> +- Generation numbers are not computed in the current version. The file\n> + format supports storing them, along with a mechanism to upgrade from\n> + a file without generation numbers to one that uses them.\n\nExactly the same comment as above applies to this item.\n\n> +- The commit graph is currently incompatible with commit grafts. This can be\n> + remedied by duplicating or refactoring the current graft logic.\n\nHmm. Can it be lifted without first allowing us to use more than\none commit graph file (i.e. one for \"traverse while honoring the\ngrafts\", the other for \"traverse while ignoring the grafts\")?\n\n> +- After computing and storing generation numbers, we must make graph\n> + walks aware of generation numbers to gain the performance benefits they\n> + enable. This will mostly be accomplished by swapping a commit-date-ordered\n> + priority queue with one ordered by generation number. The following\n> + operations are important candidates:\n> +\n> + - paint_down_to_common()\n> + - 'log --topo-order'\n\nYes.\n\n> +- The graph currently only adds commits to a previously existing graph.\n> + When writing a new graph, we could check that the ODB still contains\n> + the commits and choose to remove the commits that are deleted from the\n> + ODB. For performance reasons, this check should remain optional.\n\nThe last sentence is somehow unconvincing. It probably is not\nappropriate for the \"Future Work\" section to be making a hurried\ndesign decision before having any working verification code to run\nbenchmark on.\n\n> +- Currently, parse_commit_gently() requires filling in the root tree\n> + object for a commit. This passes through lookup_tree() and consequently\n> + lookup_object(). Also, it calls lookup_commit() when loading the parents.\n> + These method calls check the ODB for object existence, even if the\n> + consumer does not need the content. For example, we do not need the\n> + tree contents when computing merge bases. Now that commit parsing is\n> + removed from the computation time, these lookup operations are the\n> + slowest operations keeping graph walks from being fast. Consider\n> + loading these objects without verifying their existence in the ODB and\n> + only loading them fully when consumers need them. Consider a method\n> + such as \"ensure_tree_loaded(commit)\" that fully loads a tree before\n> + using commit->tree.\n\nVery good idea.\n\n> +- The current design uses the 'commit-graph' builtin to generate the graph.\n> + When this feature stabilizes enough to recommend to most users, we should\n> + add automatic graph writes to common operations that create many commits.\n> + For example, one coulde compute a graph on 'clone', 'fetch', or 'repack'\n> + commands.\n\ns/coulde/could/.\n\nAlso do not forget \"fsck\" that calls \"verify\". That is more urgent\nthan intergration with any other subcommand.\n\n> +- A server could provide a commit graph file as part of the network protocol\n> + to avoid extra calculations by clients.\n\nWe need to assess the riskiness and threat models regarding this, if\nwe really want to follow this \"could\" through. I would imagine that\nthe cost for verification is comparable to the cost for regenerating,\nin which case it may not be worth doing this _unless_ the user opts\ninto it saying that the other side over the wire is trusted without\nany reservation.\n\n"},{"id":"339748","messageId":"xmqqvaer1ga1.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"1519066406-81663-4-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v4 03/13] commit-graph: create git-commit-graph builtin","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-20T21:51:18Z","receivedAt":"2018-02-20T21:51:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> +int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n> +{\n> +\tstatic struct option builtin_commit_graph_options[] = {\n> +\t\t{ OPTION_STRING, 'p', \"object-dir\", &opts.obj_dir,\n> +\t\t\tN_(\"dir\"),\n> +\t\t\tN_(\"The object directory to store the graph\") },\n\nI have a suspicion that this was modeled after some other built-in\nthat has a similar issue (perhaps written long time ago), but isn't\nOPT_STRING() sufficient to define this element these days?\n\nOr am I missing something?\n\nWhy squat on short-and-sweet \"-p\"? For that matter, since this is\nnot expected to be end-user facing command anyway, I suspect that we\ndo not want to allocate a single letter option from day one, which\npaints ourselves into a corner from where we cannot escape.\n"},{"id":"339761","messageId":"xmqqmv031d7d.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"1519066406-81663-5-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v4 04/13] commit-graph: implement write_commit_graph()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-20T22:57:42Z","receivedAt":"2018-02-20T22:57:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> +#define GRAPH_OID_VERSION_SHA1 1\n> +#define GRAPH_OID_LEN_SHA1 20\n\nThis hardcoded 20 on the right hand side of this #define is probably\nproblematic. Unless you are planning to possibly store truncated\nhash value for some future hash algorithm, GRAPH_OID_LEN_$HASH should\nalways be the same as GIT_$HASH_RAWSZ, I would think. IOW\n\n #define GRAPH_OID_LEN_SHA1 GIT_SHA1_RAWSZ\n\nperhaps?\n\n> +static void write_graph_chunk_fanout(struct sha1file *f,\n> +\t\t\t\t struct commit **commits,\n> +\t\t\t\t int nr_commits)\n> +{\n> +\tuint32_t i, count = 0;\n> +\tstruct commit **list = commits;\n> +\tstruct commit **last = commits + nr_commits;\n> +\n> +\t/*\n> +\t * Write the first-level table (the list is sorted,\n> +\t * but we use a 256-entry lookup to be able to avoid\n> +\t * having to do eight extra binary search iterations).\n> +\t */\n> +\tfor (i = 0; i < 256; i++) {\n> +\t\twhile (list < last) {\n> +\t\t\tif ((*list)->object.oid.hash[0] != i)\n> +\t\t\t\tbreak;\n> +\t\t\tcount++;\n> +\t\t\tlist++;\n> +\t\t}\n\nIf count and list are always incremented in unison, perhaps you do\nnot need an extra variable \"last\". If typeof(nr_commits) is wider\nthan typeof(count), this loop and the next write-be32 is screwed\nanyway ;-)\n\nThis comment probably applies equally to some other uses of the same\n\"compute last pointer to compare with running pointer for\ntermination\" pattern in this patch.\n\n> +\t\tsha1write_be32(f, count);\n> +\t}\n> +}\n\n> +static int commit_pos(struct commit **commits, int nr_commits,\n> +\t\t const struct object_id *oid, uint32_t *pos)\n> +{\n\nIt is a bit unusual to see something_pos() that returns an integer\nthat does *NOT* return the position as its return value. Dropping\nthe *pos parameter, and returning \"mid\" when commits[mid] is what we\nwanted to see, and otherwise returning \"-1 - first\" to signal the\nposition at which we _would_ have found the object, if it were in\nthe table, would make it more consistent with the usual convention.\n\nDon't we even have such a generalized binary search helper already\nsomewhere in the system?\n\n> +static void write_graph_chunk_data(struct sha1file *f, int hash_len,\n> +\t\t\t\t struct commit **commits, int nr_commits)\n> +{\n> +\tstruct commit **list = commits;\n> +\tstruct commit **last = commits + nr_commits;\n> +\tuint32_t num_large_edges = 0;\n> +\n> +\twhile (list < last) {\n> +\t\tstruct commit_list *parent;\n> +\t\tuint32_t int_id;\n> +\t\tuint32_t packedDate[2];\n> +\n> +...\n> +\t\tif (!parent)\n> +\t\t\tint_id = GRAPH_PARENT_NONE;\n> +\t\telse if (parent->next)\n> +\t\t\tint_id = GRAPH_LARGE_EDGES_NEEDED | num_large_edges;\n> +\t\telse if (!commit_pos(commits, nr_commits,\n> +\t\t\t\t &(parent->item->object.oid), &int_id))\n> +\t\t\tint_id = GRAPH_PARENT_MISSING;\n> +\n> +\t\tsha1write_be32(f, int_id);\n> +\n> +\t\tif (parent && parent->next) {\n\nThis is equivalent to checking \"int_id & GRAPH_LARGE_EDGES_NEEDED\",\nright? Not suggesting to use the other form of checks, but trying\nto see what's the best way to express it in the most readable way.\n\n> +\t\t\tdo {\n> +\t\t\t\tnum_large_edges++;\n> +\t\t\t\tparent = parent->next;\n> +\t\t\t} while (parent);\n\nIt feels somewhat wasteful to traverse the commit's parents list\nonly to count, without populating the octopus table (which I\nunderstand is assumed to be minority case under this design).\n\n> +\t\t}\n> +\n> +\t\tif (sizeof((*list)->date) > 4)\n> +\t\t\tpackedDate[0] = htonl(((*list)->date >> 32) & 0x3);\n> +\t\telse\n> +\t\t\tpackedDate[0] = 0;\n\nOK, the undefined pattern in the previous round is now gone ;-) Good.\n\n> +\t\tpackedDate[1] = htonl((*list)->date);\n> +\t\tsha1write(f, packedDate, 8);\n> +\n> +\t\tlist++;\n> +\t}\n> +}\n> +\n> +static void write_graph_chunk_large_edges(struct sha1file *f,\n> +\t\t\t\t\t struct commit **commits,\n> +\t\t\t\t\t int nr_commits)\n> +{\n> +\tstruct commit **list = commits;\n> +\tstruct commit **last = commits + nr_commits;\n> +\tstruct commit_list *parent;\n> +\n> +\twhile (list < last) {\n> +\t\tint num_parents = 0;\n> +\t\tfor (parent = (*list)->parents; num_parents < 3 && parent;\n> +\t\t parent = parent->next)\n> +\t\t\tnum_parents++;\n> +\n> +\t\tif (num_parents <= 2) {\n> +\t\t\tlist++;\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\t/* Since num_parents > 2, this initializer is safe. */\n> +\t\tfor (parent = (*list)->parents->next; parent; parent = parent->next) {\n> +\t\t\tuint32_t int_id, swap_int_id;\n> +\t\t\tuint32_t last_edge = 0;\n> +\t\t\tif (!parent->next)\n> +\t\t\t\tlast_edge |= GRAPH_LAST_EDGE;\n> +\n> +\t\t\tif (commit_pos(commits, nr_commits,\n> +\t\t\t\t &(parent->item->object.oid),\n> +\t\t\t\t &int_id))\n> +\t\t\t\tswap_int_id = htonl(int_id | last_edge);\n> +\t\t\telse\n> +\t\t\t\tswap_int_id = htonl(GRAPH_PARENT_MISSING | last_edge);\n> +\t\t\tsha1write(f, &swap_int_id, 4);\n\nWhat does \"swap_\" in the name of this variable mean? For some\narchs, there is no swap. The only difference between int_id and the\nvariable is that its MSB may possibly be smudged with last_edge bit.\n\nThis is a tangent, but after having seen many instances of \"int_id\",\nI started to feel that it is grossly misnamed. We do not care about\nits \"int\" ness---what's more significant about it is that we use can\nit as a short identifier in place for a full object name, given the\ntable of known OIDs. \"oid_table_index\" may be a better name (but\nothers may be able to suggest even better one).\n\n\tint pos;\n\tpos = commit_pos(commits, nr_commits, parent->item->object.oid);\n\toid_table_pos = (pos < 0) ? GRAPH_PARENT_MISSING : pos;\n\tif (!parent->net)\n\t\toid_table_pos |= GRAPH_LAST_EDGE;\n\toid_table_pos = htonl(oid_table_pos);\n\tsha1write(f, &oid_table_pos, sizeof(oid_table_pos));\n\nor something like that, perhaps?\n\n> +static int commit_compare(const void *_a, const void *_b)\n> +{\n> +\tstruct object_id *a = (struct object_id *)_a;\n> +\tstruct object_id *b = (struct object_id *)_b;\n> +\treturn oidcmp(a, b);\n> +}\n\nI think oidcmp() takes const pointers, so there is no need to\ndiscard constness from the parameter like this code does. Also I\nthink we tend to prefer writing a_/b_ (instead of _a/_b) to appease\nlanguage lawyers who do not want us mere mortals to use names that\nbegin with underscore.\n\n> +static int if_packed_commit_add_to_list(const struct object_id *oid,\n> +\t\t\t\t\tstruct packed_git *pack,\n> +\t\t\t\t\tuint32_t pos,\n> +\t\t\t\t\tvoid *data)\n\nThat is a strange name. \"collect packed commits\", perhaps?\n\n> +char *write_commit_graph(const char *obj_dir)\n> +{\n> +\tstruct packed_oid_list oids;\n> +\tstruct packed_commit_list commits;\n> +\tstruct sha1file *f;\n> +\tint i, count_distinct = 0;\n> +\tDIR *info_dir;\n> +\tstruct strbuf tmp_file = STRBUF_INIT;\n> +\tstruct strbuf graph_file = STRBUF_INIT;\n> +\tunsigned char final_hash[GIT_MAX_RAWSZ];\n> +\tchar *graph_name;\n> +\tint fd;\n> +\tuint32_t chunk_ids[5];\n> +\tuint64_t chunk_offsets[5];\n> +\tint num_chunks;\n> +\tint num_long_edges;\n> +\tstruct commit_list *parent;\n> +\n> +\toids.nr = 0;\n> +\toids.alloc = (int)(0.15 * approximate_object_count());\n\nHeh, traditionalist would probably avoid unnecessary use of float\nand use something like 1/4 or 1/8 ;-) After all, it is merely a\nballpark guestimate.\n\n> +\tnum_long_edges = 0;\n\nThis again is about naming, but I find it a bit unnatural to call\nthe edge between a chind and its octopus parents \"long\". Individual\nedges are not long--the only thing that is long is your \"list of\nedges\". Some other codepaths in this patch seems to call the same\nconcept with s/long/large/, which I found somewhat puzzling.\n\n> +\tfor (i = 0; i < oids.nr; i++) {\n> +\t\tint num_parents = 0;\n> +\t\tif (i > 0 && !oidcmp(&oids.list[i-1], &oids.list[i]))\n> +\t\t\tcontinue;\n> +\n> +\t\tcommits.list[commits.nr] = lookup_commit(&oids.list[i]);\n> +\t\tparse_commit(commits.list[commits.nr]);\n> +\n> +\t\tfor (parent = commits.list[commits.nr]->parents;\n> +\t\t parent; parent = parent->next)\n> +\t\t\tnum_parents++;\n> +\n> +\t\tif (num_parents > 2)\n> +\t\t\tnum_long_edges += num_parents - 1;\n\nOK, so we count how many entries we will record in the overflow\nparent table, and...\n\n> +\n> +\t\tcommits.nr++;\n> +\t}\n> +\tnum_chunks = num_long_edges ? 4 : 3;\n\n... if we do not have any octopus commit, we do not need the chunk\nfor the overflow parent table. Makes sense.\n\n> +\tstrbuf_addf(&tmp_file, \"%s/info\", obj_dir);\n> +\tinfo_dir = opendir(tmp_file.buf);\n> +\n> +\tif (!info_dir && mkdir(tmp_file.buf, 0777) < 0)\n> +\t\tdie_errno(_(\"cannot mkdir %s\"), tmp_file.buf);\n> +\tif (info_dir)\n> +\t\tclosedir(info_dir);\n> +\tstrbuf_addstr(&tmp_file, \"/tmp_graph_XXXXXX\");\n> +\n> +\tfd = git_mkstemp_mode(tmp_file.buf, 0444);\n> +\tif (fd < 0)\n> +\t\tdie_errno(\"unable to create '%s'\", tmp_file.buf);\n\nIt is not performance critical, but it feels a bit wasteful to\nopendir merely to see if something exists as a directory, and it is\nmisleading to the readers (it looks as if we care about what files\nwe already have in the directory).\n\nThe approach that optimizes for the most common case would be to\n\n\t- prepare full path to the tempfile first\n\t- try create with mkstemp\n\t - if successful, you do not have to worry about creating\n\t the directory at all, which is the most common case\n - see why mkstemp step above failed. Was it because you\n\t did not have the surrounding directory?\n - if not, there is no point continuing. Just error out.\n\t - if it was due to missing directory, try creating one.\n\t- try create with mkstemp\n\t - if successful, all is well.\n - otherwise there isn't anything more we can do here.\n\n\n\n> +\n> +\tf = sha1fd(fd, tmp_file.buf);\n> +\n> +\tsha1write_be32(f, GRAPH_SIGNATURE);\n> +\n> +\tsha1write_u8(f, GRAPH_VERSION);\n> +\tsha1write_u8(f, GRAPH_OID_VERSION);\n> +\tsha1write_u8(f, num_chunks);\n> +\tsha1write_u8(f, 0); /* unused padding byte */\n> +\n> +\tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n> +\tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n> +\tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n> +\tif (num_long_edges)\n> +\t\tchunk_ids[3] = GRAPH_CHUNKID_LARGEEDGES;\n> +\telse\n> +\t\tchunk_ids[3] = 0;\n> +\tchunk_ids[4] = 0;\n> +\n> +\tchunk_offsets[0] = 8 + GRAPH_CHUNKLOOKUP_SIZE;\n> +\tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n> +\tchunk_offsets[2] = chunk_offsets[1] + GRAPH_OID_LEN * commits.nr;\n> +\tchunk_offsets[3] = chunk_offsets[2] + (GRAPH_OID_LEN + 16) * commits.nr;\n> +\tchunk_offsets[4] = chunk_offsets[3] + 4 * num_long_edges;\n\nDo we have to care about overflowing any of the above? For example,\nthe format allows only up to (1<<31)-1 commits, but did something\nactually check if commits.nr at this point stayed under that limit?\n\n"},{"id":"339825","messageId":"xmqqd10yyxsu.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"xmqqvaer1ga1.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 03/13] commit-graph: create git-commit-graph builtin","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-21T18:58:41Z","receivedAt":"2018-02-21T18:58:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Derrick Stolee writes:\n>\n>> +int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n>> +{\n>> +\tstatic struct option builtin_commit_graph_options[] = {\n>> +\t\t{ OPTION_STRING, 'p', \"object-dir\", &opts.obj_dir,\n>> +\t\t\tN_(\"dir\"),\n>> +\t\t\tN_(\"The object directory to store the graph\") },\n>\n> I have a suspicion that this was modeled after some other built-in\n> that has a similar issue (perhaps written long time ago), but isn't\n> OPT_STRING() sufficient to define this element these days?\n>\n> Or am I missing something?\n>\n> Why squat on short-and-sweet \"-p\"? For that matter, since this is\n> not expected to be end-user facing command anyway, I suspect that we\n> do not want to allocate a single letter option from day one, which\n> paints ourselves into a corner from where we cannot escape.\n\nI suspect that exactly the same comment applies to patches in this\nseries that add other subcommands (I just saw one in the patch for\nadding 'write').\n\n"},{"id":"339828","messageId":"CAGZ79kawTtSzVV2=sNn_OFsmCbcLGhVi1pt4bVrOYfZA28E+6Q@mail.gmail.com","threadId":"47724","inReplyTo":"1519066406-81663-2-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v4 01/13] commit-graph: add format document","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-21T19:23:41Z","receivedAt":"2018-02-21T19:24:01Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Feb 19, 2018 at 10:53 AM, Derrick Stolee wrote:\n> Add document specifying the binary format for commit graphs. This\n> format allows for:\n>\n> * New versions.\n> * New hash functions and hash lengths.\n> * Optional extensions.\n>\n> Basic header information is followed by a binary table of contents\n> into \"chunks\" that include:\n>\n> * An ordered list of commit object IDs.\n> * A 256-entry fanout into that list of OIDs.\n> * A list of metadata for the commits.\n> * A list of \"large edges\" to enable octopus merges.\n>\n> The format automatically includes two parent positions for every\n> commit. This favors speed over space, since using only one position\n> per commit would cause an extra level of indirection for every merge\n> commit. (Octopus merges suffer from this indirection, but they are\n> very rare.)\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/technical/commit-graph-format.txt | 90 +++++++++++++++++++++++++\n> 1 file changed, 90 insertions(+)\n> create mode 100644 Documentation/technical/commit-graph-format.txt\n>\n> diff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\n> new file mode 100644\n> index 0000000..11b18b5\n> --- /dev/null\n> +++ b/Documentation/technical/commit-graph-format.txt\n> @@ -0,0 +1,90 @@\n> +Git commit graph format\n> +=======================\n> +\n> +The Git commit graph stores a list of commit OIDs and some associated\n> +metadata, including:\n> +\n> +- The generation number of the commit. Commits with no parents have\n> + generation number 1; commits with parents have generation number\n> + one more than the maximum generation number of its parents. We\n> + reserve zero as special, and can be used to mark a generation\n> + number invalid or as \"not computed\".\n> +\n> +- The root tree OID.\n> +\n> +- The commit date.\n> +\n> +- The parents of the commit, stored using positional references within\n> + the graph file.\n> +\n> +== graph-*.graph files have the following format:\n> +\n> +In order to allow extensions that add extra data to the graph, we organize\n> +the body into \"chunks\" and provide a binary lookup table at the beginning\n> +of the body. The header includes certain values, such as number of chunks,\n> +hash lengths and types.\n> +\n> +All 4-byte numbers are in network order.\n> +\n> +HEADER:\n> +\n> + 4-byte signature:\n> + The signature is: {'C', 'G', 'P', 'H'}\n> +\n> + 1-byte version number:\n> + Currently, the only valid version is 1.\n> +\n> + 1-byte Object Id Version (1 = SHA-1)\n> +\n> + 1-byte number (C) of \"chunks\"\n> +\n> + 1-byte (reserved for later use)\n\nWhat should clients of today do with it?\n* ignore it completely [as they have no idea what it is] or\n* throw hands up in the air if it is anything other than 0 ?\n [because clearly we will increment the version\n or have new information in a new chunk instead of just sneaking\n in information here?]\n\n> +CHUNK LOOKUP:\n> +\n> + (C + 1) * 12 bytes listing the table of contents for the chunks:\n> + First 4 bytes describe chunk id. Value 0 is a terminating label.\n> + Other 8 bytes provide offset in current file for chunk to start.\n\noffset [in bytes? I could imagine having a larger granularity here,\nbecause chunks don't sound small.]\n\n> + (Chunks are ordered contiguously in the file, so you can infer\n> + the length using the next chunk position if necessary.)\n> +\n> + The remaining data in the body is described one chunk at a time, and\n> + these chunks may be given in any order. Chunks are required unless\n> + otherwise specified.\n> +\n> +CHUNK DATA:\n> +\n> + OID Fanout (ID: {'O', 'I', 'D', 'F'}) (256 * 4 bytes)\n> + The ith entry, F[i], stores the number of OIDs with first\n> + byte at most i. Thus F[255] stores the total\n> + number of commits (N).\n\n[ so in small repos, where there are fewer than 256 objects,\nF[i] == F[i+1], for all i'th where there is no object starting with i byte]\n\n> + OID Lookup (ID: {'O', 'I', 'D', 'L'}) (N * H bytes)\n> + The OIDs for all commits in the graph, sorted in ascending order.\n> +\n> + Commit Data (ID: {'C', 'G', 'E', 'T' }) (N * (H + 16) bytes)\n> + * The first H bytes are for the OID of the root tree.\n> + * The next 8 bytes are for the int-ids of the first two parents\n> + of the ith commit. Stores value 0xffffffff if no parent in that\n> + position. If there are more than two parents, the second value\n> + has its most-significant bit on and the other bits store an array\n> + position into the Large Edge List chunk.\n> + * The next 8 bytes store the generation number of the commit and\n> + the commit time in seconds since EPOCH. The generation number\n> + uses the higher 30 bits of the first 4 bytes, while the commit\n> + time uses the 32 bits of the second 4 bytes, along with the lowest\n> + 2 bits of the lowest byte, storing the 33rd and 34th bit of the\n> + commit time.\n> +\n> + Large Edge List (ID: {'E', 'D', 'G', 'E'}) [Optional]\n> + This list of 4-byte values store the second through nth parents for\n> + all octopus merges. The second parent value in the commit data stores\n> + an array position within this list along with the most-significant bit\n> + on. Starting at that array position, iterate through this list of int-ids\n> + for the parents until reaching a value with the most-significant bit on.\n> + The other bits correspond to the int-id of the last parent.\n> +\n> +TRAILER:\n> +\n> + H-byte HASH-checksum of all of the above.\n> +\n> --\n> 2.7.4\n\nMakes sense so far, I'll read on.\nI agree with Junio, that I could read this documentation without\nthe urge to point out nits. :)\n\nThanks,\nStefan\n"},{"id":"339829","messageId":"xmqq8tbmywjy.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"1519066406-81663-6-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v4 05/13] commit-graph: implement 'git-commit-graph write'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-21T19:25:37Z","receivedAt":"2018-02-21T19:25:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> +static int graph_write(int argc, const char **argv)\n> +{\n> + ...\n> +\tgraph_name = write_commit_graph(opts.obj_dir);\n> +\n> +\tif (graph_name) {\n> +\t\tprintf(\"%s\\n\", graph_name);\n> +\t\tFREE_AND_NULL(graph_name);\n> +\t}\n> +\n> +\treturn 0;\n> +}\n\nAfter successfully writing a graph file out, write_commit_graph()\nsignals that fact by returning a non-NULL pointer, so that this\ncaller can report the filename to the end user. This caller\nprotects itself from a NULL return, presumably because the callee\nuses it to signal an error when writing the graph file out? \n\nIs it OK to lose that 1-bit of information, or should we have more like\n\n\tif (graph_name) {\n\t\tprintf;\n\t\treturn 0;\n\t} else {\n\t\treturn -1;\n\t}\n\n> int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n> {\n> \tstatic struct option builtin_commit_graph_options[] = {\n> -\t\t{ OPTION_STRING, 'p', \"object-dir\", &opts.obj_dir,\n> +\t\t{ OPTION_STRING, 'o', \"object-dir\", &opts.obj_dir,\n> \t\t\tN_(\"dir\"),\n> \t\t\tN_(\"The object directory to store the graph\") },\n> \t\tOPT_END(),\n\nThe same comment for a no-op patch from an earlier step applies\nhere, and we have another one that we saw above in graph_write().\n\n> @@ -31,6 +67,11 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n> \t\t\t builtin_commit_graph_usage,\n> \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n> \n> +\tif (argc > 0) {\n> +\t\tif (!strcmp(argv[0], \"write\"))\n> +\t\t\treturn graph_write(argc, argv);\n\nAnd if we fix \"graph_write\" to report an error with negative return,\nthis needs to become something like\n\n\t\treturn !!graph_write(argc, argv);\n\nas we do not want to return a negative value to be passed via\nrun_builtin() to exit(3) in handle_builtin().\n\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> new file mode 100755\n> index 0000000..6a5e93c\n> --- /dev/null\n> +++ b/t/t5318-commit-graph.sh\n> @@ -0,0 +1,119 @@\n> +#!/bin/sh\n> +\n> +test_description='commit graph'\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'setup full repo' '\n> +\trm -rf .git &&\n\nI am perfectly OK with creating a separate subdirectory called\n'full' in the trash directory given by the test framework, but\nunless absolutely necessary I'd rather not to see \"rm -rf\", \nespecially on \".git\", in our test scripts. People can screw up\ndoing various things (like copying and pasting).\n\n> +\tmkdir full &&\n> +\tcd full &&\n> +\tgit init &&\n> +\tobjdir=\".git/objects\"\n> +'\n\nAnd I absolutely do not want to see \"cd full\" that leaves and stays\nin the subdirectory after this step is done. \n\nImagine what happens if any earlier step fails before doing \"cd\nfull\", causing this \"setup full\" step to report failure, and then\nthe test goes on to the next step? We will not be in \"full\" and\nworse yet because we do not have \"$TRASH_DIRECTORY/.git\" (you\nremoved it), the \"git commit-graph write --object-dir\" command we\nend up doing next will see the git source repository as the\nrepository it is working on. Never risk trashing our source\nrepository with your test. That is why we give you $TRASH_DIRECTORY\nto play in. Make use of it when you can.\n\nI'd make this step just a single\n\n\tgit init full\n\nand then the next one\n\n\tgit -C full commit-graph write --object-dir .\n\nIn later tests that have multi-step things, I'd instead make them\n\n\t(\n\t\tcd full &&\n\t\t... whatever you do &&\n\t\t... in that separate &&\n\t\t... 'full' repository\n\t)\n\nif I were writing this test *and* if I really wanted to do things\ninside $TRASH_DIRECTORY/full/.git repository. I am not convinced\nyet about the latter. I know that it will make certain things\nsimpler to use a separate /full hierarchy (e.g. cleaning up, having\nanother unrelated test repository, etc.) while making other things\nmore cumbersome (e.g. you need to be careful when you \"cd\" and the\neasiest way to do so is to ( do things in a subshell )). I just do\nnot know what the trade-off would look like in this particular case.\n\nA simple rule of thumb I try to follow is not to change $(pwd) for\nthe process that runs these test_expect_success shell functions.\n\n> +\n> +test_expect_success 'write graph with no packs' '\n> +\tgit commit-graph write --object-dir .\n> +'\n> +\n> +test_expect_success 'create commits and repack' '\n> +\tfor i in $(test_seq 3)\n> +\tdo\n> +\t\ttest_commit $i &&\n> +\t\tgit branch commits/$i\n> +\tdone &&\n> +\tgit repack\n> +'\n"},{"id":"339830","messageId":"CAGZ79kZy1Z9GnHjA=NRoiszjP+pYwBZCKRps_RCgwa+x-jmrWA@mail.gmail.com","threadId":"47724","inReplyTo":"1519066406-81663-3-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v4 02/13] graph: add commit graph design document","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-21T19:34:54Z","receivedAt":"2018-02-21T19:35:01Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> +[3] https://public-inbox.org/git/20170907094718.b6kuzp2uhvkmwcso@sigill.intra.peff.net/t/#m7a2ea7b355aeda962e6b86404bcbadc648abfbba\n> + More discussion about generation numbers and not storing them inside\n> + commit objects. A valuable quote:\n\nUnlike the other public inbox links this links to a discussion with\nall messages on one page,\nhttps://public-inbox.org/git/20170908034739.4op3w4f2ma5s65ku@sigill.intra.peff.net/\nwould\nhave this be more inline with the other links. (this is a super small\nnit, which I am not sure if\nwe care about at all; the rest of the doc is awesome!)\n"},{"id":"339831","messageId":"fa9754a6-2cc9-5b2e-5d53-e8acfe3a1113@gmail.com","threadId":"47724","inReplyTo":"CAGZ79kawTtSzVV2=sNn_OFsmCbcLGhVi1pt4bVrOYfZA28E+6Q@mail.gmail.com","subject":"Re: [PATCH v4 01/13] commit-graph: add format document","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-21T19:45:13Z","receivedAt":"2018-02-21T19:45:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/21/2018 2:23 PM, Stefan Beller wrote:\n> On Mon, Feb 19, 2018 at 10:53 AM, Derrick Stolee wrote:\n>> +In order to allow extensions that add extra data to the graph, we organize\n>> +the body into \"chunks\" and provide a binary lookup table at the beginning\n>> +of the body. The header includes certain values, such as number of chunks,\n>> +hash lengths and types.\n>> +\n>> +All 4-byte numbers are in network order.\n>> +\n>> +HEADER:\n>> +\n>> + 4-byte signature:\n>> + The signature is: {'C', 'G', 'P', 'H'}\n>> +\n>> + 1-byte version number:\n>> + Currently, the only valid version is 1.\n>> +\n>> + 1-byte Object Id Version (1 = SHA-1)\n>> +\n>> + 1-byte number (C) of \"chunks\"\n>> +\n>> + 1-byte (reserved for later use)\n> What should clients of today do with it?\n> * ignore it completely [as they have no idea what it is] or\n> * throw hands up in the air if it is anything other than 0 ?\n> [because clearly we will increment the version\n> or have new information in a new chunk instead of just sneaking\n> in information here?]\n\nThey should ignore it completely, which will allow using the value for \nsomething meaningful later without causing a version change (which we DO \ndie() for). A user could downgrade from a version that uses this byte \nfor something meaningful and not require a new commit-graph file.\n\nThe \"commit-graph read\" subcommand does output this byte, so we can \nverify that the \"write\" subcommand places a 0 in this position.\n\n>\n>> +CHUNK LOOKUP:\n>> +\n>> + (C + 1) * 12 bytes listing the table of contents for the chunks:\n>> + First 4 bytes describe chunk id. Value 0 is a terminating label.\n>> + Other 8 bytes provide offset in current file for chunk to start.\n> offset [in bytes? I could imagine having a larger granularity here,\n> because chunks don't sound small.]\n\nIt is good to specify \"offset in bytes\".\n\n>\n>> + (Chunks are ordered contiguously in the file, so you can infer\n>> + the length using the next chunk position if necessary.)\n>> +\n>> + The remaining data in the body is described one chunk at a time, and\n>> + these chunks may be given in any order. Chunks are required unless\n>> + otherwise specified.\n>> +\n>> +CHUNK DATA:\n>> +\n>> + OID Fanout (ID: {'O', 'I', 'D', 'F'}) (256 * 4 bytes)\n>> + The ith entry, F[i], stores the number of OIDs with first\n>> + byte at most i. Thus F[255] stores the total\n>> + number of commits (N).\n> [ so in small repos, where there are fewer than 256 objects,\n> F[i] == F[i+1], for all i'th where there is no object starting with i byte]\n\nCorrect. I'm not sure this additional information is valuable for the \ndocument, though.\n\n>\n>> + OID Lookup (ID: {'O', 'I', 'D', 'L'}) (N * H bytes)\n>> + The OIDs for all commits in the graph, sorted in ascending order.\n>> +\n>> + Commit Data (ID: {'C', 'G', 'E', 'T' }) (N * (H + 16) bytes)\n>> + * The first H bytes are for the OID of the root tree.\n>> + * The next 8 bytes are for the int-ids of the first two parents\n>> + of the ith commit. Stores value 0xffffffff if no parent in that\n>> + position. If there are more than two parents, the second value\n>> + has its most-significant bit on and the other bits store an array\n>> + position into the Large Edge List chunk.\n>> + * The next 8 bytes store the generation number of the commit and\n>> + the commit time in seconds since EPOCH. The generation number\n>> + uses the higher 30 bits of the first 4 bytes, while the commit\n>> + time uses the 32 bits of the second 4 bytes, along with the lowest\n>> + 2 bits of the lowest byte, storing the 33rd and 34th bit of the\n>> + commit time.\n>> +\n>> + Large Edge List (ID: {'E', 'D', 'G', 'E'}) [Optional]\n>> + This list of 4-byte values store the second through nth parents for\n>> + all octopus merges. The second parent value in the commit data stores\n>> + an array position within this list along with the most-significant bit\n>> + on. Starting at that array position, iterate through this list of int-ids\n>> + for the parents until reaching a value with the most-significant bit on.\n>> + The other bits correspond to the int-id of the last parent.\n>> +\n>> +TRAILER:\n>> +\n>> + H-byte HASH-checksum of all of the above.\n>> +\n>> --\n>> 2.7.4\n> Makes sense so far, I'll read on.\n> I agree with Junio, that I could read this documentation without\n> the urge to point out nits. :)\n>\n> Thanks,\n> Stefan\n\n"},{"id":"339832","messageId":"CAGZ79kY2rAG2ac06fSFMb1nr1Nh4CKE7PoqDPnh7RZg6qQHLaQ@mail.gmail.com","threadId":"47724","inReplyTo":"fa9754a6-2cc9-5b2e-5d53-e8acfe3a1113@gmail.com","subject":"Re: [PATCH v4 01/13] commit-graph: add format document","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-21T19:48:35Z","receivedAt":"2018-02-21T19:48:42Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":">>\n>> [ so in small repos, where there are fewer than 256 objects,\n>> F[i] == F[i+1], for all i'th where there is no object starting with i\n>> byte]\n>\n>\n> Correct. I'm not sure this additional information is valuable for the\n> document, though.\n\nIt is not, I was just making sure I'd understand correctly.\n"},{"id":"339835","messageId":"xmqq371uyufx.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"1519066406-81663-7-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v4 06/13] commit-graph: implement git commit-graph read","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-21T20:11:14Z","receivedAt":"2018-02-21T20:11:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> +'read'::\n> +\n> +Read a graph file given by the graph-head file and output basic\n> +details about the graph file.\n> ++\n> +With `--file=` option, consider the graph stored in the file at\n> +the path /info/.\n> +\n\nA sample reader confusion after reading the above twice:\n\n What is \"the graph-head file\" and how does the user specify it? Is\n it given by the value for the \"--file=\" command line option?\n\nAnother sample reader reaction after reading the above:\n\n What are the kind of \"basic details\" we can learn from this\n command is unclear, but perhaps there is an example to help me\n decide if this command is worth studying.\n\n> @@ -44,6 +53,12 @@ EXAMPLES\n> $ git commit-graph write\n> ------------------------------------------------\n> \n> +* Read basic information from a graph file.\n> ++\n> +------------------------------------------------\n> +$ git commit-graph read --file=\n> +------------------------------------------------\n> +\n\nAnd the sample reader is utterly disappointed at this point.\n\n> +static int graph_read(int argc, const char **argv)\n> +{\n> +\tstruct commit_graph *graph = 0;\n> +\tstruct strbuf full_path = STRBUF_INIT;\n> +\n> +\tstatic struct option builtin_commit_graph_read_options[] = {\n> +\t\t{ OPTION_STRING, 'o', \"object-dir\", &opts.obj_dir,\n> +\t\t\tN_(\"dir\"),\n> +\t\t\tN_(\"The object directory to store the graph\") },\n> +\t\t{ OPTION_STRING, 'H', \"file\", &opts.graph_file,\n> +\t\t\tN_(\"file\"),\n> +\t\t\tN_(\"The filename for a specific commit graph file in the object directory.\"),\n> +\t\t\tPARSE_OPT_OPTARG, NULL, (intptr_t) \"\" },\n> +\t\tOPT_END(),\n> +\t};\n\nThe same comment as all the previous ones apply, wrt short options\nand non-use of OPT_STRING().\n\nAlso, I suspect that these two would want to use OPT_FILENAME\ninstead, if we anticipate that the command might want to be\nsometimes run from a subdirectory. Otherwise wouldn't\n\n\tcd t && git commit-graph read --file=../.git/object/info/$whatever\n\nend up referring to a wrong place because the code that uses the\nvalue obtained from OPTION_STRING does not do the equivalent of\nparse-options.c::fix_filename()? The same applies to object-dir\nhandling.\n\n> +\targc = parse_options(argc, argv, NULL,\n> +\t\t\t builtin_commit_graph_read_options,\n> +\t\t\t builtin_commit_graph_read_usage, 0);\n> +\n> +\tif (!opts.obj_dir)\n> +\t\topts.obj_dir = get_object_directory();\n> +\n> +\tif (!opts.graph_file)\n> +\t\tdie(\"no graph hash specified\");\n> +\n> +\tstrbuf_addf(&full_path, \"%s/info/%s\", opts.obj_dir, opts.graph_file);\n\nAhh, I was fooled by a misnamed option. --file does *not* name the\nfile. It is a filename in a fixed place that is determined by other\nthings.\n\nSo it would be a mistake to use OPT_FILENAME() in the parser for\nthat misnamed \"--file\" option. The parser for --object-dir still\nwould want to be OPT_FILENAME(), but quite honestly, I do not see\nthe point of having --object-dir option in the first place. The\ngraph file is not relative to it but is forced to have /info/ in\nbetween that directory and the filename, so it is not like the user\ngets useful flexibility out of being able to specify two different\nplaces using --object-dir= option and $GIT_OBJECT_DIRECTORY\nenvironment (iow, a caller that wants to work on a specific object\ndirectory can use the environment, which is how it would tell any\nother Git subcommand which object store it wants to work with).\n\nBut stepping back a bit, I think the way --file argument is defined\nis halfway off from two possible more useful ways to define it. If\nit were just \"path to the file\" (iow, what OPT_FILENAME() is suited\nfor parsing it), then a user could say \"I have this graph file that\nI created for testing, it is not installed in its usual place in\n$GIT_OBJECT_DIRECTORY/info/ at all, but I want you to read it\nbecause I am debugging\". That is one possible useful extreme. The\nother possibility would be to allow *only* the hash part to be\nspecified, iow, not just forcing /info/ relative to object\ndirectory, you would force the \"graph-\" prefix and \".graph\" suffix.\nThat would be the other extreme that is useful (less typing and less\nerror prone).\n\nFor a low-level command line this, my gut feeling is that it would\nbe better to allow paths to the object directory and the graph file\nto be totally independently specified.\n\n> +\tif (graph_signature != GRAPH_SIGNATURE) {\n> +\t\tmunmap(graph_map, graph_size);\n> +\t\tclose(fd);\n> +\t\tdie(\"graph signature %X does not match signature %X\",\n> +\t\t\tgraph_signature, GRAPH_SIGNATURE);\n> +\t}\n> +\n> +\tgraph_version = *(unsigned char*)(data + 4);\n> +\tif (graph_version != GRAPH_VERSION) {\n> +\t\tmunmap(graph_map, graph_size);\n> +\t\tclose(fd);\n> +\t\tdie(\"graph version %X does not match version %X\",\n> +\t\t\tgraph_version, GRAPH_VERSION);\n> +\t}\n> +\n> +\thash_version = *(unsigned char*)(data + 5);\n> +\tif (hash_version != GRAPH_OID_VERSION) {\n> +\t\tmunmap(graph_map, graph_size);\n> +\t\tclose(fd);\n> +\t\tdie(\"hash version %X does not match version %X\",\n> +\t\t\thash_version, GRAPH_OID_VERSION);\n\nIt becomes a bit tiring to see munmap/close/die pattern repreated\nover and over again, doesn't it? Can we make it simpler, perhaps by\nletting die() take care of the clean-up? After all, if the very\nnext step dies because alloc_commit_graph() got NULL in xmalloc(),\nwe are letting die() there take care of the clean-up anyway already,\nand die() in the chunk parsing look has no such cleanup, either.\n\nOf course, when we later want to libify this part of the code, then\nwe wouldn't be calling die() from this codepath, but the change\nrequired to do so will not be just s/die/error/; it would be more\nlike\n\n\tif (x_version != X_VERSION) {\n\t\terror(\"X version %X does not match\",...);\n\t\tgoto cleanup_fail;\n\t}\n\nwith munmap/close done at the jumped-to label.\n\n> +\t}\n> +\n> +\tgraph = alloc_commit_graph();\n> +\n> +\tgraph->hash_len = GRAPH_OID_LEN;\n> + ...\n> +\t\tif (chunk_offset > graph_size - GIT_MAX_RAWSZ)\n> +\t\t\tdie(\"improper chunk offset %08x%08x\", (uint32_t)(chunk_offset >> 32),\n> +\t\t\t (uint32_t)chunk_offset);\n> +\n> +\t\tswitch (chunk_id) {\n> +\t\t\tcase GRAPH_CHUNKID_OIDFANOUT:\n> +\t\t\t\tgraph->chunk_oid_fanout = (uint32_t*)(data + chunk_offset);\n> +\t\t\t\tbreak;\n\nThis is over-indented from our point of view. In our codebase, case\narms aling with switch, i.e.\n\n\t\tswitch (chunk_id) {\n\t\tcase GRAPH_CHUNKID_OIDFANOUT:\n\t\t\tgraph->chunk_oid_fanout = ...;\n\t\t\tbreak;\n\nWhen the input file has GRAPH_CHUNKID_OIDFANOUT twice, I think it\nshould be flagged as a corrupt/malformed input file, causing the\nreader to reject it. It is plausible that you wanted to make it\n\"the last one wins\", but even if that is the case, I think the user\nshould at least get a warning, as (I'd imagine) it is an unusual\ncondition.\n\nThe same applies to multiple instances of any currently-defined\nchunk types.\n\n> +graph_read_expect() {\n> +\tOPTIONAL=\"\"\n> +\tNUM_CHUNKS=3\n> +\tif [ ! -z $2 ]\n\nWe use \"test\" and do not use \"[ ... ]\" or \"[[ ... ]]\".\n\nI'll stop here.\n\n"},{"id":"339839","messageId":"CAGZ79kai=KegG-XPmtZJRhTcD-M5Edsw8eocje41mdbL-RARAA@mail.gmail.com","threadId":"47724","inReplyTo":"1519066406-81663-9-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v4 08/13] commit-graph: implement --delete-expired","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-21T21:34:50Z","receivedAt":"2018-02-21T21:34:57Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Feb 19, 2018 at 10:53 AM, Derrick Stolee wrote:\n\n> graph_name = write_commit_graph(opts.obj_dir);\n>\n> if (graph_name) {\n> if (opts.set_latest)\n> set_latest_file(opts.obj_dir, graph_name);\n>\n> + if (opts.delete_expired)\n> + do_delete_expired(opts.obj_dir,\n> + old_graph_name,\n> + graph_name);\n> +\n\nSo this only allows to delete expired things and setting the latest\nwhen writing a new graph. Would we ever envision a user to produce\na new graph (e.g. via obtaining a graph that they got from a server) and\nthen manually rerouting the latest to that new graph file without writing\nthat graph file in the same process? The same for expired.\n\nI guess these operations are just available via editing the\nlatest or deleting files manually, which slightly contradicts\ne.g. \"git update-ref\", which in olden times was just a fancy way\nof rewriting the refs file manually. (though it claims to be less\nprone to errors as it takes lock files)\n\n>\n> extern char *get_graph_latest_filename(const char *obj_dir);\n> +extern char *get_graph_latest_contents(const char *obj_dir);\n\nDid\nhttps://public-inbox.org/git/20180208213806.GA6381@sigill.intra.peff.net/\never make it into tree? (It is sort of new, but I feel we'd want to\nstrive for consistency in the code base, eventually.)\n"},{"id":"339847","messageId":"CAGZ79kZbdvehOQMZj3NkkSt5RoArtv0ixb8pHvFDrHTrWmVuDg@mail.gmail.com","threadId":"47724","inReplyTo":"1519066406-81663-13-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v4 12/13] commit-graph: read only from specific pack-indexes","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-02-21T22:25:09Z","receivedAt":"2018-02-21T22:25:17Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Feb 19, 2018 at 10:53 AM, Derrick Stolee wrote:\n>\n> Teach git-commit-graph to inspect the objects only in a certain list\n> of pack-indexes within the given pack directory. This allows updating\n> the commit graph iteratively, since we add all commits stored in a\n> previous commit graph.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-commit-graph.txt | 11 +++++++++++\n> builtin/commit-graph.c | 32 +++++++++++++++++++++++++++++---\n> commit-graph.c | 26 ++++++++++++++++++++++++--\n> commit-graph.h | 4 +++-\n> packfile.c | 4 ++--\n> packfile.h | 2 ++\n> t/t5318-commit-graph.sh | 16 ++++++++++++++++\n> 7 files changed, 87 insertions(+), 8 deletions(-)\n>\n> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> index b9b4031..93d50d1 100644\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -42,6 +42,10 @@ With the `--delete-expired` option, delete the graph files in the pack\n> directory that are not referred to by the graph-latest file. To avoid race\n> conditions, do not delete the file previously referred to by the\n> graph-latest file if it is updated by the `--set-latest` option.\n> ++\n> +With the `--stdin-packs` option, generate the new commit graph by\n> +walking objects only in the specified packfiles and any commits in\n> +the existing graph-head.\n\nA general question on this series:\nHow do commit graph buildups deal with garbage collected commits?\n(my personal workflow is heavy on rebase, which generates lots of\ndangling commits, to be thrown out later)\n\nThe second half of the sentence makes it sound like once a\ncommit is in the graph it cannot be pulled out easily again, hence\nthe question on the impact of graphs on a long living repository\nwhich is garbage collected frequently.\n\nAFAICT you could just run\n git commit-graph write --set-latest [--delete-expired]\nas that actually looks up objects from outside the existing graph files,\nsuch that lost objects are ignored?\n\n> + const char **lines = NULL;\n> + int nr_lines = 0;\n> + int alloc_lines = 0;\n\n(nit:)\nI had the impression that these triplet-variables, that are used in\nALLOC_GROW are allo X, X_nr and X_allow, but I might be wrong.\n\n> @@ -170,7 +178,25 @@ static int graph_write(int argc, const char **argv)\n>\n> old_graph_name = get_graph_latest_contents(opts.obj_dir);\n>\n> - graph_name = write_commit_graph(opts.obj_dir);\n> + if (opts.stdin_packs) {\n> + struct strbuf buf = STRBUF_INIT;\n> + nr_lines = 0;\n> + alloc_lines = 128;\n\nalloc_lines has been initialized before, so why redo it here again?\nAlso what is the rationale for choosing 128 as a good default?\nI would guess 0 is just as fine, because ALLOC_GROW makes sure\nthat it growth fast in the first couple entries by having an additional\noffset. (no need to fine tune the starting allocation IMHO)\n\n> + ALLOC_ARRAY(lines, alloc_lines);\n> +\n> + while (strbuf_getline(&buf, stdin) != EOF) {\n> + ALLOC_GROW(lines, nr_lines + 1, alloc_lines);\n> + lines[nr_lines++] = buf.buf;\n> + strbuf_detach(&buf, NULL);\n\nstrbuf_detach returns its previous buf.buf, such that you can combine these\ntwo lines as\n lines[nr_lines++] = strbuf_detach(&buf, NULL);\n\n\n> + }\n> +\n> + pack_indexes = lines;\n> + nr_packs = nr_lines;\n\nTechnically we do not need to strbuf_release(&buf) here, because\nstrbuf_detach is always called, and by knowing its implementation,\nit is just as good.\n\n\n> @@ -579,7 +581,27 @@ char *write_commit_graph(const char *obj_dir)\n> oids.alloc = 1024;\n> ALLOC_ARRAY(oids.list, oids.alloc);\n>\n> - for_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n> + if (pack_indexes) {\n> + struct strbuf packname = STRBUF_INIT;\n> + int dirlen;\n> + strbuf_addf(&packname, \"%s/pack/\", obj_dir);\n> + dirlen = packname.len;\n> + for (i = 0; i < nr_packs; i++) {\n> + struct packed_git *p;\n> + strbuf_setlen(&packname, dirlen);\n> + strbuf_addstr(&packname, pack_indexes[i]);\n> + p = add_packed_git(packname.buf, packname.len, 1);\n> + if (!p)\n> + die(\"error adding pack %s\", packname.buf);\n> + if (open_pack_index(p))\n> + die(\"error opening index for %s\", packname.buf);\n> + for_each_object_in_pack(p, if_packed_commit_add_to_list, &oids);\n> + close_pack(p);\n> + }\n\nstrbuf_release(&packname);\n\n> + }\n> + else\n\n(micro style nit)\n\n } else\n"},{"id":"339933","messageId":"xmqqo9kgx4op.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"xmqq371uyufx.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 06/13] commit-graph: implement git commit-graph read","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-22T18:25:10Z","receivedAt":"2018-02-22T18:25:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Derrick Stolee writes:\n>\n>> +'read'::\n>> +\n>> +Read a graph file given by the graph-head file and output basic\n>> +details about the graph file.\n>> ++\n>> +With `--file=` option, consider the graph stored in the file at\n>> +the path /info/.\n>> +\n>\n> A sample reader confusion after reading the above twice:\n>\n> What is \"the graph-head file\" and how does the user specify it? Is\n> it given by the value for the \"--file=\" command line option?\n\nThis confusion is somewhat lightened with s/graph-head/graph-latest/\n(I just saw 07/13 to realize that the file is renamed).\n\nPerhaps describe it as \"Read the graph file currently active\n(i.e. the one pointed at by graph-latest file in the object/info\ndirectory) and output blah\" + \"With --file parameter, read the\ngraph file specified with that parameter instead\"?\n"},{"id":"339938","messageId":"xmqqk1v4x4dt.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"1519066406-81663-8-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v4 07/13] commit-graph: implement --set-latest","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-22T18:31:42Z","receivedAt":"2018-02-22T18:31:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> static struct opts_commit_graph {\n> \tconst char *obj_dir;\n> \tconst char *graph_file;\n> +\tint set_latest;\n> } opts;\n> ...\n> @@ -89,6 +106,8 @@ static int graph_write(int argc, const char **argv)\n> \t\t{ OPTION_STRING, 'o', \"object-dir\", &opts.obj_dir,\n> \t\t\tN_(\"dir\"),\n> \t\t\tN_(\"The object directory to store the graph\") },\n> +\t\tOPT_BOOL('u', \"set-latest\", &opts.set_latest,\n> +\t\t\tN_(\"update graph-head to written graph file\")),\n> \t\tOPT_END(),\n> \t};\n> \n> @@ -102,6 +121,9 @@ static int graph_write(int argc, const char **argv)\n> \tgraph_name = write_commit_graph(opts.obj_dir);\n> \n> \tif (graph_name) {\n> +\t\tif (opts.set_latest)\n> +\t\t\tset_latest_file(opts.obj_dir, graph_name);\n> +\n\nThis feels like a very strange API from potential caller's point of\nview. Because you have to decide that you are going to mark it as\nthe latest one upfront before actually writing the graph file, if\nyou forget to pass --set-latest, you have to know how to manually\nmark the file as latest anyway. I would understand if it were one\nof the following:\n\n (1) whenever a new commit graph file is written in the\n objects/info/ directory, always mark it as the latest (drop\n --set-latest option altogether); or\n\n (2) make set-latest command that takes a name of an existing graph\n file in the objects/info/ directory, and sets the latest\n pointer to point at it (make it separate from 'write' command).\n\nthough.\n\n"},{"id":"339941","messageId":"xmqqfu5sx3md.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"1519066406-81663-9-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v4 08/13] commit-graph: implement --delete-expired","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-22T18:48:10Z","receivedAt":"2018-02-22T18:53:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> Teach git-commit-graph to delete the .graph files that are siblings of a\n> newly-written graph file, except for the file referenced by 'graph-latest'\n> at the beginning of the process and the newly-written file. If we fail to\n> delete a graph file, only report a warning because another git process may\n> be using that file. In a multi-process environment, we expect the previoius\n> graph file to be used by a concurrent process, so we do not delete it to\n> avoid race conditions.\n\nI do not understand the later part of the above. On some operating\nsystems, you actually can remove a file that is open by another\nprocess without any ill effect. There are systems that do not allow\nremoving a file that is in use, and an attempt to unlink it may\nfail. The need to handle such a failure gracefully is not limited\nto the case of removing a commit graph file---we need to deal with\nit when removing file of _any_ type.\n\nEspecially the last sentence \"we do not delete it to avoid race\nconditions\" I find problematic. If a system does not allow removing\na file in use and we detect a failure after an attempt to do so, it\nis not \"we do not delete it\" --- even if you do, you won't succeed\nanyway, so there is no point saying that. And on systems that do\nallow safe removal of a file in use (i.e. they allow an open file to\nbe used by processes that have open filehandles to it after its\nremoval), there is no point refraining to delete it \"to avoid race\nconditions\", either---in fact it is unlikely that you would even know\nsomebody else had it open and was using it.\n\nIn any case, I do not think '--delete-expired' option that can be\ngiven only when you are writing makes much sense as an API. An\n'expire' command, just like 'set-latest' command, that is a separate\ncommand from 'write', may make sense, though.\n"},{"id":"340037","messageId":"de1d2550-afb4-d3d4-663c-b0a60411dcdd@gmail.com","threadId":"47724","inReplyTo":"xmqq1shf2v8d.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 02/13] graph: add commit graph design document","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-23T15:44:56Z","receivedAt":"2018-02-23T15:45:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/20/2018 4:42 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> +2. Walking the entire graph to avoid topological order mistakes.\n> You have at least one more mention of \"topological order mistakes\"\n> below, but we commonly refer to this issue and blame it for \"clock\n> skew\". Using the word highlights that there is no \"mistake\" in topo\n> order algorithm and mistakes are in the commit timestamps.\n\nI'll drop the word \"mistakes\" and instead here say:\n\n   2. Walking the entire graph to satisfy topological order constraints.\n\nand later say\n\n   This heuristic is currently used whenever the computation is allowed to\n   violate topological relationships due to clock skew (such as \"git log\"\n   with default order), but is not used when the topological order is\n   required (such as merge base calculations, \"git log --graph\").\n\n>\n>> +In practice, we expect some commits to be created recently and not stored\n>> +in the commit graph. We can treat these commits as having \"infinite\"\n>> +generation number and walk until reaching commits with known generation\n>> +number.\n> Hmm, \"pretend infinity\" is an interesting approach---I need to think\n> about it a bit more if it is sufficient.\n\nSince we require the commit graph file to be closed under reachability, \nthe commits reachable from the file all have \"finite\" generation number.\n\n>\n>> +- .graph files are managed only by the 'commit-graph' builtin. These are not\n>> + updated automatically during clone, fetch, repack, or creating new commits.\n> OK. s/builtin/subcommand/; it does not make much difference if it\n> is a built-in or standalone command.\n>\n>> +- There is no 'verify' subcommand for the 'commit-graph' builtin to verify\n>> + the contents of the graph file agree with the contents in the ODB.\n> I am not entirely sure about the merit of going into this level of\n> detail. Being able to use only a single file looks like a more\n> fundamental design limitation, which deserves to be decribed in this\n> section, and we could ship the subsystem with that limitation.\n>\n> But the lack of verify that can be called from fsck is merely the\n> matter of not the subsystem being mature enough (to be written,\n> reviewed and tested) and not a fundamental one, and we will not be\n> shipping the subsystem until that limitation is lifted.\n>\n> So I'd guess that we prefer this bullet item to be in the commit log\n> message, not here, that describes the current status of the\n> development (as opposed to the state of the subsystem).\n\nI was treating this design document as a living document that will be \nupdated as the feature matures. It is difficult to time when to discuss \nthese limitations, since in this commit the graph feature is not \nimplemented at all. But, it is important to have _some_ design document \nbefore continuing to implement the feature.\n\nI can remove this bullet, but I'm not sure which commit message would be \nappropriate to contain that information.\n\nI do intend to remove these limitations and future work bullets as they \nare implemented in later patches.\n\n>\n>> +- Generation numbers are not computed in the current version. The file\n>> + format supports storing them, along with a mechanism to upgrade from\n>> + a file without generation numbers to one that uses them.\n> Exactly the same comment as above applies to this item.\n>\n>> +- The commit graph is currently incompatible with commit grafts. This can be\n>> + remedied by duplicating or refactoring the current graft logic.\n> Hmm. Can it be lifted without first allowing us to use more than\n> one commit graph file (i.e. one for \"traverse while honoring the\n> grafts\", the other for \"traverse while ignoring the grafts\")?\n\nI consider this list unordered, but will move this bullet to the top and \nreplace its first sentence with:\n\n   The commit graph feature currently does not honor commit grafts.\n\n>\n>> +- After computing and storing generation numbers, we must make graph\n>> + walks aware of generation numbers to gain the performance benefits they\n>> + enable. This will mostly be accomplished by swapping a commit-date-ordered\n>> + priority queue with one ordered by generation number. The following\n>> + operations are important candidates:\n>> +\n>> + - paint_down_to_common()\n>> + - 'log --topo-order'\n> Yes.\n>\n>> +- The graph currently only adds commits to a previously existing graph.\n>> + When writing a new graph, we could check that the ODB still contains\n>> + the commits and choose to remove the commits that are deleted from the\n>> + ODB. For performance reasons, this check should remain optional.\n> The last sentence is somehow unconvincing. It probably is not\n> appropriate for the \"Future Work\" section to be making a hurried\n> design decision before having any working verification code to run\n> benchmark on.\n\nI'll remove this entire block, since it is not relevant starting at v4. \nI dropped this \"additive only\" step in v4 and forgot to remove the bullet.\n\n>\n>> +- Currently, parse_commit_gently() requires filling in the root tree\n>> + object for a commit. This passes through lookup_tree() and consequently\n>> + lookup_object(). Also, it calls lookup_commit() when loading the parents.\n>> + These method calls check the ODB for object existence, even if the\n>> + consumer does not need the content. For example, we do not need the\n>> + tree contents when computing merge bases. Now that commit parsing is\n>> + removed from the computation time, these lookup operations are the\n>> + slowest operations keeping graph walks from being fast. Consider\n>> + loading these objects without verifying their existence in the ODB and\n>> + only loading them fully when consumers need them. Consider a method\n>> + such as \"ensure_tree_loaded(commit)\" that fully loads a tree before\n>> + using commit->tree.\n> Very good idea.\n\nI will likely submit an orthogonal patch that does this, as it will save \ntime even without the commit graph. The time spent in 'lookup_tree()' is \nless significant when the majority of the time is spent parsing commits, \nbut it is still 1-2% in some cases.\n\n>\n>> +- The current design uses the 'commit-graph' builtin to generate the graph.\n>> + When this feature stabilizes enough to recommend to most users, we should\n>> + add automatic graph writes to common operations that create many commits.\n>> + For example, one coulde compute a graph on 'clone', 'fetch', or 'repack'\n>> + commands.\n> s/coulde/could/.\n>\n> Also do not forget \"fsck\" that calls \"verify\". That is more urgent\n> than intergration with any other subcommand.\n\nNoted.\n\n>\n>> +- A server could provide a commit graph file as part of the network protocol\n>> + to avoid extra calculations by clients.\n> We need to assess the riskiness and threat models regarding this, if\n> we really want to follow this \"could\" through. I would imagine that\n> the cost for verification is comparable to the cost for regenerating,\n> in which case it may not be worth doing this _unless_ the user opts\n> into it saying that the other side over the wire is trusted without\n> any reservation.\n\nI agree. There is a certain level of trust that is required here.\n\nThanks,\n-Stolee\n"},{"id":"340038","messageId":"c9d38f72-536b-887b-eaa7-3395746ef8e3@gmail.com","threadId":"47724","inReplyTo":"xmqqd10yyxsu.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 03/13] commit-graph: create git-commit-graph builtin","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-23T16:07:20Z","receivedAt":"2018-02-23T16:07:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/21/2018 1:58 PM, Junio C Hamano wrote:\n> Junio C Hamano writes:\n>\n>> Derrick Stolee writes:\n>>\n>>> +int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n>>> +{\n>>> +\tstatic struct option builtin_commit_graph_options[] = {\n>>> +\t\t{ OPTION_STRING, 'p', \"object-dir\", &opts.obj_dir,\n>>> +\t\t\tN_(\"dir\"),\n>>> +\t\t\tN_(\"The object directory to store the graph\") },\n>> I have a suspicion that this was modeled after some other built-in\n>> that has a similar issue (perhaps written long time ago), but isn't\n>> OPT_STRING() sufficient to define this element these days?\n>>\n>> Or am I missing something?\n\nYou are not. There are several places in this history of this patch \nwhere I was using old patterns because I was using old code as my model \n(places like 'index-pack').\n\n>> Why squat on short-and-sweet \"-p\"? For that matter, since this is\n>> not expected to be end-user facing command anyway, I suspect that we\n>> do not want to allocate a single letter option from day one, which\n>> paints ourselves into a corner from where we cannot escape.\n\nI'll drop all single-letter shortcuts.\n\n> I suspect that exactly the same comment applies to patches in this\n> series that add other subcommands (I just saw one in the patch for\n> adding 'write').\n>\n\nThanks,\n-Stolee\n"},{"id":"340045","messageId":"4d453f29-8e5b-2a98-ab11-415f63613be8@gmail.com","threadId":"47724","inReplyTo":"xmqqmv031d7d.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 04/13] commit-graph: implement write_commit_graph()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-23T17:23:26Z","receivedAt":"2018-02-23T17:23:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/20/2018 5:57 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> +#define GRAPH_OID_VERSION_SHA1 1\n>> +#define GRAPH_OID_LEN_SHA1 20\n> This hardcoded 20 on the right hand side of this #define is probably\n> problematic. Unless you are planning to possibly store truncated\n> hash value for some future hash algorithm, GRAPH_OID_LEN_$HASH should\n> always be the same as GIT_$HASH_RAWSZ, I would think. IOW\n>\n> #define GRAPH_OID_LEN_SHA1 GIT_SHA1_RAWSZ\n>\n> perhaps?\n\nYes.\n\n>\n>> +static void write_graph_chunk_fanout(struct sha1file *f,\n>> +\t\t\t\t struct commit **commits,\n>> +\t\t\t\t int nr_commits)\n>> +{\n>> +\tuint32_t i, count = 0;\n>> +\tstruct commit **list = commits;\n>> +\tstruct commit **last = commits + nr_commits;\n>> +\n>> +\t/*\n>> +\t * Write the first-level table (the list is sorted,\n>> +\t * but we use a 256-entry lookup to be able to avoid\n>> +\t * having to do eight extra binary search iterations).\n>> +\t */\n>> +\tfor (i = 0; i < 256; i++) {\n>> +\t\twhile (list < last) {\n>> +\t\t\tif ((*list)->object.oid.hash[0] != i)\n>> +\t\t\t\tbreak;\n>> +\t\t\tcount++;\n>> +\t\t\tlist++;\n>> +\t\t}\n> If count and list are always incremented in unison, perhaps you do\n> not need an extra variable \"last\". If typeof(nr_commits) is wider\n> than typeof(count), this loop and the next write-be32 is screwed\n> anyway ;-)\n>\n> This comment probably applies equally to some other uses of the same\n> \"compute last pointer to compare with running pointer for\n> termination\" pattern in this patch.\n\nYes. Also turning i and count into int to match nr_commits.\n\n>\n>> +\t\tsha1write_be32(f, count);\n>> +\t}\n>> +}\n>> +static int commit_pos(struct commit **commits, int nr_commits,\n>> +\t\t const struct object_id *oid, uint32_t *pos)\n>> +{\n> It is a bit unusual to see something_pos() that returns an integer\n> that does *NOT* return the position as its return value. Dropping\n> the *pos parameter, and returning \"mid\" when commits[mid] is what we\n> wanted to see, and otherwise returning \"-1 - first\" to signal the\n> position at which we _would_ have found the object, if it were in\n> the table, would make it more consistent with the usual convention.\n\nI can make this change. I found the boolean return to make the \nconsumer's logic simpler, but it isn't that much simpler.\n\n> Don't we even have such a generalized binary search helper already\n> somewhere in the system?\n\njt/binsearch-with-fanout introduces one when there is a 256-entry fanout \ntable (not the case here).\n\nThe bsearch() method in search.h (and used in \npack-write.c:need_large_offset) does not return the _position_ of a \nfound element.\n\nNeither of these suit my needs, but I could just be searching for the \nwrong strings. Also, I could divert my energies in this area to create a \ngeneric search in the style of jt/binsearch-with-fanout.\n\n>\n>> +static void write_graph_chunk_data(struct sha1file *f, int hash_len,\n>> +\t\t\t\t struct commit **commits, int nr_commits)\n>> +{\n>> +\tstruct commit **list = commits;\n>> +\tstruct commit **last = commits + nr_commits;\n>> +\tuint32_t num_large_edges = 0;\n>> +\n>> +\twhile (list < last) {\n>> +\t\tstruct commit_list *parent;\n>> +\t\tuint32_t int_id;\n>> +\t\tuint32_t packedDate[2];\n>> +\n>> +...\n>> +\t\tif (!parent)\n>> +\t\t\tint_id = GRAPH_PARENT_NONE;\n>> +\t\telse if (parent->next)\n>> +\t\t\tint_id = GRAPH_LARGE_EDGES_NEEDED | num_large_edges;\n>> +\t\telse if (!commit_pos(commits, nr_commits,\n>> +\t\t\t\t &(parent->item->object.oid), &int_id))\n>> +\t\t\tint_id = GRAPH_PARENT_MISSING;\n>> +\n>> +\t\tsha1write_be32(f, int_id);\n>> +\n>> +\t\tif (parent && parent->next) {\n> This is equivalent to checking \"int_id & GRAPH_LARGE_EDGES_NEEDED\",\n> right? Not suggesting to use the other form of checks, but trying\n> to see what's the best way to express it in the most readable way.\n\nYou're right, we already set the bit above, so let's make use of that \ncheck. Important to note that GRAPH_LARGE_EDGES_NEEDED & \nGRAPH_PARENT_MISSING == 0.\n\n>\n>> +\t\t\tdo {\n>> +\t\t\t\tnum_large_edges++;\n>> +\t\t\t\tparent = parent->next;\n>> +\t\t\t} while (parent);\n> It feels somewhat wasteful to traverse the commit's parents list\n> only to count, without populating the octopus table (which I\n> understand is assumed to be minority case under this design).\n\nSince we are writing the commit graph file in-order, we cannot write the \noctopus table until after the chunk lengths are known. We could store \nthe octopus table in memory and then dump it into the file later, but \nwalking the parents is quite fast after all the commits are loaded. I'm \nnot sure the time optimization merits the extra complexity here. (I'm \nhappy to revisit this if we do see this performance lacking.)\n\nP.S. I really like the name \"octopus table\" and will use that for \ninformal discussions of this format.\n\n>\n>> +\t\t}\n>> +\n>> +\t\tif (sizeof((*list)->date) > 4)\n>> +\t\t\tpackedDate[0] = htonl(((*list)->date >> 32) & 0x3);\n>> +\t\telse\n>> +\t\t\tpackedDate[0] = 0;\n> OK, the undefined pattern in the previous round is now gone ;-) Good.\n>\n>> +\t\tpackedDate[1] = htonl((*list)->date);\n>> +\t\tsha1write(f, packedDate, 8);\n>> +\n>> +\t\tlist++;\n>> +\t}\n>> +}\n>> +\n>> +static void write_graph_chunk_large_edges(struct sha1file *f,\n>> +\t\t\t\t\t struct commit **commits,\n>> +\t\t\t\t\t int nr_commits)\n>> +{\n>> +\tstruct commit **list = commits;\n>> +\tstruct commit **last = commits + nr_commits;\n>> +\tstruct commit_list *parent;\n>> +\n>> +\twhile (list < last) {\n>> +\t\tint num_parents = 0;\n>> +\t\tfor (parent = (*list)->parents; num_parents < 3 && parent;\n>> +\t\t parent = parent->next)\n>> +\t\t\tnum_parents++;\n>> +\n>> +\t\tif (num_parents <= 2) {\n>> +\t\t\tlist++;\n>> +\t\t\tcontinue;\n>> +\t\t}\n>> +\n>> +\t\t/* Since num_parents > 2, this initializer is safe. */\n>> +\t\tfor (parent = (*list)->parents->next; parent; parent = parent->next) {\n>> +\t\t\tuint32_t int_id, swap_int_id;\n>> +\t\t\tuint32_t last_edge = 0;\n>> +\t\t\tif (!parent->next)\n>> +\t\t\t\tlast_edge |= GRAPH_LAST_EDGE;\n>> +\n>> +\t\t\tif (commit_pos(commits, nr_commits,\n>> +\t\t\t\t &(parent->item->object.oid),\n>> +\t\t\t\t &int_id))\n>> +\t\t\t\tswap_int_id = htonl(int_id | last_edge);\n>> +\t\t\telse\n>> +\t\t\t\tswap_int_id = htonl(GRAPH_PARENT_MISSING | last_edge);\n>> +\t\t\tsha1write(f, &swap_int_id, 4);\n> What does \"swap_\" in the name of this variable mean? For some\n> archs, there is no swap. The only difference between int_id and the\n> variable is that its MSB may possibly be smudged with last_edge bit.\n\nSorry, I tried to catch all of these, but some fell through the cracks. \nI should be using sha1write_be32() after modifying int_id directly.\n\nThis whole block is a bit of a mess. I'll replace it with something like:\n\n                         uint32_t int_id;\n                         if (!commit_pos(commits, nr_commits,\n&(parent->item->object.oid),\n                                        &int_id))\n                                 int_id = GRAPH_PARENT_MISSING;\n                         else if (!parent->next)\n                                 int_id |= GRAPH_LAST_EDGE;\n\n                         sha1write_be32(f, int_id);\n\n> This is a tangent, but after having seen many instances of \"int_id\",\n> I started to feel that it is grossly misnamed. We do not care about\n> its \"int\" ness---what's more significant about it is that we use can\n> it as a short identifier in place for a full object name, given the\n> table of known OIDs. \"oid_table_index\" may be a better name (but\n> others may be able to suggest even better one).\n>\n> \tint pos;\n> \tpos = commit_pos(commits, nr_commits, parent->item->object.oid);\n> \toid_table_pos = (pos < 0) ? GRAPH_PARENT_MISSING : pos;\n> \tif (!parent->net)\n> \t\toid_table_pos |= GRAPH_LAST_EDGE;\n> \toid_table_pos = htonl(oid_table_pos);\n> \tsha1write(f, &oid_table_pos, sizeof(oid_table_pos));\n>\n> or something like that, perhaps?\n\nYou're right that int_id isn't great, and your more-specific \n\"oid_table_pos\" shows an extra reason why it isn't great: when we add \nthe GRAPH_LAST_EDGE bit or set it to GRAPH_PARENT_MISSING, the value is \nNOT a table position.\n\nI'll rework references of \"int_id\" into \"edge_value\" to store a value \nthat goes into - or is read from - the file, either in the two parent \ncolumns or the octopus table.\n\n>\n>> +static int commit_compare(const void *_a, const void *_b)\n>> +{\n>> +\tstruct object_id *a = (struct object_id *)_a;\n>> +\tstruct object_id *b = (struct object_id *)_b;\n>> +\treturn oidcmp(a, b);\n>> +}\n> I think oidcmp() takes const pointers, so there is no need to\n> discard constness from the parameter like this code does. Also I\n> think we tend to prefer writing a_/b_ (instead of _a/_b) to appease\n> language lawyers who do not want us mere mortals to use names that\n> begin with underscore.\n>\n>> +static int if_packed_commit_add_to_list(const struct object_id *oid,\n>> +\t\t\t\t\tstruct packed_git *pack,\n>> +\t\t\t\t\tuint32_t pos,\n>> +\t\t\t\t\tvoid *data)\n> That is a strange name. \"collect packed commits\", perhaps?\n\nWe are walking all objects in the pack-index and calling this method. If \nthe object is a commit, we add it to the list; otherwise do nothing. \n\"data\" points to the list.\n\nI think the current name makes the following call very clear:\n\n   for_each_object_in_pack(p, if_packed_commit_add_to_list, &oids);\n\ni.e. \"for each object in the pack p: if it is a commit, then add it to \nthe list of oids\".\n\n>\n>> +char *write_commit_graph(const char *obj_dir)\n>> +{\n>> +\tstruct packed_oid_list oids;\n>> +\tstruct packed_commit_list commits;\n>> +\tstruct sha1file *f;\n>> +\tint i, count_distinct = 0;\n>> +\tDIR *info_dir;\n>> +\tstruct strbuf tmp_file = STRBUF_INIT;\n>> +\tstruct strbuf graph_file = STRBUF_INIT;\n>> +\tunsigned char final_hash[GIT_MAX_RAWSZ];\n>> +\tchar *graph_name;\n>> +\tint fd;\n>> +\tuint32_t chunk_ids[5];\n>> +\tuint64_t chunk_offsets[5];\n>> +\tint num_chunks;\n>> +\tint num_long_edges;\n>> +\tstruct commit_list *parent;\n>> +\n>> +\toids.nr = 0;\n>> +\toids.alloc = (int)(0.15 * approximate_object_count());\n> Heh, traditionalist would probably avoid unnecessary use of float\n> and use something like 1/4 or 1/8 ;-) After all, it is merely a\n> ballpark guestimate.\n>\n>> +\tnum_long_edges = 0;\n> This again is about naming, but I find it a bit unnatural to call\n> the edge between a chind and its octopus parents \"long\". Individual\n> edges are not long--the only thing that is long is your \"list of\n> edges\". Some other codepaths in this patch seems to call the same\n> concept with s/long/large/, which I found somewhat puzzling.\n\nHow about \"num_extra_edges\" which counts the \"overflow\" into the octopus \ntable. Note: \"num_octopus_edges\" sounds like summing the out-degree of \noctopus merges, but this count is really \"(total number of parents of \noctopus merges) - (number of octopus merges)\".\n\n>\n>> +\tfor (i = 0; i < oids.nr; i++) {\n>> +\t\tint num_parents = 0;\n>> +\t\tif (i > 0 && !oidcmp(&oids.list[i-1], &oids.list[i]))\n>> +\t\t\tcontinue;\n>> +\n>> +\t\tcommits.list[commits.nr] = lookup_commit(&oids.list[i]);\n>> +\t\tparse_commit(commits.list[commits.nr]);\n>> +\n>> +\t\tfor (parent = commits.list[commits.nr]->parents;\n>> +\t\t parent; parent = parent->next)\n>> +\t\t\tnum_parents++;\n>> +\n>> +\t\tif (num_parents > 2)\n>> +\t\t\tnum_long_edges += num_parents - 1;\n> OK, so we count how many entries we will record in the overflow\n> parent table, and...\n>\n>> +\n>> +\t\tcommits.nr++;\n>> +\t}\n>> +\tnum_chunks = num_long_edges ? 4 : 3;\n> ... if we do not have any octopus commit, we do not need the chunk\n> for the overflow parent table. Makes sense.\n>\n>> +\tstrbuf_addf(&tmp_file, \"%s/info\", obj_dir);\n>> +\tinfo_dir = opendir(tmp_file.buf);\n>> +\n>> +\tif (!info_dir && mkdir(tmp_file.buf, 0777) < 0)\n>> +\t\tdie_errno(_(\"cannot mkdir %s\"), tmp_file.buf);\n>> +\tif (info_dir)\n>> +\t\tclosedir(info_dir);\n>> +\tstrbuf_addstr(&tmp_file, \"/tmp_graph_XXXXXX\");\n>> +\n>> +\tfd = git_mkstemp_mode(tmp_file.buf, 0444);\n>> +\tif (fd < 0)\n>> +\t\tdie_errno(\"unable to create '%s'\", tmp_file.buf);\n> It is not performance critical, but it feels a bit wasteful to\n> opendir merely to see if something exists as a directory, and it is\n> misleading to the readers (it looks as if we care about what files\n> we already have in the directory).\n>\n> The approach that optimizes for the most common case would be to\n>\n> \t- prepare full path to the tempfile first\n> \t- try create with mkstemp\n> \t - if successful, you do not have to worry about creating\n> \t the directory at all, which is the most common case\n> - see why mkstemp step above failed. Was it because you\n> \t did not have the surrounding directory?\n> - if not, there is no point continuing. Just error out.\n> \t - if it was due to missing directory, try creating one.\n> \t- try create with mkstemp\n> \t - if successful, all is well.\n> - otherwise there isn't anything more we can do here.\n\nIt looks like sha1_file.c:create_tmpfile() has code I can use as a model \nhere. Thanks.\n\nI wonder: should we move that method into wrapper.c and have its \nexternal definition be available in cache.h? Then I can just consume it \nfrom here.\n\n>\n>> +\n>> +\tf = sha1fd(fd, tmp_file.buf);\n>> +\n>> +\tsha1write_be32(f, GRAPH_SIGNATURE);\n>> +\n>> +\tsha1write_u8(f, GRAPH_VERSION);\n>> +\tsha1write_u8(f, GRAPH_OID_VERSION);\n>> +\tsha1write_u8(f, num_chunks);\n>> +\tsha1write_u8(f, 0); /* unused padding byte */\n>> +\n>> +\tchunk_ids[0] = GRAPH_CHUNKID_OIDFANOUT;\n>> +\tchunk_ids[1] = GRAPH_CHUNKID_OIDLOOKUP;\n>> +\tchunk_ids[2] = GRAPH_CHUNKID_DATA;\n>> +\tif (num_long_edges)\n>> +\t\tchunk_ids[3] = GRAPH_CHUNKID_LARGEEDGES;\n>> +\telse\n>> +\t\tchunk_ids[3] = 0;\n>> +\tchunk_ids[4] = 0;\n>> +\n>> +\tchunk_offsets[0] = 8 + GRAPH_CHUNKLOOKUP_SIZE;\n>> +\tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n>> +\tchunk_offsets[2] = chunk_offsets[1] + GRAPH_OID_LEN * commits.nr;\n>> +\tchunk_offsets[3] = chunk_offsets[2] + (GRAPH_OID_LEN + 16) * commits.nr;\n>> +\tchunk_offsets[4] = chunk_offsets[3] + 4 * num_long_edges;\n> Do we have to care about overflowing any of the above? For example,\n> the format allows only up to (1<<31)-1 commits, but did something\n> actually check if commits.nr at this point stayed under that limit?\n\nThanks for pointing this out. It should be a while before we have a repo \nwith 2 billion commits, but there's no time like the present to be safe.\n\nThanks,\n-Stolee\n"},{"id":"340046","messageId":"deac2f34-d3c3-b17d-e7f0-b1c044c0d3ff@gmail.com","threadId":"47724","inReplyTo":"CAGZ79kai=KegG-XPmtZJRhTcD-M5Edsw8eocje41mdbL-RARAA@mail.gmail.com","subject":"Re: [PATCH v4 08/13] commit-graph: implement --delete-expired","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-23T17:43:23Z","receivedAt":"2018-02-23T17:43:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/21/2018 4:34 PM, Stefan Beller wrote:\n> On Mon, Feb 19, 2018 at 10:53 AM, Derrick Stolee wrote:\n>\n>> graph_name = write_commit_graph(opts.obj_dir);\n>>\n>> if (graph_name) {\n>> if (opts.set_latest)\n>> set_latest_file(opts.obj_dir, graph_name);\n>>\n>> + if (opts.delete_expired)\n>> + do_delete_expired(opts.obj_dir,\n>> + old_graph_name,\n>> + graph_name);\n>> +\n> So this only allows to delete expired things and setting the latest\n> when writing a new graph. Would we ever envision a user to produce\n> a new graph (e.g. via obtaining a graph that they got from a server) and\n> then manually rerouting the latest to that new graph file without writing\n> that graph file in the same process? The same for expired.\n>\n> I guess these operations are just available via editing the\n> latest or deleting files manually, which slightly contradicts\n> e.g. \"git update-ref\", which in olden times was just a fancy way\n> of rewriting the refs file manually. (though it claims to be less\n> prone to errors as it takes lock files)\n\nI imagine these alternatives for placing a new, latest commit graph file \nwould want Git to handle rewriting the \"graph-latest\" file. Given such a \nuse case, we could consider extending the 'commit-graph' interface, but \nI don't want to plan for it now.\n\n>\n>> extern char *get_graph_latest_filename(const char *obj_dir);\n>> +extern char *get_graph_latest_contents(const char *obj_dir);\n> Did\n> https://public-inbox.org/git/20180208213806.GA6381@sigill.intra.peff.net/\n> ever make it into tree? (It is sort of new, but I feel we'd want to\n> strive for consistency in the code base, eventually.)\n\nThank you for the reminder. I've removed the externs from 'commit-graph.h'.\n\nShould I also remove the externs from other methods I introduce even \nthough their surrounding definitions include 'extern'?\n\nThanks,\n-Stolee\n"},{"id":"340047","messageId":"e7ed962e-aa39-a8e7-2b24-a86dcd30dfa9@gmail.com","threadId":"47724","inReplyTo":"xmqqk1v4x4dt.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 07/13] commit-graph: implement --set-latest","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-23T17:53:32Z","receivedAt":"2018-02-23T17:53:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/22/2018 1:31 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> static struct opts_commit_graph {\n>> \tconst char *obj_dir;\n>> \tconst char *graph_file;\n>> +\tint set_latest;\n>> } opts;\n>> ...\n>> @@ -89,6 +106,8 @@ static int graph_write(int argc, const char **argv)\n>> \t\t{ OPTION_STRING, 'o', \"object-dir\", &opts.obj_dir,\n>> \t\t\tN_(\"dir\"),\n>> \t\t\tN_(\"The object directory to store the graph\") },\n>> +\t\tOPT_BOOL('u', \"set-latest\", &opts.set_latest,\n>> +\t\t\tN_(\"update graph-head to written graph file\")),\n>> \t\tOPT_END(),\n>> \t};\n>> \n>> @@ -102,6 +121,9 @@ static int graph_write(int argc, const char **argv)\n>> \tgraph_name = write_commit_graph(opts.obj_dir);\n>> \n>> \tif (graph_name) {\n>> +\t\tif (opts.set_latest)\n>> +\t\t\tset_latest_file(opts.obj_dir, graph_name);\n>> +\n> This feels like a very strange API from potential caller's point of\n> view. Because you have to decide that you are going to mark it as\n> the latest one upfront before actually writing the graph file, if\n> you forget to pass --set-latest, you have to know how to manually\n> mark the file as latest anyway. I would understand if it were one\n> of the following:\n>\n> (1) whenever a new commit graph file is written in the\n> objects/info/ directory, always mark it as the latest (drop\n> --set-latest option altogether); or\n>\n> (2) make set-latest command that takes a name of an existing graph\n> file in the objects/info/ directory, and sets the latest\n> pointer to point at it (make it separate from 'write' command).\n>\n> though.\n\nPerhaps the 'write' subcommand should be replaced with 'replace' which \ndoes the following:\n\n1. Write a new commit graph based on the starting commits (from all \npacks, from specified packs, from OIDs).\n2. Update 'graph-latest' to point to that new file.\n3. Delete all \"expired\" commit graph files.\n\n(Hence, we would drop the \"--set-latest\" and \"--delete-expired\" options.)\n\nDue to the concerns with concurrency, I really don't want to split these \noperations into independent processes that consumers need to call in \nseries. Since this sequence of events is the only real interaction we \nexpect (for now), this interface will simplify the design.\n\nThe biggest reason I didn't design it like this in the first place is \nthat the behavior changes as the patch develops.\n\nThanks,\n-Stolee\n"},{"id":"340048","messageId":"67871288-ea70-69d4-9c3d-5cc946d8dbe2@gmail.com","threadId":"47724","inReplyTo":"xmqqfu5sx3md.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 08/13] commit-graph: implement --delete-expired","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-23T17:59:00Z","receivedAt":"2018-02-23T17:59:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/22/2018 1:48 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> Teach git-commit-graph to delete the .graph files that are siblings of a\n>> newly-written graph file, except for the file referenced by 'graph-latest'\n>> at the beginning of the process and the newly-written file. If we fail to\n>> delete a graph file, only report a warning because another git process may\n>> be using that file. In a multi-process environment, we expect the previoius\n>> graph file to be used by a concurrent process, so we do not delete it to\n>> avoid race conditions.\n> I do not understand the later part of the above. On some operating\n> systems, you actually can remove a file that is open by another\n> process without any ill effect. There are systems that do not allow\n> removing a file that is in use, and an attempt to unlink it may\n> fail. The need to handle such a failure gracefully is not limited\n> to the case of removing a commit graph file---we need to deal with\n> it when removing file of _any_ type.\n\nMy thought is that we should _warn_ when we fail to delete a .graph file \nthat we think should be safe to delete. However, if we are warning for a \nfile that is currently being accessed (as is the case on Windows, at \nleast), then we will add a lot of noise. This is especially true when \nusing IDEs that run 'status' or 'fetch' in the background, frequently.\n\n> Especially the last sentence \"we do not delete it to avoid race\n> conditions\" I find problematic. If a system does not allow removing\n> a file in use and we detect a failure after an attempt to do so, it\n> is not \"we do not delete it\" --- even if you do, you won't succeed\n> anyway, so there is no point saying that. And on systems that do\n> allow safe removal of a file in use (i.e. they allow an open file to\n> be used by processes that have open filehandles to it after its\n> removal), there is no point refraining to delete it \"to avoid race\n> conditions\", either---in fact it is unlikely that you would even know\n> somebody else had it open and was using it.\n\nThe (unlikely, but possible) race condition involves two processes (P1 \nand P2):\n\n1. P1 reads from graph-latest to see commit graph file F1.\n2. P2 updates graph-latest to point to F2 and deletes F1.\n3. P1 tries to read F1 and fails.\n\nI could explicitly mention this condition in the message, or we can just \nlet P2 fail by deleting all files other than the one referenced by \n'graph-latest'. Thoughts?\n\n> In any case, I do not think '--delete-expired' option that can be\n> given only when you are writing makes much sense as an API. An\n> 'expire' command, just like 'set-latest' command, that is a separate\n> command from 'write', may make sense, though.\n\nIn another message, I proposed dropping the argument and assuming \nexpires happen on every write.\n\nThanks,\n-Stolee\n"},{"id":"340054","messageId":"3609f373-28e9-e849-7126-cd88b54013b9@gmail.com","threadId":"47724","inReplyTo":"CAGZ79kZbdvehOQMZj3NkkSt5RoArtv0ixb8pHvFDrHTrWmVuDg@mail.gmail.com","subject":"Re: [PATCH v4 12/13] commit-graph: read only from specific pack-indexes","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-23T19:19:23Z","receivedAt":"2018-02-23T19:19:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/21/2018 5:25 PM, Stefan Beller wrote:\n> On Mon, Feb 19, 2018 at 10:53 AM, Derrick Stolee wrote:\n>> Teach git-commit-graph to inspect the objects only in a certain list\n>> of pack-indexes within the given pack directory. This allows updating\n>> the commit graph iteratively, since we add all commits stored in a\n>> previous commit graph.\n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> Documentation/git-commit-graph.txt | 11 +++++++++++\n>> builtin/commit-graph.c | 32 +++++++++++++++++++++++++++++---\n>> commit-graph.c | 26 ++++++++++++++++++++++++--\n>> commit-graph.h | 4 +++-\n>> packfile.c | 4 ++--\n>> packfile.h | 2 ++\n>> t/t5318-commit-graph.sh | 16 ++++++++++++++++\n>> 7 files changed, 87 insertions(+), 8 deletions(-)\n>>\n>> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n>> index b9b4031..93d50d1 100644\n>> --- a/Documentation/git-commit-graph.txt\n>> +++ b/Documentation/git-commit-graph.txt\n>> @@ -42,6 +42,10 @@ With the `--delete-expired` option, delete the graph files in the pack\n>> directory that are not referred to by the graph-latest file. To avoid race\n>> conditions, do not delete the file previously referred to by the\n>> graph-latest file if it is updated by the `--set-latest` option.\n>> ++\n>> +With the `--stdin-packs` option, generate the new commit graph by\n>> +walking objects only in the specified packfiles and any commits in\n>> +the existing graph-head.\n> A general question on this series:\n> How do commit graph buildups deal with garbage collected commits?\n> (my personal workflow is heavy on rebase, which generates lots of\n> dangling commits, to be thrown out later)\n>\n> The second half of the sentence makes it sound like once a\n> commit is in the graph it cannot be pulled out easily again, hence\n> the question on the impact of graphs on a long living repository\n> which is garbage collected frequently.\n\nThis is another place that I failed to update when I stopped \nautomatically including commits from the existing graph. As of v4, the \nnew graph should only contain commits reachable from the commits \ndiscovered by the three mechanisms (inspecting all packs, inspecting the \n--stdin-packs, or reading the OIDs with --stdin-commits). Thus, commits \nthat are GC'd will not appear in the new graph.\n\nIf a commit has been GC'd, then parse_commit_gently() will never be \ncalled since it is called after lookup_object() to create the struct \ncommit. The only case we could have is where we navigate to a parent \nusing the commmit graph but that parent is GC'd (this does not make sense).\n\nIt may be helpful to add an \"--additive\" argument to specify that we \nwant to keep all commits that are already in the graph.\n\n> AFAICT you could just run\n> git commit-graph write --set-latest [--delete-expired]\n> as that actually looks up objects from outside the existing graph files,\n> such that lost objects are ignored?\n>\n>> + const char **lines = NULL;\n>> + int nr_lines = 0;\n>> + int alloc_lines = 0;\n> (nit:)\n> I had the impression that these triplet-variables, that are used in\n> ALLOC_GROW are allo X, X_nr and X_allow, but I might be wrong.\n\n\"git grep ALLOC_GROW\" confirms your impression. Will fix.\n\n>\n>> @@ -170,7 +178,25 @@ static int graph_write(int argc, const char **argv)\n>>\n>> old_graph_name = get_graph_latest_contents(opts.obj_dir);\n>>\n>> - graph_name = write_commit_graph(opts.obj_dir);\n>> + if (opts.stdin_packs) {\n>> + struct strbuf buf = STRBUF_INIT;\n>> + nr_lines = 0;\n>> + alloc_lines = 128;\n> alloc_lines has been initialized before, so why redo it here again?\n> Also what is the rationale for choosing 128 as a good default?\n> I would guess 0 is just as fine, because ALLOC_GROW makes sure\n> that it growth fast in the first couple entries by having an additional\n> offset. (no need to fine tune the starting allocation IMHO)\n\nI was unaware that ALLOC_GROW() handled the alloc == 0 case. Thanks.\n\n>\n>> + ALLOC_ARRAY(lines, alloc_lines);\n>> +\n>> + while (strbuf_getline(&buf, stdin) != EOF) {\n>> + ALLOC_GROW(lines, nr_lines + 1, alloc_lines);\n>> + lines[nr_lines++] = buf.buf;\n>> + strbuf_detach(&buf, NULL);\n> strbuf_detach returns its previous buf.buf, such that you can combine these\n> two lines as\n> lines[nr_lines++] = strbuf_detach(&buf, NULL);\n>\n>\n>> + }\n>> +\n>> + pack_indexes = lines;\n>> + nr_packs = nr_lines;\n> Technically we do not need to strbuf_release(&buf) here, because\n> strbuf_detach is always called, and by knowing its implementation,\n> it is just as good.\n>\n>\n>> @@ -579,7 +581,27 @@ char *write_commit_graph(const char *obj_dir)\n>> oids.alloc = 1024;\n>> ALLOC_ARRAY(oids.list, oids.alloc);\n>>\n>> - for_each_packed_object(if_packed_commit_add_to_list, &oids, 0);\n>> + if (pack_indexes) {\n>> + struct strbuf packname = STRBUF_INIT;\n>> + int dirlen;\n>> + strbuf_addf(&packname, \"%s/pack/\", obj_dir);\n>> + dirlen = packname.len;\n>> + for (i = 0; i < nr_packs; i++) {\n>> + struct packed_git *p;\n>> + strbuf_setlen(&packname, dirlen);\n>> + strbuf_addstr(&packname, pack_indexes[i]);\n>> + p = add_packed_git(packname.buf, packname.len, 1);\n>> + if (!p)\n>> + die(\"error adding pack %s\", packname.buf);\n>> + if (open_pack_index(p))\n>> + die(\"error opening index for %s\", packname.buf);\n>> + for_each_object_in_pack(p, if_packed_commit_add_to_list, &oids);\n>> + close_pack(p);\n>> + }\n> strbuf_release(&packname);\n>\n>> + }\n>> + else\n> (micro style nit)\n>\n> } else\n\n"},{"id":"340055","messageId":"xmqqsh9rtsg0.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"4d453f29-8e5b-2a98-ab11-415f63613be8@gmail.com","subject":"Re: [PATCH v4 04/13] commit-graph: implement write_commit_graph()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-23T19:30:07Z","receivedAt":"2018-02-23T19:30:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> jt/binsearch-with-fanout introduces one when there is a 256-entry\n> fanout table (not the case here).\n>\n> The bsearch() method in search.h (and used in\n> pack-write.c:need_large_offset) does not return the _position_ of a\n> found element.\n>\n> Neither of these suit my needs, but I could just be searching for the\n> wrong strings. Also, I could divert my energies in this area to create\n> a generic search in the style of jt/binsearch-with-fanout.\n\n... me goes and digs ...\n\nWhat I had in mind was the one in sha1-lookup.c, actually. Having\nsaid that, hand-rolling another one is not too huge a deal;\neventually people will notice and consolidate code after the series\nstabilizes anyway ;-)\n\n>>> +\t\t\t\tnum_large_edges++;\n>>> +\t\t\t\tparent = parent->next;\n>>> +\t\t\t} while (parent);\n>> It feels somewhat wasteful to traverse the commit's parents list\n>> only to count, without populating the octopus table (which I\n>> understand is assumed to be minority case under this design).\n>\n> Since we are writing the commit graph file in-order, we cannot write\n> the octopus table until after the chunk lengths are known.\n\nOh, my \"minority case\" comment was me wondering \"since we expect\nthere are only a few, why not keep them in memory while we discover\nthem here, so that the writing phase that come later do not have to\ngo through all commits again counting their parents? would it be\nmore performant and a better trade-off?\" We can certainly do such\nan optimization later (iow, it is not all that crucial issue and\ncertainly I didn't mention the above as something that needs to be\n\"fixed\"--there is nothing broken).\n\n> store the octopus table in memory and then dump it into the file\n> later, but walking the parents is quite fast after all the commits are\n> loaded. I'm not sure the time optimization merits the extra complexity\n> here. (I'm happy to revisit this if we do see this performance\n> lacking.)\n>\n> P.S. I really like the name \"octopus table\" and will use that for\n> informal discussions of this format.\n\nI actually do not mind that name used as the official term. I find\nit far more descriptive and understandable than \"long edge\" / \"large\nedge\" at least to a Git person.\n\n> You're right that int_id isn't great, and your more-specific\n> \"oid_table_pos\" shows an extra reason why it isn't great: when we add\n> the GRAPH_LAST_EDGE bit or set it to GRAPH_PARENT_MISSING, the value\n> is NOT a table position.\n\nPerhaps I am somewhat biased, but it is quite natural for our\ncodebase and internal API to say something like this:\n\n x_pos(table, key) function's return value is the non-negative\n position for the key in the table when the key is there; when it\n returns a negative value, it is (-1 - position) where the \"position\"\n is the position in the table they key would have been found if\n it was in there.\n\nand store the return value from such a function in a variable called\n\"pos\". Surely, sometimes \"pos\" does not have _the_ position, but\nthat does not make it a bad name.\n\nSaying \"MISSING is a special value that denotes 'nothing is here'\"\nand allowing it to be set to a variable that meant to hold the\nposition is not such a bad thing, though. After all, that is how\nyou use NULL as a special value for a pointer variable ;-).\n\nSame for using the high bit to mean something else. Taking these\ntogether you would explain \"low 31-bit of pos holds the position for\nthe item in the table. MISSING is a special value that you can use\nto denote there is nothing. The LAST_EDGE bit denotes that one\ngroup of positions ends there\", or something like that.\n\n> I think the current name makes the following call very clear:\n\nIt is still a strange name nevertheless.\n\n>>> +char *write_commit_graph(const char *obj_dir)\n>>> +{\n>>> +\tstruct packed_oid_list oids;\n>>> +\tstruct packed_commit_list commits;\n>>> +\tstruct sha1file *f;\n>>> +\tint i, count_distinct = 0;\n>>> +\tDIR *info_dir;\n>>> +\tstruct strbuf tmp_file = STRBUF_INIT;\n>>> +\tstruct strbuf graph_file = STRBUF_INIT;\n>>> +\tunsigned char final_hash[GIT_MAX_RAWSZ];\n>>> +\tchar *graph_name;\n>>> +\tint fd;\n>>> +\tuint32_t chunk_ids[5];\n>>> +\tuint64_t chunk_offsets[5];\n>>> +\tint num_chunks;\n>>> +\tint num_long_edges;\n>>> +\tstruct commit_list *parent;\n>>> +\n>>> +\toids.nr = 0;\n>>> +\toids.alloc = (int)(0.15 * approximate_object_count());\n>> Heh, traditionalist would probably avoid unnecessary use of float\n>> and use something like 1/4 or 1/8 ;-) After all, it is merely a\n>> ballpark guestimate.\n>>\n>>> +\tnum_long_edges = 0;\n>> This again is about naming, but I find it a bit unnatural to call\n>> the edge between a chind and its octopus parents \"long\". Individual\n>> edges are not long--the only thing that is long is your \"list of\n>> edges\". Some other codepaths in this patch seems to call the same\n>> concept with s/long/large/, which I found somewhat puzzling.\n>\n> How about \"num_extra_edges\"...\n\nYes, \"extra\" in the name makes it very understandable.\n\n"},{"id":"340056","messageId":"xmqqo9kftsa5.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"67871288-ea70-69d4-9c3d-5cc946d8dbe2@gmail.com","subject":"Re: [PATCH v4 08/13] commit-graph: implement --delete-expired","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-23T19:33:38Z","receivedAt":"2018-02-23T19:33:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> The (unlikely, but possible) race condition involves two processes (P1\n> and P2):\n>\n> 1. P1 reads from graph-latest to see commit graph file F1.\n> 2. P2 updates graph-latest to point to F2 and deletes F1.\n> 3. P1 tries to read F1 and fails.\n>\n> I could explicitly mention this condition in the message, or we can\n> just let P2 fail by deleting all files other than the one referenced\n> by 'graph-latest'. Thoughts?\n\nThe established way we do this is not to have -latest pointer, I\nwould think, and instead, make -latest be the actual thing. That is\nhow $GIT_DIR/index is updated, for example, by first writing into a\ntemporary file and then moving it to the final destination. Is\nthere a reason why the same pattern cannot be used?\n"},{"id":"340058","messageId":"8c958ada-b96b-cdc7-594c-c6f6cdb02b70@gmail.com","threadId":"47724","inReplyTo":"xmqqo9kftsa5.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 08/13] commit-graph: implement --delete-expired","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-23T19:41:01Z","receivedAt":"2018-02-23T19:41:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/23/2018 2:33 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> The (unlikely, but possible) race condition involves two processes (P1\n>> and P2):\n>>\n>> 1. P1 reads from graph-latest to see commit graph file F1.\n>> 2. P2 updates graph-latest to point to F2 and deletes F1.\n>> 3. P1 tries to read F1 and fails.\n>>\n>> I could explicitly mention this condition in the message, or we can\n>> just let P2 fail by deleting all files other than the one referenced\n>> by 'graph-latest'. Thoughts?\n> The established way we do this is not to have -latest pointer, I\n> would think, and instead, make -latest be the actual thing. That is\n> how $GIT_DIR/index is updated, for example, by first writing into a\n> temporary file and then moving it to the final destination. Is\n> there a reason why the same pattern cannot be used?\n\nMy thought was that using a fixed name (e.g. \n.git/objects/info/commit-graph) would block making the graph \nincremental. Upon thinking again, this is not the case. That feature \ncould be designed with a fixed name for the small, frequently-updated \nfile and use .../info/graph-.graph names for the \"base\" graph files.\n\nI'll spend some time thinking about the ramifications of this fixed-name \nconcept. At the moment, it would remove two commits from this patch \nseries, which is nice.\n\nThanks,\n-Stolee\n"},{"id":"340060","messageId":"xmqqk1v3trm0.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"xmqqsh9rtsg0.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 04/13] commit-graph: implement write_commit_graph()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-23T19:48:07Z","receivedAt":"2018-02-23T19:48:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> I think the current name makes the following call very clear:\n>\n> It is still a strange name nevertheless.\n\nSorry for simply repeating \"strange\" without spelling out why in the\nprevious message. This certainly is subjective and depends on your\ncultural background, but in our codebase, I tried to name functions\nafter \"what\" they do and \"why\", rather than \"how\" they do so. In a\nway, it's the same kind of uneasiness I feel when I see variables\nnamed in hungarian notation.\n\nYou would inspect the object and treat 'data' as a list and add to\nthe object if it is a commit, and if_packed_commit_add_to_list()\ncertainly is a name that describes all of that well, but does it\ngive readers a good answer when they wonder why the function is\ndoing so? You described with the name of the function how it\ncollects commits that are in the pack, without explicitly saying\nthat you want to collect packed commits and that is why you are\ninspecting for type and doing so only for commit (i.e.\n\"if_packed_commit\" part of the name) and why you are adding to a\nlist.\n"},{"id":"340062","messageId":"xmqqbmgftrfl.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"8c958ada-b96b-cdc7-594c-c6f6cdb02b70@gmail.com","subject":"Re: [PATCH v4 08/13] commit-graph: implement --delete-expired","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-23T19:51:58Z","receivedAt":"2018-02-23T19:52:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> My thought was that using a fixed name\n> (e.g. .git/objects/info/commit-graph) would block making the graph\n> incremental. Upon thinking again, this is not the case. That feature\n> could be designed with a fixed name for the small, frequently-updated\n> file and use .../info/graph-.graph names for the \"base\" graph\n> files.\n\nI guess that is in line with the way how split-index folks did their\nthing ;-)\n"},{"id":"340065","messageId":"8d29382b-e0a7-968f-b231-f27f89f275d3@gmail.com","threadId":"47724","inReplyTo":"xmqqsh9rtsg0.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 04/13] commit-graph: implement write_commit_graph()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-23T20:02:21Z","receivedAt":"2018-02-23T20:02:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"\n\nOn 2/23/2018 2:30 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n>\n>> jt/binsearch-with-fanout introduces one when there is a 256-entry\n>> fanout table (not the case here).\n>>\n>> The bsearch() method in search.h (and used in\n>> pack-write.c:need_large_offset) does not return the _position_ of a\n>> found element.\n>>\n>> Neither of these suit my needs, but I could just be searching for the\n>> wrong strings. Also, I could divert my energies in this area to create\n>> a generic search in the style of jt/binsearch-with-fanout.\n> ... me goes and digs ...\n>\n> What I had in mind was the one in sha1-lookup.c, actually. Having\n> said that, hand-rolling another one is not too huge a deal;\n> eventually people will notice and consolidate code after the series\n> stabilizes anyway ;-)\n\nAh, sha1_pos(). That definitely satisfies my use case. Thanks! My local \nbranch has this replacement.\n\n>\n>>>> +\t\t\t\tnum_large_edges++;\n>>>> +\t\t\t\tparent = parent->next;\n>>>> +\t\t\t} while (parent);\n>>> It feels somewhat wasteful to traverse the commit's parents list\n>>> only to count, without populating the octopus table (which I\n>>> understand is assumed to be minority case under this design).\n>> Since we are writing the commit graph file in-order, we cannot write\n>> the octopus table until after the chunk lengths are known.\n> Oh, my \"minority case\" comment was me wondering \"since we expect\n> there are only a few, why not keep them in memory while we discover\n> them here, so that the writing phase that come later do not have to\n> go through all commits again counting their parents? would it be\n> more performant and a better trade-off?\" We can certainly do such\n> an optimization later (iow, it is not all that crucial issue and\n> certainly I didn't mention the above as something that needs to be\n> \"fixed\"--there is nothing broken).\n>\n>> store the octopus table in memory and then dump it into the file\n>> later, but walking the parents is quite fast after all the commits are\n>> loaded. I'm not sure the time optimization merits the extra complexity\n>> here. (I'm happy to revisit this if we do see this performance\n>> lacking.)\n>>\n>> P.S. I really like the name \"octopus table\" and will use that for\n>> informal discussions of this format.\n> I actually do not mind that name used as the official term. I find\n> it far more descriptive and understandable than \"long edge\" / \"large\n> edge\" at least to a Git person.\n\nI will consider using this in the format and design documents.\n\n>\n>> You're right that int_id isn't great, and your more-specific\n>> \"oid_table_pos\" shows an extra reason why it isn't great: when we add\n>> the GRAPH_LAST_EDGE bit or set it to GRAPH_PARENT_MISSING, the value\n>> is NOT a table position.\n> Perhaps I am somewhat biased, but it is quite natural for our\n> codebase and internal API to say something like this:\n>\n> x_pos(table, key) function's return value is the non-negative\n> position for the key in the table when the key is there; when it\n> returns a negative value, it is (-1 - position) where the \"position\"\n> is the position in the table they key would have been found if\n> it was in there.\n>\n> and store the return value from such a function in a variable called\n> \"pos\". Surely, sometimes \"pos\" does not have _the_ position, but\n> that does not make it a bad name.\n>\n> Saying \"MISSING is a special value that denotes 'nothing is here'\"\n> and allowing it to be set to a variable that meant to hold the\n> position is not such a bad thing, though. After all, that is how\n> you use NULL as a special value for a pointer variable ;-).\n>\n> Same for using the high bit to mean something else. Taking these\n> together you would explain \"low 31-bit of pos holds the position for\n> the item in the table. MISSING is a special value that you can use\n> to denote there is nothing. The LAST_EDGE bit denotes that one\n> group of positions ends there\", or something like that.\n>\n>> I think the current name makes the following call very clear:\n> It is still a strange name nevertheless.\n>\n>>>> +char *write_commit_graph(const char *obj_dir)\n>>>> +{\n>>>> +\tstruct packed_oid_list oids;\n>>>> +\tstruct packed_commit_list commits;\n>>>> +\tstruct sha1file *f;\n>>>> +\tint i, count_distinct = 0;\n>>>> +\tDIR *info_dir;\n>>>> +\tstruct strbuf tmp_file = STRBUF_INIT;\n>>>> +\tstruct strbuf graph_file = STRBUF_INIT;\n>>>> +\tunsigned char final_hash[GIT_MAX_RAWSZ];\n>>>> +\tchar *graph_name;\n>>>> +\tint fd;\n>>>> +\tuint32_t chunk_ids[5];\n>>>> +\tuint64_t chunk_offsets[5];\n>>>> +\tint num_chunks;\n>>>> +\tint num_long_edges;\n>>>> +\tstruct commit_list *parent;\n>>>> +\n>>>> +\toids.nr = 0;\n>>>> +\toids.alloc = (int)(0.15 * approximate_object_count());\n>>> Heh, traditionalist would probably avoid unnecessary use of float\n>>> and use something like 1/4 or 1/8 ;-) After all, it is merely a\n>>> ballpark guestimate.\n>>>\n>>>> +\tnum_long_edges = 0;\n>>> This again is about naming, but I find it a bit unnatural to call\n>>> the edge between a chind and its octopus parents \"long\". Individual\n>>> edges are not long--the only thing that is long is your \"list of\n>>> edges\". Some other codepaths in this patch seems to call the same\n>>> concept with s/long/large/, which I found somewhat puzzling.\n>> How about \"num_extra_edges\"...\n> Yes, \"extra\" in the name makes it very understandable.\n>\n\n"},{"id":"340307","messageId":"CAM0VKjk2dphJsMfuG2F4-G0xW=YfmVsDsPpxm_XoXTCgQHA7=g@mail.gmail.com","threadId":"47724","inReplyTo":"1519066406-81663-5-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v4 04/13] commit-graph: implement write_commit_graph()","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-02-26T16:10:18Z","receivedAt":"2018-02-26T16:10:25Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Feb 19, 2018 at 7:53 PM, Derrick Stolee wrote:\n\n> +static int if_packed_commit_add_to_list(const struct object_id *oid,\n> + struct packed_git *pack,\n> + uint32_t pos,\n> + void *data)\n> +{\n> + struct packed_oid_list *list = (struct packed_oid_list*)data;\n> + enum object_type type;\n> + unsigned long size;\n> + void *inner_data;\n> + off_t offset = nth_packed_object_offset(pack, pos);\n> + inner_data = unpack_entry(pack, offset, &type, &size);\n> +\n> + if (inner_data)\n> + free(inner_data);\n\nThe condition is unnecessary, free() can handle a NULL argument just fine.\n\n(Suggested by Coccinelle and 'contrib/coccinelle/free.cocci.patch'.)\n"},{"id":"340308","messageId":"20180226162539.7463-1-szeder.dev@gmail.com","threadId":"47724","inReplyTo":"1519066406-81663-4-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v4 03/13] commit-graph: create git-commit-graph builtin","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-02-26T16:25:39Z","receivedAt":"2018-02-26T16:26:58Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"> Teach git the 'commit-graph' builtin that will be used for writing and\n> reading packed graph files. The current implementation is mostly\n> empty, except for an '--object-dir' option.\n\nSince 'git commit-graph' is a builtin command, it shouldn't show up in\ncompletion when doing 'git co'.\nPlease squash in the patch below to make it so.\n\nFurthermore, please have a look at\n \n https://public-inbox.org/git/20180202160132.31550-1-szeder.dev@gmail.com/\n\nfor an other oneliner change.\n\n\ndiff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash\nindex 17929b0809..fafed13c06 100644\n--- a/contrib/completion/git-completion.bash\n+++ b/contrib/completion/git-completion.bash\n@@ -841,6 +841,7 @@ __git_list_porcelain_commands ()\n \t\tcheck-ref-format) : plumbing;;\n \t\tcheckout-index) : plumbing;;\n \t\tcolumn) : internal helper;;\n+\t\tcommit-graph) : plumbing;;\n \t\tcommit-tree) : plumbing;;\n \t\tcount-objects) : infrequent;;\n \t\tcredential) : credentials;;\n\n"},{"id":"340309","messageId":"755dead9-17fc-4735-99d2-d3f2dcb102ae@gmail.com","threadId":"47724","inReplyTo":"20180226162539.7463-1-szeder.dev@gmail.com","subject":"Re: [PATCH v4 03/13] commit-graph: create git-commit-graph builtin","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-02-26T17:08:19Z","receivedAt":"2018-02-26T17:08:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/26/2018 11:25 AM, SZEDER Gábor wrote:\n>> Teach git the 'commit-graph' builtin that will be used for writing and\n>> reading packed graph files. The current implementation is mostly\n>> empty, except for an '--object-dir' option.\n> Since 'git commit-graph' is a builtin command, it shouldn't show up in\n> completion when doing 'git co'.\n> Please squash in the patch below to make it so.\n>\n> Furthermore, please have a look at\n> \n> https://public-inbox.org/git/20180202160132.31550-1-szeder.dev@gmail.com/\n>\n> for an other oneliner change.\n>\n>\n> diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash\n> index 17929b0809..fafed13c06 100644\n> --- a/contrib/completion/git-completion.bash\n> +++ b/contrib/completion/git-completion.bash\n> @@ -841,6 +841,7 @@ __git_list_porcelain_commands ()\n> \t\tcheck-ref-format) : plumbing;;\n> \t\tcheckout-index) : plumbing;;\n> \t\tcolumn) : internal helper;;\n> +\t\tcommit-graph) : plumbing;;\n> \t\tcommit-tree) : plumbing;;\n> \t\tcount-objects) : infrequent;;\n> \t\tcredential) : credentials;;\n\nThanks for this, and the reminder. I made these changes locally, so they \nwill be in v5.\n\n-Stolee\n"},{"id":"340591","messageId":"xmqqinahkl3b.fsf@gitster-ct.c.googlers.com","threadId":"47724","inReplyTo":"1519066406-81663-5-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v4 04/13] commit-graph: implement write_commit_graph()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-28T18:47:20Z","receivedAt":"2018-02-28T18:47:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> diff --git a/commit-graph.h b/commit-graph.h\n> new file mode 100644\n> index 0000000..dc8c73a\n> --- /dev/null\n> +++ b/commit-graph.h\n> @@ -0,0 +1,7 @@\n> +#ifndef COMMIT_GRAPH_H\n> +#define COMMIT_GRAPH_H\n> +\n> +extern char *write_commit_graph(const char *obj_dir);\n> +\n> +#endif\n> +\n\nTrailing blank line at the end of the file. Remove it.\n\nt5318 has the same issue.\n\nThanks.\n"},{"id":"343404","messageId":"867eptkeeq.fsf@gmail.com","threadId":"47724","inReplyTo":"1519066406-81663-1-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v4 00/13] Serialized Git Commit Graph","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-03-30T11:10:05Z","receivedAt":"2018-03-30T11:10:19Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"I hope that I am addressing the most recent version of this series.\n\nDerrick Stolee writes:\n\n> As promised [1], this patch contains a way to serialize the commit graph.\n> The current implementation defines a new file format to store the graph\n> structure (parent relationships) and basic commit metadata (commit date,\n> root tree OID) in order to prevent parsing raw commits while performing\n> basic graph walks. For example, we do not need to parse the full commit\n> when performing these walks:\n>\n> * 'git log --topo-order -1000' walks all reachable commits to avoid\n> incorrect topological orders, but only needs the commit message for\n> the top 1000 commits.\n>\n> * 'git merge-base ' may walk many commits to find the correct\n> boundary between the commits reachable from A and those reachable\n> from B. No commit messages are needed.\n>\n> * 'git branch -vv' checks ahead/behind status for all local branches\n> compared to their upstream remote branches. This is essentially as\n> hard as computing merge bases for each.\n>\n> The current patch speeds up these calculations by injecting a check in\n> parse_commit_gently() to check if there is a graph file and using that\n> to provide the required metadata to the struct commit.\n\nThat's nice.\n\nWhat are the assumptions about the serialized commit graph format? Does\nit need to be:\n - extensible without rewriting (e.g. append-only)?\n - like the above, but may need rewriting for optimal performance?\n - extending it needs to rewrite whole file?\n\nExcuse me if it waas already asked and answered.\n\n>\n> The file format has room to store generation numbers, which will be\n> provided as a patch after this framework is merged. Generation numbers\n> are referenced by the design document but not implemented in order to\n> make the current patch focus on the graph construction process. Once\n> that is stable, it will be easier to add generation numbers and make\n> graph walks aware of generation numbers one-by-one.\n\nAs the serialized commit graph format is versioned, I wonder if it would\nbe possible to speed up graph walks even more by adding to it FELINE\nindex (pair of numbers) from \"Reachability Queries in Very Large Graphs:\nA Fast Refined Olnine Search Approach\" (2014) - available at\nhttp://openproceedings.org/EDBT/2014/paper_166.pdf\n\nThe implementation would probably need adjustments to make it\nunambiguous and unambiguously extensible; unless there is place for\nindices that are local-only and need to be recalculated from scratch\nwhen graph changes (to cover all graph).\n\n>\n> Here are some performance results for a copy of the Linux repository\n> where 'master' has 704,766 reachable commits and is behind 'origin/master'\n> by 19,610 commits.\n>\n> | Command | Before | After | Rel % |\n> |----------------------------------|--------|--------|-------|\n> | log --oneline --topo-order -1000 | 5.9s | 0.7s | -88% |\n> | branch -vv | 0.42s | 0.27s | -35% |\n> | rev-list --all | 6.4s | 1.0s | -84% |\n> | rev-list --all --objects | 32.6s | 27.6s | -15% |\n\nThat's the \"Rel %\" of \"Before\", that is delta/before, isn't it?\n\n> To test this yourself, run the following on your repo:\n>\n> git config core.commitGraph true\n> git show-ref -s | git commit-graph write --set-latest --stdin-commits\n>\n> The second command writes a commit graph file containing every commit\n> reachable from your refs. Now, all git commands that walk commits will\n> check your graph first before consulting the ODB. You can run your own\n> performance comparisions by toggling the 'core.commitgraph' setting.\n\nGood. It is nicely similar to how bitmap indices (of reachability) are\nhandled.\n\nI just wonder what happens in the (rare) presence of grafts (old\nmechanism), or \"git replace\"-d commits...\n\nRegards,\n-- \nJakub Narębski\n"},{"id":"343417","messageId":"86370hk84x.fsf@gmail.com","threadId":"47724","inReplyTo":"1519066406-81663-2-git-send-email-dstolee@microsoft.com","subject":"Re: [PATCH v4 01/13] commit-graph: add format document","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-03-30T13:25:34Z","receivedAt":"2018-03-30T13:25:44Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> +== graph-*.graph files have the following format:\n\nWhat is this '*' here?\n\n[...]\n> + The remaining data in the body is described one chunk at a time, and\n> + these chunks may be given in any order. Chunks are required unless\n> + otherwise specified.\n\nDoes Git need to understand all chunks, or could there be optional\nchunks that can be safely ignored (like in PNG format)? Though this may\nbe overkill, and could be left for later revision of the format if\ndeemed necessary.\n\n> +\n> +CHUNK DATA:\n> +\n> + OID Fanout (ID: {'O', 'I', 'D', 'F'}) (256 * 4 bytes)\n> + The ith entry, F[i], stores the number of OIDs with first\n> + byte at most i. Thus F[255] stores the total\n> + number of commits (N).\n\nAll right, it is small enough that can be required even for a very small\nnumber of commits.\n\n> +\n> + OID Lookup (ID: {'O', 'I', 'D', 'L'}) (N * H bytes)\n> + The OIDs for all commits in the graph, sorted in ascending order.\n> +\n> + Commit Data (ID: {'C', 'G', 'E', 'T' }) (N * (H + 16) bytes)\n\nDo commits need to be put here in the ascending order of OIDs?\n\nIf so, this would mean that it is not possible to add information about\nnew commits by only appending data and maybe overwriting some fields, I\nthink. You would need to do full rewrite to insert new commit in\nappropriate place.\n\n> + * The first H bytes are for the OID of the root tree.\n> + * The next 8 bytes are for the int-ids of the first two parents\n> + of the ith commit. Stores value 0xffffffff if no parent in that\n> + position. If there are more than two parents, the second value\n> + has its most-significant bit on and the other bits store an array\n> + position into the Large Edge List chunk.\n> + * The next 8 bytes store the generation number of the commit and\n> + the commit time in seconds since EPOCH. The generation number\n> + uses the higher 30 bits of the first 4 bytes, while the commit\n> + time uses the 32 bits of the second 4 bytes, along with the lowest\n> + 2 bits of the lowest byte, storing the 33rd and 34th bit of the\n> + commit time.\n> +\n> + Large Edge List (ID: {'E', 'D', 'G', 'E'}) [Optional]\n> + This list of 4-byte values store the second through nth parents for\n> + all octopus merges. The second parent value in the commit data stores\n> + an array position within this list along with the most-significant bit\n> + on. Starting at that array position, iterate through this list of int-ids\n> + for the parents until reaching a value with the most-significant bit on.\n> + The other bits correspond to the int-id of the last parent.\n\nAll right, that is one chunk that cannot use fixed-length records; this\nshouldn't matter much, as we iterate only up to the number of parents\nless two.\n\nA question: what happens to the last list of parents? Is there a\nguardian value of 0xffffffff at last place?\n\n> +\n> +TRAILER:\n> +\n> +\tH-byte HASH-checksum of all of the above.\n> +\n\nBest,\n--\nJakub Narębski\n"},{"id":"343579","messageId":"f5d8934e-afc2-39d4-b7d5-e5ba5e5021a1@gmail.com","threadId":"47724","inReplyTo":"867eptkeeq.fsf@gmail.com","subject":"Re: [PATCH v4 00/13] Serialized Git Commit Graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-02T13:02:11Z","receivedAt":"2018-04-02T13:02:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/30/2018 7:10 AM, Jakub Narebski wrote:\n> I hope that I am addressing the most recent version of this series.\n\nHi Jakub. Thanks for the interest in this patch series.\n\nThe most-recent version is v6 [1], but I will re-roll to v7 soon (after \nv2.17.0 is marked).\n\n[1] \nhttps://public-inbox.org/git/20180314192736.70602-1-dstolee@microsoft.com/T/#u\n\n> Derrick Stolee writes:\n>\n>> As promised [1], this patch contains a way to serialize the commit graph.\n>> The current implementation defines a new file format to store the graph\n>> structure (parent relationships) and basic commit metadata (commit date,\n>> root tree OID) in order to prevent parsing raw commits while performing\n>> basic graph walks. For example, we do not need to parse the full commit\n>> when performing these walks:\n>>\n>> * 'git log --topo-order -1000' walks all reachable commits to avoid\n>> incorrect topological orders, but only needs the commit message for\n>> the top 1000 commits.\n>>\n>> * 'git merge-base ' may walk many commits to find the correct\n>> boundary between the commits reachable from A and those reachable\n>> from B. No commit messages are needed.\n>>\n>> * 'git branch -vv' checks ahead/behind status for all local branches\n>> compared to their upstream remote branches. This is essentially as\n>> hard as computing merge bases for each.\n>>\n>> The current patch speeds up these calculations by injecting a check in\n>> parse_commit_gently() to check if there is a graph file and using that\n>> to provide the required metadata to the struct commit.\n> That's nice.\n>\n> What are the assumptions about the serialized commit graph format? Does\n> it need to be:\n> - extensible without rewriting (e.g. append-only)?\n> - like the above, but may need rewriting for optimal performance?\n> - extending it needs to rewrite whole file?\n>\n> Excuse me if it waas already asked and answered.\n\nIt is not extensible without rewriting. Reducing write time was not a \nmain goal, since the graph will be written only occasionally during data \nmanagement phases (like 'gc' or 'repack'; this integration is not \nimplemented yet).\n\n>\n>> The file format has room to store generation numbers, which will be\n>> provided as a patch after this framework is merged. Generation numbers\n>> are referenced by the design document but not implemented in order to\n>> make the current patch focus on the graph construction process. Once\n>> that is stable, it will be easier to add generation numbers and make\n>> graph walks aware of generation numbers one-by-one.\n> As the serialized commit graph format is versioned, I wonder if it would\n> be possible to speed up graph walks even more by adding to it FELINE\n> index (pair of numbers) from \"Reachability Queries in Very Large Graphs:\n> A Fast Refined Olnine Search Approach\" (2014) - available at\n> http://openproceedings.org/EDBT/2014/paper_166.pdf\n>\n> The implementation would probably need adjustments to make it\n> unambiguous and unambiguously extensible; unless there is place for\n> indices that are local-only and need to be recalculated from scratch\n> when graph changes (to cover all graph).\n\nThe chunk-based format is intended to allow extra indexes like the one \nyou recommend, without needing to increase the version number. Using an \noptional chunk allows older versions of Git to read the file without \nerror, since the data is \"extra\", and newer versions can take advantage \nof the acceleration.\n\nAt one point, I was investigating these reachability indexes (I read \n\"SCARAB: Scaling Reachability Computation on Large Graphs\" by Jihn, \nRuan, Dey, and Xu [2]) but find the question that these indexes target \nto be lacking for most of the Git uses. That is, they ask the boolean \nquestion \"Can X reach Y?\". More often, Git needs to answer \"What is the \nset of commits reachable from X but not from Y\" or \"Topologically sort \ncommits reachable from X\" or \"How many commits are in each part of the \nsymmetric difference between reachable from X or reachable from Y?\"\n\nThe case for \"Can X reach Y?\" is mostly for commands like 'git branch \n--contains', when 'git fetch' checks for forced-updates of branches, or \nwhen the server decides enough negotiation has occurred during a 'git \nfetch'. While these may be worth investigating, they also benefit \ngreatly from the accelerated graph walk introduced in the current format.\n\nI would be happy to review any effort to extend the commit-graph format \nto include such indexes, as long as the performance benefits outweigh \nthe complexity to create them.\n\n[2] \nhttp://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.719.8396&rep=rep1&type=pdf\n\n>\n>> Here are some performance results for a copy of the Linux repository\n>> where 'master' has 704,766 reachable commits and is behind 'origin/master'\n>> by 19,610 commits.\n>>\n>> | Command | Before | After | Rel % |\n>> |----------------------------------|--------|--------|-------|\n>> | log --oneline --topo-order -1000 | 5.9s | 0.7s | -88% |\n>> | branch -vv | 0.42s | 0.27s | -35% |\n>> | rev-list --all | 6.4s | 1.0s | -84% |\n>> | rev-list --all --objects | 32.6s | 27.6s | -15% |\n> That's the \"Rel %\" of \"Before\", that is delta/before, isn't it?\n\nI do mean the relative change.\n\n>\n>> To test this yourself, run the following on your repo:\n>>\n>> git config core.commitGraph true\n>> git show-ref -s | git commit-graph write --set-latest --stdin-commits\n>>\n>> The second command writes a commit graph file containing every commit\n>> reachable from your refs. Now, all git commands that walk commits will\n>> check your graph first before consulting the ODB. You can run your own\n>> performance comparisions by toggling the 'core.commitgraph' setting.\n> Good. It is nicely similar to how bitmap indices (of reachability) are\n> handled.\n>\n> I just wonder what happens in the (rare) presence of grafts (old\n> mechanism), or \"git replace\"-d commits...\n\nIn the design document, I mention that the current implementation does \nnot work with grafts (it will ignore them). A later patch will refactor \nthe graft code so we can access it from the commit-graph parsing of a \ncommit without copy-pasting the code out of parse_commit_gently().\n\nThe commit-graph is only a compact representation of the object \ndatabase. If a commit is replaced with 'git replace' before 'git \ncommit-graph write' then the commit-graph write will write the replaced \nobject. I haven't tested what happens when a commit-graph is written and \nthen a commit is replaced, but my guess is that the replacement does not \noccur until a full parse is attempted (i.e. when reading author or \ncommit message information). This will lead to unknown results.\n\nThanks for pointing out the interaction with 'git replace'. I have items \nto fix grafts and replaced commits before integrating commit-graph \nwrites into automatic actions like 'gc.auto'.\n\nThanks,\n-Stolee\n"},{"id":"343580","messageId":"433a2523-d04a-08bb-d128-6c8e578916fa@gmail.com","threadId":"47724","inReplyTo":"86370hk84x.fsf@gmail.com","subject":"Re: [PATCH v4 01/13] commit-graph: add format document","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-02T13:09:29Z","receivedAt":"2018-04-02T13:09:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/30/2018 9:25 AM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n>\n>> +== graph-*.graph files have the following format:\n> What is this '*' here?\n\nNo longer necessary. It used to be a placeholder for a hash value, but \nnow the graph is stored in objects/info/commit-graph.\n\n>\n> [...]\n>> + The remaining data in the body is described one chunk at a time, and\n>> + these chunks may be given in any order. Chunks are required unless\n>> + otherwise specified.\n> Does Git need to understand all chunks, or could there be optional\n> chunks that can be safely ignored (like in PNG format)? Though this may\n> be overkill, and could be left for later revision of the format if\n> deemed necessary.\n\nIn v6, the format and design documents are edited to make clear the use \nof optional chunks, specifically for future extension without increasing \nthe version number.\n\n>\n>> +\n>> +CHUNK DATA:\n>> +\n>> + OID Fanout (ID: {'O', 'I', 'D', 'F'}) (256 * 4 bytes)\n>> + The ith entry, F[i], stores the number of OIDs with first\n>> + byte at most i. Thus F[255] stores the total\n>> + number of commits (N).\n> All right, it is small enough that can be required even for a very small\n> number of commits.\n>\n>> +\n>> + OID Lookup (ID: {'O', 'I', 'D', 'L'}) (N * H bytes)\n>> + The OIDs for all commits in the graph, sorted in ascending order.\n>> +\n>> + Commit Data (ID: {'C', 'G', 'E', 'T' }) (N * (H + 16) bytes)\n> Do commits need to be put here in the ascending order of OIDs?\n\nYes.\n\n> If so, this would mean that it is not possible to add information about\n> new commits by only appending data and maybe overwriting some fields, I\n> think. You would need to do full rewrite to insert new commit in\n> appropriate place.\n\nThat is the idea. This file is not updated with every new commit, but \ninstead will be updated on some scheduled cleanup events. The \ncommit-graph file is designed in a way to be non-critical, and not tied \nto the packfile layout. This allows flexibility for when to do the write.\n\nFor example, in GVFS, we will write a new commit-graph when there are \nnew daily prefetch packs.\n\nThis could also integrate with 'gc' and 'repack' so whenever they are \ntriggered the commit-graph is written as well.\n\nCommits that do not exist in the commit-graph file will load from the \nobject database as normal (after a failed lookup in the commit-graph file).\n\n>> + * The first H bytes are for the OID of the root tree.\n>> + * The next 8 bytes are for the int-ids of the first two parents\n>> + of the ith commit. Stores value 0xffffffff if no parent in that\n>> + position. If there are more than two parents, the second value\n>> + has its most-significant bit on and the other bits store an array\n>> + position into the Large Edge List chunk.\n>> + * The next 8 bytes store the generation number of the commit and\n>> + the commit time in seconds since EPOCH. The generation number\n>> + uses the higher 30 bits of the first 4 bytes, while the commit\n>> + time uses the 32 bits of the second 4 bytes, along with the lowest\n>> + 2 bits of the lowest byte, storing the 33rd and 34th bit of the\n>> + commit time.\n>> +\n>> + Large Edge List (ID: {'E', 'D', 'G', 'E'}) [Optional]\n>> + This list of 4-byte values store the second through nth parents for\n>> + all octopus merges. The second parent value in the commit data stores\n>> + an array position within this list along with the most-significant bit\n>> + on. Starting at that array position, iterate through this list of int-ids\n>> + for the parents until reaching a value with the most-significant bit on.\n>> + The other bits correspond to the int-id of the last parent.\n> All right, that is one chunk that cannot use fixed-length records; this\n> shouldn't matter much, as we iterate only up to the number of parents\n> less two.\n\nLess one: the second \"parent\" column of the commit data chunk is used to \npoint into this list, so (P-1) parents are in this chunk for a commit \nwith P parents.\n\n> A question: what happens to the last list of parents? Is there a\n> guardian value of 0xffffffff at last place?\n\nThe termination condition is in the position of the last parent, since \nthe most-significant bit is on. The other 31 bits contain the int-id of \nthe parent.\n\nThanks,\n-Stolee\n\n"},{"id":"343581","messageId":"866059em48.fsf@gmail.com","threadId":"47724","inReplyTo":"433a2523-d04a-08bb-d128-6c8e578916fa@gmail.com","subject":"Re: [PATCH v4 01/13] commit-graph: add format document","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-02T14:09:11Z","receivedAt":"2018-04-02T14:09:28Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> On 3/30/2018 9:25 AM, Jakub Narebski wrote:\n>> Derrick Stolee writes:\n>>\n>>> +== graph-*.graph files have the following format:\n>> What is this '*' here?\n>\n> No longer necessary. It used to be a placeholder for a hash value, but\n> now the graph is stored in objects/info/commit-graph.\n\nAll right.\n\nExcuse me replying to v4 instead of v6 of the patch series, where it\nwould be answered or rather made moot already.\n\n>>\n>> [...]\n>>> + The remaining data in the body is described one chunk at a time, and\n>>> + these chunks may be given in any order. Chunks are required unless\n>>> + otherwise specified.\n>> Does Git need to understand all chunks, or could there be optional\n>> chunks that can be safely ignored (like in PNG format)? Though this may\n>> be overkill, and could be left for later revision of the format if\n>> deemed necessary.\n>\n> In v6, the format and design documents are edited to make clear the\n> use of optional chunks, specifically for future extension without\n> increasing the version number.\n\nThat's good.\n\n>>> +CHUNK DATA:\n>>> +\n>>> + OID Fanout (ID: {'O', 'I', 'D', 'F'}) (256 * 4 bytes)\n>>> + The ith entry, F[i], stores the number of OIDs with first\n>>> + byte at most i. Thus F[255] stores the total\n>>> + number of commits (N).\n>> All right, it is small enough that can be required even for a very small\n>> number of commits.\n>>\n>>> +\n>>> + OID Lookup (ID: {'O', 'I', 'D', 'L'}) (N * H bytes)\n>>> + The OIDs for all commits in the graph, sorted in ascending order.\n>>> +\n>>> + Commit Data (ID: {'C', 'G', 'E', 'T' }) (N * (H + 16) bytes)\n>> Do commits need to be put here in the ascending order of OIDs?\n>\n> Yes.\n>\n>> If so, this would mean that it is not possible to add information about\n>> new commits by only appending data and maybe overwriting some fields, I\n>> think. You would need to do full rewrite to insert new commit in\n>> appropriate place.\n>\n> That is the idea. This file is not updated with every new commit, but\n> instead will be updated on some scheduled cleanup events. The\n> commit-graph file is designed in a way to be non-critical, and not\n> tied to the packfile layout. This allows flexibility for when to do\n> the write.\n>\n> For example, in GVFS, we will write a new commit-graph when there are\n> new daily prefetch packs.\n>\n> This could also integrate with 'gc' and 'repack' so whenever they are\n> triggered the commit-graph is written as well.\n\nI wonder if it would be possible to use existing hooks...\n\n> Commits that do not exist in the commit-graph file will load from the\n> object database as normal (after a failed lookup in the commit-graph\n> file).\n\nAh. I thought wrongly that it would (or at least could) be something\nthat can be kept up to date, and extended when adding any new commit.\n\n>>> + * The first H bytes are for the OID of the root tree.\n>>> + * The next 8 bytes are for the int-ids of the first two parents\n>>> + of the ith commit. Stores value 0xffffffff if no parent in that\n>>> + position. If there are more than two parents, the second value\n>>> + has its most-significant bit on and the other bits store an array\n>>> + position into the Large Edge List chunk.\n>>> + * The next 8 bytes store the generation number of the commit and\n>>> + the commit time in seconds since EPOCH. The generation number\n>>> + uses the higher 30 bits of the first 4 bytes, while the commit\n>>> + time uses the 32 bits of the second 4 bytes, along with the lowest\n>>> + 2 bits of the lowest byte, storing the 33rd and 34th bit of the\n>>> + commit time.\n>>> +\n>>> + Large Edge List (ID: {'E', 'D', 'G', 'E'}) [Optional]\n>>> + This list of 4-byte values store the second through nth parents for\n>>> + all octopus merges. The second parent value in the commit data stores\n>>> + an array position within this list along with the most-significant bit\n>>> + on. Starting at that array position, iterate through this list of int-ids\n>>> + for the parents until reaching a value with the most-significant bit on.\n>>> + The other bits correspond to the int-id of the last parent.\n>>\n>> All right, that is one chunk that cannot use fixed-length records; this\n>> shouldn't matter much, as we iterate only up to the number of parents\n>> less two.\n>\n> Less one: the second \"parent\" column of the commit data chunk is used\n> to point into this list, so (P-1) parents are in this chunk for a\n> commit with P parents.\n\nRight.\n\n>> A question: what happens to the last list of parents? Is there a\n>> guardian value of 0xffffffff at last place?\n>\n> The termination condition is in the position of the last parent, since\n> the most-significant bit is on. The other 31 bits contain the int-id\n> of the parent.\n\nAh. I have misunderstood the format: I thought that first entry is\nmarked with most-significant bit set to 1, and all the rest to 0, while\nit is last entry (last parent) has most-significant bit set, while all\nothers (if any) do not. So there is no need for guardian value.\n\nBest regards,\n-- \nJakub Narębski\n"},{"id":"343582","messageId":"861sfxekeo.fsf@gmail.com","threadId":"47724","inReplyTo":"f5d8934e-afc2-39d4-b7d5-e5ba5e5021a1@gmail.com","subject":"Re: [PATCH v4 00/13] Serialized Git Commit Graph","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-02T14:46:07Z","receivedAt":"2018-04-02T14:46:18Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> On 3/30/2018 7:10 AM, Jakub Narebski wrote:\n>> I hope that I am addressing the most recent version of this series.\n>\n> Hi Jakub. Thanks for the interest in this patch series.\n>\n> The most-recent version is v6 [1], but I will re-roll to v7 soon\n> (after v2.17.0 is marked).\n>\n> [1] https://public-inbox.org/git/20180314192736.70602-1-dstolee@microsoft.com/T/#u\n\nOoops. Sorry about that.\n\n>> Derrick Stolee writes:\n[...]\n\n>> What are the assumptions about the serialized commit graph format? Does\n>> it need to be:\n>> - extensible without rewriting (e.g. append-only)?\n>> - like the above, but may need rewriting for optimal performance?\n>> - extending it needs to rewrite whole file?\n>>\n>> Excuse me if it waas already asked and answered.\n>\n> It is not extensible without rewriting. Reducing write time was not a\n> main goal, since the graph will be written only occasionally during\n> data management phases (like 'gc' or 'repack'; this integration is not\n> implemented yet).\n\nAh. I thought that it could be something easily extensible in-place,\nand thus easy to keep up to date on each commit.\n\nRecalculating it on 'gc' or 'repack' is still good, especially that it\nworks even when there are come commits outside commit-graph, without\nthis information.\n\n>>\n>>> The file format has room to store generation numbers, which will be\n>>> provided as a patch after this framework is merged. Generation numbers\n>>> are referenced by the design document but not implemented in order to\n>>> make the current patch focus on the graph construction process. Once\n>>> that is stable, it will be easier to add generation numbers and make\n>>> graph walks aware of generation numbers one-by-one.\n>>>\n>> As the serialized commit graph format is versioned, I wonder if it would\n>> be possible to speed up graph walks even more by adding to it FELINE\n>> index (pair of numbers) from \"Reachability Queries in Very Large Graphs:\n>> A Fast Refined Olnine Search Approach\" (2014) - available at\n>> http://openproceedings.org/EDBT/2014/paper_166.pdf\n>>\n>> The implementation would probably need adjustments to make it\n>> unambiguous and unambiguously extensible; unless there is place for\n>> indices that are local-only and need to be recalculated from scratch\n>> when graph changes (to cover all graph).\n>\n> The chunk-based format is intended to allow extra indexes like the one\n> you recommend, without needing to increase the version number. Using\n> an optional chunk allows older versions of Git to read the file\n> without error, since the data is \"extra\", and newer versions can take\n> advantage of the acceleration.\n\nThat's good.\n\n> At one point, I was investigating these reachability indexes (I read\n> \"SCARAB: Scaling Reachability Computation on Large Graphs\" by Jihn,\n> Ruan, Dey, and Xu [2]) but find the question that these indexes target\n> to be lacking for most of the Git uses. That is, they ask the boolean\n> question \"Can X reach Y?\". More often, Git needs to answer \"What is\n> the set of commits reachable from X but not from Y\" or \"Topologically\n> sort commits reachable from X\" or \"How many commits are in each part\n> of the symmetric difference between reachable from X or reachable from\n> Y?\"\n\nIn the \"Reachability Queries in Very Large Graphs...\" by Veloso, Cerf,\nMeira and Zaki FELINE-index work, authors mention SCARAB as something\nthat can be used in addition to FELINE-index, as a complementary data\n(FELINE-SCARAB in the work, section 4.4).\n\nI see the FELINE-index as a stronger form of generation numbers (called\nalso level of the vertex / node), in that it allows to negative-cut even\nmore, pruning paths that are known to be unreachable (or marking nodes\nknown to be unreachable in the \"calculate difference\" scenario). \n\nAlso, FELINE-index uses two integer numbers (coordinates in 2d space);\none of those indices can be the topological numbering (topological\nsorting order) of nodes in the commit graph. That would help to answer\neven more Git questions.\n\n> The case for \"Can X reach Y?\" is mostly for commands like 'git branch\n> --contains', when 'git fetch' checks for forced-updates of branches,\n> or when the server decides enough negotiation has occurred during a\n> 'git fetch'. While these may be worth investigating, they also benefit\n> greatly from the accelerated graph walk introduced in the current\n> format.\n>\n> I would be happy to review any effort to extend the commit-graph\n> format to include such indexes, as long as the performance benefits\n> outweigh the complexity to create them.\n>\n> [2] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.719.8396&rep=rep1&type=pdf\n\nThe complexity of calculating FELINE index is O(|V| log(|V|) + |E|), the\nstorage complexity is 2*|V|.\n\n>>\n>>> Here are some performance results for a copy of the Linux repository\n>>> where 'master' has 704,766 reachable commits and is behind 'origin/master'\n>>> by 19,610 commits.\n>>>\n>>> | Command | Before | After | Rel % |\n>>> |----------------------------------|--------|--------|-------|\n>>> | log --oneline --topo-order -1000 | 5.9s | 0.7s | -88% |\n>>> | branch -vv | 0.42s | 0.27s | -35% |\n>>> | rev-list --all | 6.4s | 1.0s | -84% |\n>>> | rev-list --all --objects | 32.6s | 27.6s | -15% |\n>>\n>> That's the \"Rel %\" of \"Before\", that is delta/before, isn't it?\n>\n> I do mean the relative change.\n\nBut is it relative to the state before, or relative to the state after?\n\n[...]\n>> I just wonder what happens in the (rare) presence of grafts (old\n>> mechanism), or \"git replace\"-d commits...\n>\n> In the design document, I mention that the current implementation does\n> not work with grafts (it will ignore them). A later patch will\n> refactor the graft code so we can access it from the commit-graph\n> parsing of a commit without copy-pasting the code out of\n> parse_commit_gently().\n>\n> The commit-graph is only a compact representation of the object\n> database. If a commit is replaced with 'git replace' before 'git\n> commit-graph write' then the commit-graph write will write the\n> replaced object. I haven't tested what happens when a commit-graph is\n> written and then a commit is replaced, but my guess is that the\n> replacement does not occur until a full parse is attempted (i.e. when\n> reading author or commit message information). This will lead to\n> unknown results.\n>\n> Thanks for pointing out the interaction with 'git replace'. I have\n> items to fix grafts and replaced commits before integrating\n> commit-graph writes into automatic actions like 'gc.auto'.\n\nNote that you can make Git ignore replacements with appropriate command\nline option for \"git\" wrapper; the transfer mechanism can safely\nignore replacements (treating refs/replacements just like it would any\nother refs/).\n\nBest regards,\n--\nJakub Narębski\n"},{"id":"343583","messageId":"d3060103-7fab-3b6c-e968-cb49f3f23acb@gmail.com","threadId":"47724","inReplyTo":"861sfxekeo.fsf@gmail.com","subject":"Re: [PATCH v4 00/13] Serialized Git Commit Graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-02T15:02:09Z","receivedAt":"2018-04-02T15:02:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/2/2018 10:46 AM, Jakub Narebski wrote:\n> Derrick Stolee writes:\n[...]\n>> At one point, I was investigating these reachability indexes (I read\n>> \"SCARAB: Scaling Reachability Computation on Large Graphs\" by Jihn,\n>> Ruan, Dey, and Xu [2]) but find the question that these indexes target\n>> to be lacking for most of the Git uses. That is, they ask the boolean\n>> question \"Can X reach Y?\". More often, Git needs to answer \"What is\n>> the set of commits reachable from X but not from Y\" or \"Topologically\n>> sort commits reachable from X\" or \"How many commits are in each part\n>> of the symmetric difference between reachable from X or reachable from\n>> Y?\"\n> In the \"Reachability Queries in Very Large Graphs...\" by Veloso, Cerf,\n> Meira and Zaki FELINE-index work, authors mention SCARAB as something\n> that can be used in addition to FELINE-index, as a complementary data\n> (FELINE-SCARAB in the work, section 4.4).\n>\n> I see the FELINE-index as a stronger form of generation numbers (called\n> also level of the vertex / node), in that it allows to negative-cut even\n> more, pruning paths that are known to be unreachable (or marking nodes\n> known to be unreachable in the \"calculate difference\" scenario).\n>\n> Also, FELINE-index uses two integer numbers (coordinates in 2d space);\n> one of those indices can be the topological numbering (topological\n> sorting order) of nodes in the commit graph. That would help to answer\n> even more Git questions.\n\nThis two-dimensional generation number is helpful for non-reachability \nqueries, but is something that needs the \"full\" commit graph in order to \ndefine the value for a single commit (hence the O(N lg N) performance \nmentioned below). Generation numbers are effective while being easy to \ncompute and immutable.\n\nI wonder if FELINE was compared directly to a one-dimensional index (I \napologize that I have not read the paper in detail, so I don't \nunderstand the indexes they compare against). It also appears the graphs \nthey use for their tests are not commit graphs, which have a different \nshape than many of the digraphs studies by that work.\n\nThis is all to say: I would love to see an interesting study in this \ndirection, specifically comparing this series' definition of generation \nnumbers to the 2-dimensional system in FELINE, and on a large sample of \ncommit graphs available in open-source data sets (Linux kernel, Git, \netc.) and possibly on interesting closed-source data sets.\n\n>\n>> The case for \"Can X reach Y?\" is mostly for commands like 'git branch\n>> --contains', when 'git fetch' checks for forced-updates of branches,\n>> or when the server decides enough negotiation has occurred during a\n>> 'git fetch'. While these may be worth investigating, they also benefit\n>> greatly from the accelerated graph walk introduced in the current\n>> format.\n>>\n>> I would be happy to review any effort to extend the commit-graph\n>> format to include such indexes, as long as the performance benefits\n>> outweigh the complexity to create them.\n>>\n>> [2] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.719.8396&rep=rep1&type=pdf\n> The complexity of calculating FELINE index is O(|V| log(|V|) + |E|), the\n> storage complexity is 2*|V|.\n>\n\nThis would be very easy to add as an optional chunk, since it can use \none row per commit.\n\nThanks,\n-Stolee\n"},{"id":"343588","messageId":"CAGZ79kaoDNNcu8zqaTnoXWfJ0FnLwqvZfOHUniG-izndAjvDVw@mail.gmail.com","threadId":"47724","inReplyTo":"d3060103-7fab-3b6c-e968-cb49f3f23acb@gmail.com","subject":"Re: [PATCH v4 00/13] Serialized Git Commit Graph","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-04-02T17:35:49Z","receivedAt":"2018-04-02T17:35:56Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Apr 2, 2018 at 8:02 AM, Derrick Stolee wrote:\n>>>\n>>> I would be happy to review any effort to extend the commit-graph\n>>> format to include such indexes, as long as the performance benefits\n>>> outweigh the complexity to create them.\n>>>\n>>> [2]\n>>> http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.719.8396&rep=rep1&type=pdf\n>>\n>> The complexity of calculating FELINE index is O(|V| log(|V|) + |E|), the\n>> storage complexity is 2*|V|.\n>>\n>\n> This would be very easy to add as an optional chunk, since it can use one\n> row per commit.\n\nGiven this discussion, I wonder if we want to include generation numbers\nas a first class citizen in the current format. They could also go as\nan optional\nchunk and we may want to discuss further if we want generation numbers or\nFELINE numbers or GRAIL or SCARAB, which are all graph related speedup\nmechanism AFAICT.\nIn case we decide against generation numbers in the long run,\nthe row of mandatory generation numbers would be dead weight\nthat we'd need to carry.\n\nI only glanced at the paper, but it looks like a \"more advanced 2d\ngeneration number\" that seems to be able to answer questions\nthat gen numbers can answer, but that paper also refers\nto SCARAB as well as GRAIL as the state of the art, so maybe\nthere are even more papers to explore?\n\nStefan\n"},{"id":"343591","messageId":"17430580-aae6-8693-c4c9-0935104bc535@gmail.com","threadId":"47724","inReplyTo":"CAGZ79kaoDNNcu8zqaTnoXWfJ0FnLwqvZfOHUniG-izndAjvDVw@mail.gmail.com","subject":"Re: [PATCH v4 00/13] Serialized Git Commit Graph","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-04-02T17:54:12Z","receivedAt":"2018-04-02T17:54:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/2/2018 1:35 PM, Stefan Beller wrote:\n> On Mon, Apr 2, 2018 at 8:02 AM, Derrick Stolee wrote:\n>>>> I would be happy to review any effort to extend the commit-graph\n>>>> format to include such indexes, as long as the performance benefits\n>>>> outweigh the complexity to create them.\n>>>>\n>>>> [2]\n>>>> http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.719.8396&rep=rep1&type=pdf\n>>> The complexity of calculating FELINE index is O(|V| log(|V|) + |E|), the\n>>> storage complexity is 2*|V|.\n>>>\n>> This would be very easy to add as an optional chunk, since it can use one\n>> row per commit.\n> Given this discussion, I wonder if we want to include generation numbers\n> as a first class citizen in the current format. They could also go as\n> an optional\n> chunk and we may want to discuss further if we want generation numbers or\n> FELINE numbers or GRAIL or SCARAB, which are all graph related speedup\n> mechanism AFAICT.\n> In case we decide against generation numbers in the long run,\n> the row of mandatory generation numbers would be dead weight\n> that we'd need to carry.\n\nCurrently, the format includes 8 bytes to share between the generation \nnumber and commit date. Due to alignment concerns, we will want to keep \nthis as 8 bytes or truncate it to 4-bytes. Either we would be wasting at \nleast 3 bytes or truncating dates too much (presenting the 2038 problem \n[1] since dates are signed).\n\n> I only glanced at the paper, but it looks like a \"more advanced 2d\n> generation number\" that seems to be able to answer questions\n> that gen numbers can answer, but that paper also refers\n> to SCARAB as well as GRAIL as the state of the art, so maybe\n> there are even more papers to explore?\n\nThe biggest reason I can say to advance this series (and the small \nfollow-up series that computes and consumes generation numbers) is that \ngeneration numbers are _extremely simple_. You only need to know your \nparents and their generation numbers to compute your own. These other \nreachability indexes require examining the entire graph to create \"good\" \nindex values.\n\nThe hard part about using generation numbers (or any other reachability \nindex) in Git is refactoring the revision-walk machinery to take \nadvantage of them; current code requires O(reachable commits) to \ntopo-order instead of O(commits that will be output). I think we should \ntable any discussion of these advanced indexes until that work is done \nand a valuable comparison can be done. \"Premature optimization is the \nroot of all evil\" and all that.\n\nThanks,\n-Stolee\n\n[1] https://en.wikipedia.org/wiki/Year_2038_problem\n"},{"id":"343593","messageId":"CAGZ79kZZCFTm-+=E-HOMNt-3wKS=ZSyCz-rrPsY+mPQXq3VXAg@mail.gmail.com","threadId":"47724","inReplyTo":"17430580-aae6-8693-c4c9-0935104bc535@gmail.com","subject":"Re: [PATCH v4 00/13] Serialized Git Commit Graph","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-04-02T18:02:49Z","receivedAt":"2018-04-02T18:02:55Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> Currently, the format includes 8 bytes to share between the generation\n> number and commit date. Due to alignment concerns, we will want to keep this\n> as 8 bytes or truncate it to 4-bytes. Either we would be wasting at least 3\n> bytes or truncating dates too much (presenting the 2038 problem [1] since\n> dates are signed).\n\nGood point. I forgot about them while writing the previous email.\nThat is reason enough to keep the generation numbers, sorry\nfor the noise.\n\n>\n>> I only glanced at the paper, but it looks like a \"more advanced 2d\n>> generation number\" that seems to be able to answer questions\n>> that gen numbers can answer, but that paper also refers\n>> to SCARAB as well as GRAIL as the state of the art, so maybe\n>> there are even more papers to explore?\n>\n>\n> The biggest reason I can say to advance this series (and the small follow-up\n> series that computes and consumes generation numbers) is that generation\n> numbers are _extremely simple_. You only need to know your parents and their\n> generation numbers to compute your own. These other reachability indexes\n> require examining the entire graph to create \"good\" index values.\n\nYes, that is a good point, too. Generation numbers can be computed\n\"commit locally\" and do not need expensive setups, which the others\npresumably need.\n\n> The hard part about using generation numbers (or any other reachability\n> index) in Git is refactoring the revision-walk machinery to take advantage\n> of them; current code requires O(reachable commits) to topo-order instead of\n> O(commits that will be output). I think we should table any discussion of\n> these advanced indexes until that work is done and a valuable comparison can\n> be done. \"Premature optimization is the root of all evil\" and all that.\n\nagreed,\n\nStefan\n"},{"id":"344077","messageId":"86po3aej8n.fsf@gmail.com","threadId":"47724","inReplyTo":"d3060103-7fab-3b6c-e968-cb49f3f23acb@gmail.com","subject":"Re: [PATCH v4 00/13] Serialized Git Commit Graph","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2018-04-07T22:37:12Z","receivedAt":"2018-04-07T22:37:28Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Derrick Stolee writes:\n\n> On 4/2/2018 10:46 AM, Jakub Narebski wrote:\n>> Derrick Stolee writes:\n> [...]\n>> I see the FELINE-index as a stronger form of generation numbers (called\n>> also level of the vertex / node), in that it allows to negative-cut even\n>> more, pruning paths that are known to be unreachable (or marking nodes\n>> known to be unreachable in the \"calculate difference\" scenario).\n>>\n>> Also, FELINE-index uses two integer numbers (coordinates in 2d space);\n>> one of those indices can be the topological numbering (topological\n>> sorting order) of nodes in the commit graph. That would help to answer\n>> even more Git questions.\n>\n> This two-dimensional generation number is helpful for non-reachability\n> queries, but is something that needs the \"full\" commit graph in order\n> to define the value for a single commit (hence the O(N lg N)\n> performance mentioned below). Generation numbers are effective while\n> being easy to compute and immutable.\n\nO(N log N) is the cost of sort algorithm, namely peforming topological\nsorting of commits. Which Git can currently do.\n\nWe can use the main idea behind FELINE index, namely weak dominance\ndrawing, while perhaps changing the detail. The main idea is that if\nyou draw edges of DAG always to be in selected quadrant -- the default\nis drawing them up and to the right, then paths would also always be in\nthe same quadrant; if target node is not in given quadrant with respect\nto source node, then target node cannot be reached from source node.\n\nFor fast answering of reachability queries it is important to have\nminimal number of false positives (falsely implied paths), that is\nsituation where FELINE index does imply that there could be a path, but\nin fact target is not reachable from the source. The authors propose a\nheuristics: use positions in [some] topological order for X coordinates\nof FELINE index, and generate new optimal locally topological sorting to\nuse for Y coordinates.\n\n\nGeneration numbers (which can be used together with FELINE index, and\nwhat paper does use -- calling them Level Filter) have the advantage of\nbeing able to be calculated incrementally. I think this is what you\nwanted to say.\n\nI think it would be possible to calculate FELINE index incrementally,\ntoo. If Git's topological order is stable, at least for X coordinates\nof FELINE index it would be easy.\n\n\nI have started experimenting with this approach in Jupyter Notebook,\navailable on Google Colaboratory (you can examine it, run it and edit it\nfrom web browser -- the default is to use remote runtime on Google\ncloud). Currently I am at the stage of reproducing the theoretical\nparts of the FELINE paper.\n\n https://colab.research.google.com/drive/1V-U7_slu5Z3s5iEEMFKhLXtaxSu5xyzg\n https://drive.google.com/file/d/1V-U7_slu5Z3s5iEEMFKhLXtaxSu5xyzg/view?usp=sharing\n\n> I wonder if FELINE was compared directly to a one-dimensional index (I\n> apologize that I have not read the paper in detail, so I don't\n> understand the indexes they compare against).\n\nThey have compared FELINE using real graphs (like ArXiv, Pubmed,\nCiteseerX, Uniprot150m) and synthetic DAG against:\n - GRAIL (Graph Reachability Indexing via RAndomized Interval Labeling)\n - FERRARI (Flexible and Efficient Reachability Range Assignment for\n gRapth Indexing), with interval set compression to approximate ones\n - Nuutila's INTERVAL, which extracts complete transitive closure of\n the graph and stores it using PWAH bit-vector compression\n - TF-Label (Topological Folding LABELling)\n\n> It also appears the\n> graphs they use for their tests are not commit graphs, which have a\n> different shape than many of the digraphs studies by that work.\n\nI plan to check how FELINE index works for commit graphs in\nabove-mentioned Jupyter Notebook.\n\n> This is all to say: I would love to see an interesting study in this\n> direction, specifically comparing this series' definition of\n> generation numbers to the 2-dimensional system in FELINE, and on a\n> large sample of commit graphs available in open-source data sets\n> (Linux kernel, Git, etc.) and possibly on interesting closed-source\n> data sets.\n\nI wonder if there are more recent works, with even faster solutions.\n\n>>> The case for \"Can X reach Y?\" is mostly for commands like 'git branch\n>>> --contains', when 'git fetch' checks for forced-updates of branches,\n>>> or when the server decides enough negotiation has occurred during a\n>>> 'git fetch'. While these may be worth investigating, they also benefit\n>>> greatly from the accelerated graph walk introduced in the current\n>>> format.\n>>>\n>>> I would be happy to review any effort to extend the commit-graph\n>>> format to include such indexes, as long as the performance benefits\n>>> outweigh the complexity to create them.\n>>>\n>>> [2] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.719.8396&rep=rep1&type=pdf\n\nI wonder if next low-hanging branch would be to store topological\nordering of commits. It could be done, I think, with two chunks (or two\nparts of one chunk): first to store position in topological order for\neach commit (entries sorted by hash), second to store list of commits in\ntopological order (entries sorted by topological sort).\n\n>>\n>> The complexity of calculating FELINE index is O(|V| log(|V|) + |E|), the\n>> storage complexity is 2*|V|.\n>>\n>\n> This would be very easy to add as an optional chunk, since it can use\n> one row per commit.\n\nRight.\n\n-- \nJakub Narębski\n"}]}