{"thread":{"id":"49914","subject":"[PATCH 0/5] Add a new \"sparse\" tree walk algorithm","startedAt":"2018-11-28T21:52:44Z","lastAt":"2019-01-16T18:26:14Z","messageCount":51,"participants":["Derrick Stolee via GitGitGadget","Stefan Beller","Ævar Arnfjörð Bjarmason","Derrick Stolee","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"364253","messageId":"pull.89.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":null,"subject":"[PATCH 0/5] Add a new \"sparse\" tree walk algorithm","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-11-28T21:52:41Z","receivedAt":"2018-11-28T21:52:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"One of the biggest remaining pain points for users of very large\nrepositories is the time it takes to run 'git push'. We inspected some slow\npushes by our developers and found that the \"Enumerating Objects\" phase of a\npush was very slow. This is unsurprising, because this is why reachability\nbitmaps exist. However, reachability bitmaps are not available to us because\nof the single pack-file requirement. The bitmap approach is intended for\nservers anyway, and clients have a much different behavior pattern.\n\nSpecifically, clients are normally pushing a very small number of objects\ncompared to the entire working directory. A typical user changes only a\nsmall cone of the working directory, so let's use that to our benefit.\n\nCreate a new \"sparse\" mode for 'git pack-objects' that uses the paths that\nintroduce new objects to direct our search into the reachable trees. By\ncollecting trees at each path, we can then recurse into a path only when\nthere are uninteresting and interesting trees at that path. This gains a\nsignificant performance boost for small topics while presenting a\npossibility of packing extra objects.\n\nThe main algorithm change is in patch 4, but is set up a little bit in\npatches 1 and 2.\n\nAs demonstrated in the included test script, we see that the existing\nalgorithm can send extra objects due to the way we specify the \"frontier\".\nBut we can send even more objects if a user copies objects from one folder\nto another. I say \"copy\" because a rename would (usually) change the\noriginal folder and trigger a walk into that path, discovering the objects.\n\nIn order to benefit from this approach, the user can opt-in using the\npack.useSparse config setting. This setting can be overridden using the\n'--no-sparse' option.\n\nDerrick Stolee (5):\n  revision: add mark_tree_uninteresting_sparse\n  list-objects: consume sparse tree walk\n  pack-objects: add --sparse option\n  revision: implement sparse algorithm\n  pack-objects: create pack.useSparse setting\n\n Documentation/git-pack-objects.txt |   9 +-\n bisect.c                           |   2 +-\n builtin/pack-objects.c             |   9 +-\n builtin/rev-list.c                 |   2 +-\n http-push.c                        |   2 +-\n list-objects.c                     |  51 ++++++++++-\n list-objects.h                     |   4 +-\n revision.c                         | 113 +++++++++++++++++++++++\n revision.h                         |   2 +\n t/t5322-pack-objects-sparse.sh     | 139 +++++++++++++++++++++++++++++\n 10 files changed, 323 insertions(+), 10 deletions(-)\n create mode 100755 t/t5322-pack-objects-sparse.sh\n\n\nbase-commit: a1598010f775d82b5adf12c29d0f5bc9b41434c6\nPublished-As: https://github.com/gitgitgadget/git/releases/tags/pr-89%2Fderrickstolee%2Fpush%2Fsparse-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-89/derrickstolee/push/sparse-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/89\n-- \ngitgitgadget\n"},{"id":"364254","messageId":"b73b8de98c9c69a0396b435cb093ee522632796d.1543441959.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.git.gitgitgadget@gmail.com","subject":"[PATCH 1/5] revision: add mark_tree_uninteresting_sparse","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-11-28T21:52:43Z","receivedAt":"2018-11-28T21:52:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nIn preparation for a new algorithm that walks fewer trees when\ncreating a pack from a set of revisions, create a method that\ntakes an oidset of tree oids and marks reachable objects as\nUNINTERESTING.\n\nThe current implementation uses the existing\nmark_tree_uninteresting to recursively walk the trees and blobs.\nThis will walk the same number of trees as the old mechanism.\n\nThere is one new assumption in this approach: we are also given\nthe oids of the interesting trees. This implementation does not\nuse those trees at the moment, but we will use them in a later\nrewrite of this method.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n revision.c | 22 ++++++++++++++++++++++\n revision.h |  2 ++\n 2 files changed, 24 insertions(+)\n\ndiff --git a/revision.c b/revision.c\nindex 13e0519c02..3a62c7c187 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -99,6 +99,28 @@ void mark_tree_uninteresting(struct repository *r, struct tree *tree)\n \tmark_tree_contents_uninteresting(r, tree);\n }\n \n+void mark_trees_uninteresting_sparse(struct repository *r,\n+\t\t\t\t     struct oidset *set)\n+{\n+\tstruct object_id *oid;\n+\tstruct oidset_iter iter;\n+\n+\toidset_iter_init(set, &iter);\n+\twhile ((oid = oidset_iter_next(&iter))) {\n+\t\tstruct tree *tree = lookup_tree(r, oid);\n+\n+\t\tif (tree->object.flags & UNINTERESTING) {\n+\t\t\t/*\n+\t\t\t * Remove the flag so the next call\n+\t\t\t * is not a no-op. The flag is added\n+\t\t\t * in mark_tree_unintersting().\n+\t\t\t */\n+\t\t\ttree->object.flags ^= UNINTERESTING;\n+\t\t\tmark_tree_uninteresting(r, tree);\n+\t\t}\n+\t}\n+}\n+\n struct commit_stack {\n \tstruct commit **items;\n \tsize_t nr, alloc;\ndiff --git a/revision.h b/revision.h\nindex 7987bfcd2e..f828e91ae9 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -67,6 +67,7 @@ struct rev_cmdline_info {\n #define REVISION_WALK_NO_WALK_SORTED 1\n #define REVISION_WALK_NO_WALK_UNSORTED 2\n \n+struct oidset;\n struct topo_walk_info;\n \n struct rev_info {\n@@ -327,6 +328,7 @@ void put_revision_mark(const struct rev_info *revs,\n \n void mark_parents_uninteresting(struct commit *commit);\n void mark_tree_uninteresting(struct repository *r, struct tree *tree);\n+void mark_trees_uninteresting_sparse(struct repository *r, struct oidset *set);\n \n void show_object_with_name(FILE *, struct object *, const char *);\n \n-- \ngitgitgadget\n\n"},{"id":"364255","messageId":"9bf04c748bbe9b0c9787cc9af92cbb0fb69e81f8.1543441959.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.git.gitgitgadget@gmail.com","subject":"[PATCH 2/5] list-objects: consume sparse tree walk","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-11-28T21:52:44Z","receivedAt":"2018-11-28T21:52:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen creating a pack-file using 'git pack-objects --revs' we provide\na list of interesting and uninteresting commits. For example, a push\noperation would make the local topic branch be interesting and the\nknown remote refs as uninteresting. We want to discover the set of\nnew objects to send to the server as a thin pack.\n\nWe walk these commits until we discover a frontier of commits such\nthat every commit walk starting at interesting commits ends in a root\ncommit or unintersting commit. We then need to discover which\nnon-commit objects are reachable from  uninteresting commits.\n\nThe mark_edges_unintersting() method in list-objects.c iterates on\nthe commit list and does the following:\n\n* If the commit is UNINTERSTING, then mark its root tree and every\n  object it can reach as UNINTERESTING.\n\n* If the commit is interesting, then mark the root tree of every\n  UNINTERSTING parent (and all objects that tree can reach) as\n  UNINTERSTING.\n\nAt the very end, we repeat the process on every commit directly\ngiven to the revision walk from stdin. This helps ensure we properly\ncover shallow commits that otherwise were not included in the\nfrontier.\n\nThe logic to recursively follow trees is in the\nmark_tree_uninteresting() method in revision.c. The algorithm avoids\nduplicate work by not recursing into trees that are already marked\nUNINTERSTING.\n\nAdd a new 'sparse' option to the mark_edges_uninteresting() method\nthat performs this logic in a slightly new way. As we iterate over\nthe commits, we add all of the root trees to an oidset. Then, call\nmark_trees_uninteresting_sparse() on that oidset. Note that we\ninclude interesting trees in this process. The current implementation\nof mark_trees_unintersting_sparse() will walk the same trees as\nthe old logic, but this will be replaced in a later change.\n\nThe sparse option is not used by any callers at the moment, but\nwill be wired to 'git pack-objects' in the next change.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n bisect.c               |  2 +-\n builtin/pack-objects.c |  2 +-\n builtin/rev-list.c     |  2 +-\n http-push.c            |  2 +-\n list-objects.c         | 51 ++++++++++++++++++++++++++++++++++++++----\n list-objects.h         |  4 +++-\n 6 files changed, 54 insertions(+), 9 deletions(-)\n\ndiff --git a/bisect.c b/bisect.c\nindex 487675c672..842f8b4b8f 100644\n--- a/bisect.c\n+++ b/bisect.c\n@@ -656,7 +656,7 @@ static void bisect_common(struct rev_info *revs)\n \tif (prepare_revision_walk(revs))\n \t\tdie(\"revision walk setup failed\");\n \tif (revs->tree_objects)\n-\t\tmark_edges_uninteresting(revs, NULL);\n+\t\tmark_edges_uninteresting(revs, NULL, 0);\n }\n \n static void exit_if_skipped_commits(struct commit_list *tried,\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 411aefd687..5f70d840a7 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3135,7 +3135,7 @@ static void get_object_list(int ac, const char **av)\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n-\tmark_edges_uninteresting(&revs, show_edge);\n+\tmark_edges_uninteresting(&revs, show_edge, 0);\n \n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 2880ed37e3..9663cbfae0 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -543,7 +543,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \tif (revs.tree_objects)\n-\t\tmark_edges_uninteresting(&revs, show_edge);\n+\t\tmark_edges_uninteresting(&revs, show_edge, 0);\n \n \tif (bisect_list) {\n \t\tint reaches, all;\ndiff --git a/http-push.c b/http-push.c\nindex cd48590912..ea52d6f9f6 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -1933,7 +1933,7 @@ int cmd_main(int argc, const char **argv)\n \t\tpushing = 0;\n \t\tif (prepare_revision_walk(&revs))\n \t\t\tdie(\"revision walk setup failed\");\n-\t\tmark_edges_uninteresting(&revs, NULL);\n+\t\tmark_edges_uninteresting(&revs, NULL, 0);\n \t\tobjects_to_send = get_delta(&revs, ref_lock);\n \t\tfinish_all_active_slots();\n \ndiff --git a/list-objects.c b/list-objects.c\nindex c41cc80db5..9bb93d1640 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -222,25 +222,68 @@ static void mark_edge_parents_uninteresting(struct commit *commit,\n \t}\n }\n \n-void mark_edges_uninteresting(struct rev_info *revs, show_edge_fn show_edge)\n+static void add_edge_parents(struct commit *commit,\n+\t\t\t     struct rev_info *revs,\n+\t\t\t     show_edge_fn show_edge,\n+\t\t\t     struct oidset *set)\n+{\n+\tstruct commit_list *parents;\n+\n+\tfor (parents = commit->parents; parents; parents = parents->next) {\n+\t\tstruct commit *parent = parents->item;\n+\t\tstruct tree *tree = get_commit_tree(parent);\n+\t\toidset_insert(set, &tree->object.oid);\n+\n+\t\tif (!(parent->object.flags & UNINTERESTING))\n+\t\t\tcontinue;\n+\t\ttree->object.flags |= UNINTERESTING;\n+\n+\t\tif (revs->edge_hint && !(parent->object.flags & SHOWN)) {\n+\t\t\tparent->object.flags |= SHOWN;\n+\t\t\tshow_edge(parent);\n+\t\t}\n+\t}\n+}\n+\n+void mark_edges_uninteresting(struct rev_info *revs,\n+\t\t\t      show_edge_fn show_edge,\n+\t\t\t      int sparse)\n {\n \tstruct commit_list *list;\n+\tstruct oidset set;\n \tint i;\n \n+\tif (sparse)\n+\t\toidset_init(&set, 16);\n+\n \tfor (list = revs->commits; list; list = list->next) {\n \t\tstruct commit *commit = list->item;\n+\t\t\n+\t\tif (sparse) {\n+\t\t\tstruct tree *tree = get_commit_tree(commit);\n+\t\t\t\n+\t\t\tif (commit->object.flags & UNINTERESTING)\n+\t\t\t\ttree->object.flags |= UNINTERESTING;\n \n-\t\tif (commit->object.flags & UNINTERESTING) {\n+\t\t\toidset_insert(&set, &tree->object.oid);\n+\t\t\tadd_edge_parents(commit, revs, show_edge, &set);\n+\t\t} else if (commit->object.flags & UNINTERESTING) {\n \t\t\tmark_tree_uninteresting(revs->repo,\n \t\t\t\t\t\tget_commit_tree(commit));\n \t\t\tif (revs->edge_hint_aggressive && !(commit->object.flags & SHOWN)) {\n \t\t\t\tcommit->object.flags |= SHOWN;\n \t\t\t\tshow_edge(commit);\n \t\t\t}\n-\t\t\tcontinue;\n+\t\t} else {\n+\t\t\tmark_edge_parents_uninteresting(commit, revs, show_edge);\n \t\t}\n-\t\tmark_edge_parents_uninteresting(commit, revs, show_edge);\n \t}\n+\n+\tif (sparse) {\n+\t\tmark_trees_uninteresting_sparse(revs->repo, &set);\n+\t\toidset_clear(&set);\n+\t}\n+\n \tif (revs->edge_hint_aggressive) {\n \t\tfor (i = 0; i < revs->cmdline.nr; i++) {\n \t\t\tstruct object *obj = revs->cmdline.rev[i].item;\ndiff --git a/list-objects.h b/list-objects.h\nindex ad40762926..a952680e46 100644\n--- a/list-objects.h\n+++ b/list-objects.h\n@@ -10,7 +10,9 @@ typedef void (*show_object_fn)(struct object *, const char *, void *);\n void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);\n \n typedef void (*show_edge_fn)(struct commit *);\n-void mark_edges_uninteresting(struct rev_info *, show_edge_fn);\n+void mark_edges_uninteresting(struct rev_info *revs,\n+\t\t\t      show_edge_fn show_edge,\n+\t\t\t      int sparse);\n \n struct oidset;\n struct list_objects_filter_options;\n-- \ngitgitgadget\n\n"},{"id":"364256","messageId":"9d6b8f6d0602e85652b2a748c58eeed4cbf4359e.1543441960.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.git.gitgitgadget@gmail.com","subject":"[PATCH 3/5] pack-objects: add --sparse option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-11-28T21:52:45Z","receivedAt":"2018-11-28T21:52:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nAdd a '--sparse' option flag to the pack-objects builtin. This\nallows the user to specify that they want to use the new logic\nfor walking trees. This logic currently does not differ from the\nexisting output, but will in a later change.\n\nCreate a new test script, t5322-pack-objects-sparse.sh, to ensure\nthe object list that is selected matches what we expect. When we\nupdate the logic to walk in a sparse fashion, the final test will\nbe updated to show the extra objects that are added.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n Documentation/git-pack-objects.txt |   9 ++-\n builtin/pack-objects.c             |   5 +-\n t/t5322-pack-objects-sparse.sh     | 115 +++++++++++++++++++++++++++++\n 3 files changed, 127 insertions(+), 2 deletions(-)\n create mode 100755 t/t5322-pack-objects-sparse.sh\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 40c825c381..ced2630eb3 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -14,7 +14,7 @@ SYNOPSIS\n \t[--local] [--incremental] [--window=<n>] [--depth=<n>]\n \t[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\n \t[--stdout [--filter=<filter-spec>] | base-name]\n-\t[--shallow] [--keep-true-parents] < object-list\n+\t[--shallow] [--keep-true-parents] [--sparse] < object-list\n \n \n DESCRIPTION\n@@ -196,6 +196,13 @@ depth is 4095.\n \tAdd --no-reuse-object if you want to force a uniform compression\n \tlevel on all data no matter the source.\n \n+--sparse::\n+\tUse the \"sparse\" algorithm to determine which objects to include in\n+\tthe pack. This can have significant performance benefits when computing\n+\ta pack to send a small change. However, it is possible that extra\n+\tobjects are added to the pack-file if the included commits contain\n+\tcertain types of direct renames.\n+\n --thin::\n \tCreate a \"thin\" pack by omitting the common objects between a\n \tsender and a receiver in order to reduce network transfer. This\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5f70d840a7..7d5b0735e3 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -84,6 +84,7 @@ static unsigned long pack_size_limit;\n static int depth = 50;\n static int delta_search_threads;\n static int pack_to_stdout;\n+static int sparse;\n static int thin;\n static int num_preferred_base;\n static struct progress *progress_state;\n@@ -3135,7 +3136,7 @@ static void get_object_list(int ac, const char **av)\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n-\tmark_edges_uninteresting(&revs, show_edge, 0);\n+\tmark_edges_uninteresting(&revs, show_edge, sparse);\n \n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\n@@ -3292,6 +3293,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t{ OPTION_CALLBACK, 0, \"unpack-unreachable\", NULL, N_(\"time\"),\n \t\t  N_(\"unpack unreachable objects newer than <time>\"),\n \t\t  PARSE_OPT_OPTARG, option_parse_unpack_unreachable },\n+\t\tOPT_BOOL(0, \"sparse\", &sparse,\n+\t\t\t N_(\"use the sparse reachability algorithm\")),\n \t\tOPT_BOOL(0, \"thin\", &thin,\n \t\t\t N_(\"create thin packs\")),\n \t\tOPT_BOOL(0, \"shallow\", &shallow,\ndiff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\nnew file mode 100755\nindex 0000000000..81f6805bc3\n--- /dev/null\n+++ b/t/t5322-pack-objects-sparse.sh\n@@ -0,0 +1,115 @@\n+#!/bin/sh\n+\n+test_description='pack-objects object selection using sparse algorithm'\n+. ./test-lib.sh\n+\n+test_expect_success 'setup repo' '\n+\ttest_commit initial &&\n+\tfor i in $(test_seq 1 3)\n+\tdo\n+\t\tmkdir f$i &&\n+\t\tfor j in $(test_seq 1 3)\n+\t\tdo\n+\t\t\tmkdir f$i/f$j &&\n+\t\t\techo $j >f$i/f$j/data.txt\n+\t\tdone\n+\tdone &&\n+\tgit add . &&\n+\tgit commit -m \"Initialized trees\" &&\n+\tfor i in $(test_seq 1 3)\n+\tdo\n+\t\tgit checkout -b topic$i master &&\n+\t\techo change-$i >f$i/f$i/data.txt &&\n+\t\tgit commit -a -m \"Changed f$i/f$i/data.txt\"\n+\tdone &&\n+\tcat >packinput.txt <<-EOF &&\n+\ttopic1\n+\t^topic2\n+\t^topic3\n+\tEOF\n+\tgit rev-parse\t\t\t\\\n+\t\ttopic1\t\t\t\\\n+\t\ttopic1^{tree}\t\t\\\n+\t\ttopic1:f1\t\t\\\n+\t\ttopic1:f1/f1\t\t\\\n+\t\ttopic1:f1/f1/data.txt | sort >expect_objects.txt\n+'\n+\n+test_expect_success 'non-sparse pack-objects' '\n+\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n+\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n+\ttest_cmp expect_objects.txt nonsparse_objects.txt\n+'\n+\n+test_expect_success 'sparse pack-objects' '\n+\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_objects.txt sparse_objects.txt\n+'\n+\n+# Demonstrate that both algorithms send \"extra\" objects because\n+# they are not in the frontier.\n+\n+test_expect_success 'duplicate a folder from f3 and commit to topic1' '\n+\tgit checkout topic1 &&\n+\techo change-3 >f3/f3/data.txt &&\n+\tgit commit -a -m \"Changed f3/f3/data.txt\" &&\n+\tgit rev-parse\t\t\t\\\n+\t\ttopic1~1\t\t\\\n+\t\ttopic1~1^{tree}\t\t\\\n+\t\ttopic1^{tree}\t\t\\\n+\t\ttopic1\t\t\t\\\n+\t\ttopic1:f1\t\t\\\n+\t\ttopic1:f1/f1\t\t\\\n+\t\ttopic1:f1/f1/data.txt\t\\\n+\t\ttopic1:f3\t\t\\\n+\t\ttopic1:f3/f3\t\t\\\n+\t\ttopic1:f3/f3/data.txt | sort >expect_objects.txt\n+'\n+\n+test_expect_success 'non-sparse pack-objects' '\n+\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n+\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n+\ttest_cmp expect_objects.txt nonsparse_objects.txt\n+'\n+\n+test_expect_success 'sparse pack-objects' '\n+\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_objects.txt sparse_objects.txt\n+'\n+\n+test_expect_success 'duplicate a folder from f1 into f3' '\n+\tmkdir f3/f4 &&\n+\tcp -r f1/f1/* f3/f4 &&\n+\tgit add f3/f4 &&\n+\tgit commit -m \"Copied f1/f1 to f3/f4\" &&\n+\tcat >packinput.txt <<-EOF &&\n+\ttopic1\n+\t^topic1~1\n+\tEOF\n+\tgit rev-parse\t\t\\\n+\t\ttopic1\t\t\\\n+\t\ttopic1^{tree}\t\\\n+\t\ttopic1:f3 | sort >expect_objects.txt\n+'\n+\n+test_expect_success 'non-sparse pack-objects' '\n+\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n+\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n+\ttest_cmp expect_objects.txt nonsparse_objects.txt\n+'\n+\n+test_expect_success 'sparse pack-objects' '\n+\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_objects.txt sparse_objects.txt\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"364257","messageId":"bbc3f7818293f5bc4db30ed2b4c60fd67ae2ca1f.1543441960.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.git.gitgitgadget@gmail.com","subject":"[PATCH 5/5] pack-objects: create pack.useSparse setting","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-11-28T21:52:48Z","receivedAt":"2018-11-28T21:52:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nThe '--sparse' flag in 'git pack-objects' changes the algorithm\nused to enumerate objects to one that is faster for individual\nusers pushing new objects that change only a small cone of the\nworking directory. The sparse algorithm is not recommended for a\nserver, which likely sends new objects that appear across the\nentire working directory.\n\nCreate a 'pack.useSparse' setting that enables this new algorithm.\nThis allows 'git push' to use this algorithm without passing a\n'--sparse' flag all the way through four levels of run_command()\ncalls.\n\nIf the '--no-sparse' flag is set, then this config setting is\noverridden.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n builtin/pack-objects.c         |  4 ++++\n t/t5322-pack-objects-sparse.sh | 15 +++++++++++++++\n 2 files changed, 19 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7d5b0735e3..124b1bafc4 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2711,6 +2711,10 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(k, \"pack.usesparse\")) {\n+\t\tsparse = git_config_bool(k, v);\n+\t\treturn 0;\n+\t}\n \tif (!strcmp(k, \"pack.threads\")) {\n \t\tdelta_search_threads = git_config_int(k, v);\n \t\tif (delta_search_threads < 0)\ndiff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\nindex 45dba6e014..8f5699bd91 100755\n--- a/t/t5322-pack-objects-sparse.sh\n+++ b/t/t5322-pack-objects-sparse.sh\n@@ -121,4 +121,19 @@ test_expect_success 'sparse pack-objects' '\n \ttest_cmp expect_sparse_objects.txt sparse_objects.txt\n '\n \n+test_expect_success 'pack.useSparse enables algorithm' '\n+\tgit config pack.useSparse true &&\n+\tgit pack-objects --stdout --revs <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_sparse_objects.txt sparse_objects.txt\n+'\n+\n+test_expect_success 'pack.useSparse overridden' '\n+\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_objects.txt sparse_objects.txt\n+'\n+\n test_done\n-- \ngitgitgadget\n"},{"id":"364258","messageId":"0725aac4bbf3d6881af483bc64e3499a59bcf1a9.1543441960.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.git.gitgitgadget@gmail.com","subject":"[PATCH 4/5] revision: implement sparse algorithm","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-11-28T21:52:46Z","receivedAt":"2018-11-28T21:52:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen enumerating objects to place in a pack-file during 'git\npack-objects --revs', we discover the \"frontier\" of commits\nthat we care about and the boundary with commit we find\nuninteresting. From that point, we walk trees to discover which\ntrees and blobs are uninteresting. Finally, we walk trees to find\nthe interesting trees.\n\nThis commit introduces a new, \"sparse\" way to discover the\nuninteresting trees. We use the perspective of a single user trying\nto push their topic to a large repository. That user likely changed\na very small fraction of the paths in their working directory, but\nwe spend a lot of time walking all reachable trees.\n\nThe way to switch the logic to work in this sparse way is to start\ncaring about which paths introduce new trees. While it is not\npossible to generate a diff between the frontier boundary and all\nof the interesting commits, we can simulate that behavior by\ninspecting all of the root trees as a whole, then recursing down\nto the set of trees at each path.\n\nWe already had taken the first step by passing an oidset to\nmark_trees_uninteresting_sparse(). We now create a dictionary\nwhose keys are paths and values are oidsets. We consider the set\nof trees that appear at each path. While we inspect a tree, we\nadd its subtrees to the oidsets corresponding to the tree entry's\npath. We also mark trees as UNINTERESTING if the tree we are\nparsing is UNINTERESTING.\n\nTo actually improve the peformance, we need to terminate our\nrecursion unless the oidset contains some intersting trees and\nsome uninteresting trees. Technically, we only need one interesting\ntree for this to speed up in most cases, but we also will not mark\nanything UNINTERESTING if there are no uninteresting trees, so\nthat would be wasted effort.\n\nThere are a few ways that this is not a universally better option.\n\nFirst, we can pack extra objects. If someone copies a subtree\nfrom one tree to another, the first tree will appear UNINTERESTING\nand we will not recurse to see that the subtree should also be\nUNINTERESTING. We will walk the new tree and see the subtree as\na \"new\" object and add it to the pack. We add a test case that\ndemonstrates this as a way to prove that the --sparse option is\nactually working.\n\nSecond, we can have extra memory pressure. If instead of being a\nsingle user pushing a small topic we are a server sending new\nobjects from across the entire working directory, then we will\ngain very little (the recursion will rarely terminate early) but\nwill spend extra time maintaining the path-oidset dictionaries.\n\nDespite these potential drawbacks, the benefits of the algorithm\nare clear. By adding a counter to 'add_children_by_path' and\n'mark_tree_contents_uninteresting', I measured the number of\nparsed trees for the two algorithms in a variety of repos.\n\nFor git.git, I used the following input:\n\n\tv2.19.0\n\t^v2.19.0~10\n\n Objects to pack: 550\nWalked (old alg): 282\nWalked (new alg): 130\n\nFor the Linux repo, I used the following input:\n\n\tv4.18\n\t^v4.18~10\n\n Objects to pack:   518\nWalked (old alg): 4,836\nWalked (new alg):   188\n\nThe two repos above are rather \"wide and flat\" compared to\nother repos that I have used in the past. As a comparison,\nI tested an old topic branch in the Azure DevOps repo, which\nhas a much deeper folder structure than the Linux repo.\n\n Objects to pack:    220\nWalked (old alg): 22,804\nWalked (new alg):    129\n\nI used the number of walked trees the main metric above because\nit is consistent across multiple runs. When I ran my tests, the\nperformance of the pack-objects command with the same options\ncould change the end-to-end time by 10x depending on the file\nsystem being warm. However, by repeating the same test on repeat\nI could get more consistent timing results. The git.git and\nLinux tests were too fast overall (less than 0.5s) to measure\nan end-to-end difference. The Azure DevOps case was slow enough\nto see the time improve from 15s to 1s in the warm case. The\ncold case was 90s to 9s in my testing.\n\nThese improvements will have even larger benefits in the super-\nlarge Windows repository. In our experiments, we see the\n\"Enumerate objects\" phase of pack-objects taking 60-80% of the\nend-to-end time of non-trivial pushes, taking longer than the\nnetwork time to send the pack and the server time to verify the\npack.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n revision.c                     | 111 ++++++++++++++++++++++++++++++---\n t/t5322-pack-objects-sparse.sh |  21 +++++--\n 2 files changed, 116 insertions(+), 16 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex 3a62c7c187..7e4bfe621a 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -99,26 +99,117 @@ void mark_tree_uninteresting(struct repository *r, struct tree *tree)\n \tmark_tree_contents_uninteresting(r, tree);\n }\n \n+struct paths_and_oids {\n+\tstruct string_list list;\n+};\n+\n+static void paths_and_oids_init(struct paths_and_oids *po)\n+{\n+\tstring_list_init(&po->list, 1);\n+}\n+\n+static void paths_and_oids_clear(struct paths_and_oids *po)\n+{\n+\tint i;\n+\tfor (i = 0; i < po->list.nr; i++) {\n+\t\toidset_clear(po->list.items[i].util);\n+\t\tfree(po->list.items[i].util);\n+\t}\n+\n+\tstring_list_clear(&po->list, 0);\n+}\n+\n+static void paths_and_oids_insert(struct paths_and_oids *po,\n+\t\t\t\t  const char *path,\n+\t\t\t\t  const struct object_id *oid)\n+{\n+\tstruct string_list_item *item = string_list_insert(&po->list, path);\n+\tstruct oidset *set;\n+\n+\tif (!item->util) {\n+\t\tset = xcalloc(1, sizeof(struct oidset));\n+\t\toidset_init(set, 16);\n+\t\titem->util = set;\n+\t} else {\n+\t\tset = item->util;\n+\t}\n+\n+\toidset_insert(set, oid);\n+}\n+\n+static void add_children_by_path(struct repository *r,\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct paths_and_oids *po)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\n+\tif (parse_tree_gently(tree, 1) < 0)\n+\t\treturn;\n+\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tswitch (object_type(entry.mode)) {\n+\t\tcase OBJ_TREE:\n+\t\t\tpaths_and_oids_insert(po, entry.path, entry.oid);\n+\n+\t\t\tif (tree->object.flags & UNINTERESTING) {\n+\t\t\t\tstruct tree *child = lookup_tree(r, entry.oid);\n+\t\t\t\tchild->object.flags |= UNINTERESTING;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\tcase OBJ_BLOB:\n+\t\t\tif (tree->object.flags & UNINTERESTING) {\n+\t\t\t\tstruct blob *child = lookup_blob(r, entry.oid);\n+\t\t\t\tchild->object.flags |= UNINTERESTING;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\t/* Subproject commit - not in this repository */\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfree_tree_buffer(tree);\n+}\n+\n void mark_trees_uninteresting_sparse(struct repository *r,\n \t\t\t\t     struct oidset *set)\n {\n+\tint i;\n+\tunsigned has_interesting = 0, has_uninteresting = 0;\n+\tstruct paths_and_oids po;\n \tstruct object_id *oid;\n \tstruct oidset_iter iter;\n \n \toidset_iter_init(set, &iter);\n-\twhile ((oid = oidset_iter_next(&iter))) {\n+\twhile ((!has_interesting || !has_uninteresting) &&\n+\t       (oid = oidset_iter_next(&iter))) {\n \t\tstruct tree *tree = lookup_tree(r, oid);\n \n-\t\tif (tree->object.flags & UNINTERESTING) {\n-\t\t\t/*\n-\t\t\t * Remove the flag so the next call\n-\t\t\t * is not a no-op. The flag is added\n-\t\t\t * in mark_tree_unintersting().\n-\t\t\t */\n-\t\t\ttree->object.flags ^= UNINTERESTING;\n-\t\t\tmark_tree_uninteresting(r, tree);\n-\t\t}\n+\t\tif (tree->object.flags & UNINTERESTING)\n+\t\t\thas_uninteresting = 1;\n+\t\telse\n+\t\t\thas_interesting = 1;\n \t}\n+\n+\t/* Do not walk unless we have both types of trees. */\n+\tif (!has_uninteresting || !has_interesting)\n+\t\treturn;\n+\n+\tpaths_and_oids_init(&po);\n+\n+\toidset_iter_init(set, &iter);\n+\twhile ((oid = oidset_iter_next(&iter))) {\n+\t\tstruct tree *tree = lookup_tree(r, oid);\n+\t\tadd_children_by_path(r, tree, &po);\n+\t}\n+\n+\tfor (i = 0; i < po.list.nr; i++)\n+\t\tmark_trees_uninteresting_sparse(\n+\t\t\tr, (struct oidset *)po.list.items[i].util);\n+\n+\tpaths_and_oids_clear(&po);\n }\n \n struct commit_stack {\ndiff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\nindex 81f6805bc3..45dba6e014 100755\n--- a/t/t5322-pack-objects-sparse.sh\n+++ b/t/t5322-pack-objects-sparse.sh\n@@ -83,22 +83,25 @@ test_expect_success 'sparse pack-objects' '\n \ttest_cmp expect_objects.txt sparse_objects.txt\n '\n \n+# Demonstrate that the algorithms differ when we copy a tree wholesale\n+# from one folder to another.\n+\n test_expect_success 'duplicate a folder from f1 into f3' '\n \tmkdir f3/f4 &&\n \tcp -r f1/f1/* f3/f4 &&\n \tgit add f3/f4 &&\n \tgit commit -m \"Copied f1/f1 to f3/f4\" &&\n-\tcat >packinput.txt <<-EOF &&\n+\tcat >packinput.txt <<-EOF\n \ttopic1\n \t^topic1~1\n \tEOF\n-\tgit rev-parse\t\t\\\n-\t\ttopic1\t\t\\\n-\t\ttopic1^{tree}\t\\\n-\t\ttopic1:f3 | sort >expect_objects.txt\n '\n \n test_expect_success 'non-sparse pack-objects' '\n+\tgit rev-parse\t\t\t\\\n+\t\ttopic1\t\t\t\\\n+\t\ttopic1^{tree}\t\t\\\n+\t\ttopic1:f3 | sort >expect_objects.txt &&\n \tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n \tgit index-pack -o nonsparse.idx nonsparse.pack &&\n \tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n@@ -106,10 +109,16 @@ test_expect_success 'non-sparse pack-objects' '\n '\n \n test_expect_success 'sparse pack-objects' '\n+\tgit rev-parse\t\t\t\\\n+\t\ttopic1\t\t\t\\\n+\t\ttopic1^{tree}\t\t\\\n+\t\ttopic1:f3\t\t\\\n+\t\ttopic1:f3/f4\t\t\\\n+\t\ttopic1:f3/f4/data.txt | sort >expect_sparse_objects.txt &&\n \tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n \tgit index-pack -o sparse.idx sparse.pack &&\n \tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n-\ttest_cmp expect_objects.txt sparse_objects.txt\n+\ttest_cmp expect_sparse_objects.txt sparse_objects.txt\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"364260","messageId":"CAGZ79kbVGyX=Ky9HvmUGWPoHdLJYqnPTNda2kT3aC8XuHnH3+A@mail.gmail.com","threadId":"49914","inReplyTo":"9d6b8f6d0602e85652b2a748c58eeed4cbf4359e.1543441960.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/5] pack-objects: add --sparse option","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-11-28T22:11:18Z","receivedAt":"2018-11-28T22:11:32Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> +--sparse::\n> +       Use the \"sparse\" algorithm to determine which objects to include in\n> +       the pack. This can have significant performance benefits when computing\n> +       a pack to send a small change. However, it is possible that extra\n> +       objects are added to the pack-file if the included commits contain\n> +       certain types of direct renames.\n\nAs a user, where do I find a discussion of these walking algorithms?\n(i.e. how can I tell if I can really expect to gain performance benefits,\nwhat are the tradeoffs? \"If it were strictly better than the original,\nit would be on by default\" might be a thought of a user)\n"},{"id":"364261","messageId":"874lc0zw0p.fsf@evledraar.gmail.com","threadId":"49914","inReplyTo":"pull.89.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/5] Add a new \"sparse\" tree walk algorithm","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-28T22:18:14Z","receivedAt":"2018-11-28T22:18:24Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Nov 28 2018, Derrick Stolee via GitGitGadget wrote:\n\n> One of the biggest remaining pain points for users of very large\n> repositories is the time it takes to run 'git push'. We inspected some slow\n> pushes by our developers and found that the \"Enumerating Objects\" phase of a\n> push was very slow. This is unsurprising, because this is why reachability\n> bitmaps exist. However, reachability bitmaps are not available to us because\n> of the single pack-file requirement. The bitmap approach is intended for\n> servers anyway, and clients have a much different behavior pattern.\n>\n> Specifically, clients are normally pushing a very small number of objects\n> compared to the entire working directory. A typical user changes only a\n> small cone of the working directory, so let's use that to our benefit.\n>\n> Create a new \"sparse\" mode for 'git pack-objects' that uses the paths that\n> introduce new objects to direct our search into the reachable trees. By\n> collecting trees at each path, we can then recurse into a path only when\n> there are uninteresting and interesting trees at that path. This gains a\n> significant performance boost for small topics while presenting a\n> possibility of packing extra objects.\n>\n> The main algorithm change is in patch 4, but is set up a little bit in\n> patches 1 and 2.\n>\n> As demonstrated in the included test script, we see that the existing\n> algorithm can send extra objects due to the way we specify the \"frontier\".\n> But we can send even more objects if a user copies objects from one folder\n> to another. I say \"copy\" because a rename would (usually) change the\n> original folder and trigger a walk into that path, discovering the objects.\n>\n> In order to benefit from this approach, the user can opt-in using the\n> pack.useSparse config setting. This setting can be overridden using the\n> '--no-sparse' option.\n\nThis is really interesting. I tested this with:\n\n    diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n    index 124b1bafc4..5c7615f06c 100644\n    --- a/builtin/pack-objects.c\n    +++ b/builtin/pack-objects.c\n    @@ -3143 +3143 @@ static void get_object_list(int ac, const char **av)\n    -       mark_edges_uninteresting(&revs, show_edge, sparse);\n    +       mark_edges_uninteresting(&revs, show_edge, 1);\n\nTo emulate having a GIT_TEST_* mode for this, which seems like a good\nidea since it turned up a lot of segfaults in pack-objects. I wasn't\nable to get a backtrace for that since it always happens indirectly, and\nI didn't dig enough to see how to manually invoke it the right way.\n"},{"id":"364282","messageId":"d739ad00-d4f7-2a1b-7e03-3b69d74bdb5f@gmail.com","threadId":"49914","inReplyTo":"874lc0zw0p.fsf@evledraar.gmail.com","subject":"Re: [PATCH 0/5] Add a new \"sparse\" tree walk algorithm","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-29T04:05:55Z","receivedAt":"2018-11-29T04:06:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/28/2018 5:18 PM, Ævar Arnfjörð Bjarmason wrote:\n> This is really interesting. I tested this with:\n>\n>      diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n>      index 124b1bafc4..5c7615f06c 100644\n>      --- a/builtin/pack-objects.c\n>      +++ b/builtin/pack-objects.c\n>      @@ -3143 +3143 @@ static void get_object_list(int ac, const char **av)\n>      -       mark_edges_uninteresting(&revs, show_edge, sparse);\n>      +       mark_edges_uninteresting(&revs, show_edge, 1);\n>\n> To emulate having a GIT_TEST_* mode for this, which seems like a good\n> idea since it turned up a lot of segfaults in pack-objects. I wasn't\n> able to get a backtrace for that since it always happens indirectly, and\n> I didn't dig enough to see how to manually invoke it the right way.\n\nThanks for double-checking this. I had run a similar test in my \nprototype implementation, but over-simplified some code when rewriting \nit for submission (and then forgot to re-run that test).\n\nSpecifically, these null checks are important:\n\ndiff --git a/list-objects.c b/list-objects.c\nindex 9bb93d1640..7e864b4db8 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -232,6 +232,10 @@ static void add_edge_parents(struct commit *commit,\n         for (parents = commit->parents; parents; parents = parents->next) {\n                 struct commit *parent = parents->item;\n                 struct tree *tree = get_commit_tree(parent);\n+\n+               if (!tree)\n+                       continue;\n+\n                 oidset_insert(set, &tree->object.oid);\n\n                 if (!(parent->object.flags & UNINTERESTING))\n@@ -261,6 +265,8 @@ void mark_edges_uninteresting(struct rev_info *revs,\n\n                 if (sparse) {\n                         struct tree *tree = get_commit_tree(commit);\n+                       if (!tree)\n+                               continue;\n\n                         if (commit->object.flags & UNINTERESTING)\n                                 tree->object.flags |= UNINTERESTING;\n\nI will definitely include a GIT_TEST_* variable in v2.\n\nThanks,\n\n-Stolee\n\n"},{"id":"364319","messageId":"1e9d4d2d-561e-fcbc-48cf-374dcb9ce009@gmail.com","threadId":"49914","inReplyTo":"CAGZ79kbVGyX=Ky9HvmUGWPoHdLJYqnPTNda2kT3aC8XuHnH3+A@mail.gmail.com","subject":"Re: [PATCH 3/5] pack-objects: add --sparse option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-29T14:20:29Z","receivedAt":"2018-11-29T14:20:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/28/2018 5:11 PM, Stefan Beller wrote:\n>> +--sparse::\n>> +       Use the \"sparse\" algorithm to determine which objects to include in\n>> +       the pack. This can have significant performance benefits when computing\n>> +       a pack to send a small change. However, it is possible that extra\n>> +       objects are added to the pack-file if the included commits contain\n>> +       certain types of direct renames.\n> As a user, where do I find a discussion of these walking algorithms?\n> (i.e. how can I tell if I can really expect to gain performance benefits,\n> what are the tradeoffs? \"If it were strictly better than the original,\n> it would be on by default\" might be a thought of a user)\n\nYou're right that having this hidden as an opt-in config variable makes \nit hard to discover as a typical user.\n\nI would argue that we should actually make the config setting true by \ndefault, and recommend that servers opt-out. Here are my reasons:\n\n1. The vast majority of users are clients.\n\n2. Client users are not likely to know about and tweak these settings.\n\n3. Server users are more likely to keep an eye on the different knobs \nthey can tweak.\n\n4. Servers should use the reachability bitmaps, which don't use this \nlogic anyway.\n\nWhile _eventually_ we should make this opt-out, we shouldn't do that \nuntil it has cooked a while.\n\nThanks,\n-Stolee\n"},{"id":"364320","messageId":"9644f6ff0446e79c6840ccd5deba9ef6895ef82b.1543501438.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 3/6] pack-objects: add --sparse option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-11-29T14:24:04Z","receivedAt":"2018-11-29T14:24:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nAdd a '--sparse' option flag to the pack-objects builtin. This\nallows the user to specify that they want to use the new logic\nfor walking trees. This logic currently does not differ from the\nexisting output, but will in a later change.\n\nCreate a new test script, t5322-pack-objects-sparse.sh, to ensure\nthe object list that is selected matches what we expect. When we\nupdate the logic to walk in a sparse fashion, the final test will\nbe updated to show the extra objects that are added.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n Documentation/git-pack-objects.txt |   9 ++-\n builtin/pack-objects.c             |   5 +-\n t/t5322-pack-objects-sparse.sh     | 115 +++++++++++++++++++++++++++++\n 3 files changed, 127 insertions(+), 2 deletions(-)\n create mode 100755 t/t5322-pack-objects-sparse.sh\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 40c825c381..ced2630eb3 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -14,7 +14,7 @@ SYNOPSIS\n \t[--local] [--incremental] [--window=<n>] [--depth=<n>]\n \t[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\n \t[--stdout [--filter=<filter-spec>] | base-name]\n-\t[--shallow] [--keep-true-parents] < object-list\n+\t[--shallow] [--keep-true-parents] [--sparse] < object-list\n \n \n DESCRIPTION\n@@ -196,6 +196,13 @@ depth is 4095.\n \tAdd --no-reuse-object if you want to force a uniform compression\n \tlevel on all data no matter the source.\n \n+--sparse::\n+\tUse the \"sparse\" algorithm to determine which objects to include in\n+\tthe pack. This can have significant performance benefits when computing\n+\ta pack to send a small change. However, it is possible that extra\n+\tobjects are added to the pack-file if the included commits contain\n+\tcertain types of direct renames.\n+\n --thin::\n \tCreate a \"thin\" pack by omitting the common objects between a\n \tsender and a receiver in order to reduce network transfer. This\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5f70d840a7..7d5b0735e3 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -84,6 +84,7 @@ static unsigned long pack_size_limit;\n static int depth = 50;\n static int delta_search_threads;\n static int pack_to_stdout;\n+static int sparse;\n static int thin;\n static int num_preferred_base;\n static struct progress *progress_state;\n@@ -3135,7 +3136,7 @@ static void get_object_list(int ac, const char **av)\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n-\tmark_edges_uninteresting(&revs, show_edge, 0);\n+\tmark_edges_uninteresting(&revs, show_edge, sparse);\n \n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\n@@ -3292,6 +3293,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t{ OPTION_CALLBACK, 0, \"unpack-unreachable\", NULL, N_(\"time\"),\n \t\t  N_(\"unpack unreachable objects newer than <time>\"),\n \t\t  PARSE_OPT_OPTARG, option_parse_unpack_unreachable },\n+\t\tOPT_BOOL(0, \"sparse\", &sparse,\n+\t\t\t N_(\"use the sparse reachability algorithm\")),\n \t\tOPT_BOOL(0, \"thin\", &thin,\n \t\t\t N_(\"create thin packs\")),\n \t\tOPT_BOOL(0, \"shallow\", &shallow,\ndiff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\nnew file mode 100755\nindex 0000000000..81f6805bc3\n--- /dev/null\n+++ b/t/t5322-pack-objects-sparse.sh\n@@ -0,0 +1,115 @@\n+#!/bin/sh\n+\n+test_description='pack-objects object selection using sparse algorithm'\n+. ./test-lib.sh\n+\n+test_expect_success 'setup repo' '\n+\ttest_commit initial &&\n+\tfor i in $(test_seq 1 3)\n+\tdo\n+\t\tmkdir f$i &&\n+\t\tfor j in $(test_seq 1 3)\n+\t\tdo\n+\t\t\tmkdir f$i/f$j &&\n+\t\t\techo $j >f$i/f$j/data.txt\n+\t\tdone\n+\tdone &&\n+\tgit add . &&\n+\tgit commit -m \"Initialized trees\" &&\n+\tfor i in $(test_seq 1 3)\n+\tdo\n+\t\tgit checkout -b topic$i master &&\n+\t\techo change-$i >f$i/f$i/data.txt &&\n+\t\tgit commit -a -m \"Changed f$i/f$i/data.txt\"\n+\tdone &&\n+\tcat >packinput.txt <<-EOF &&\n+\ttopic1\n+\t^topic2\n+\t^topic3\n+\tEOF\n+\tgit rev-parse\t\t\t\\\n+\t\ttopic1\t\t\t\\\n+\t\ttopic1^{tree}\t\t\\\n+\t\ttopic1:f1\t\t\\\n+\t\ttopic1:f1/f1\t\t\\\n+\t\ttopic1:f1/f1/data.txt | sort >expect_objects.txt\n+'\n+\n+test_expect_success 'non-sparse pack-objects' '\n+\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n+\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n+\ttest_cmp expect_objects.txt nonsparse_objects.txt\n+'\n+\n+test_expect_success 'sparse pack-objects' '\n+\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_objects.txt sparse_objects.txt\n+'\n+\n+# Demonstrate that both algorithms send \"extra\" objects because\n+# they are not in the frontier.\n+\n+test_expect_success 'duplicate a folder from f3 and commit to topic1' '\n+\tgit checkout topic1 &&\n+\techo change-3 >f3/f3/data.txt &&\n+\tgit commit -a -m \"Changed f3/f3/data.txt\" &&\n+\tgit rev-parse\t\t\t\\\n+\t\ttopic1~1\t\t\\\n+\t\ttopic1~1^{tree}\t\t\\\n+\t\ttopic1^{tree}\t\t\\\n+\t\ttopic1\t\t\t\\\n+\t\ttopic1:f1\t\t\\\n+\t\ttopic1:f1/f1\t\t\\\n+\t\ttopic1:f1/f1/data.txt\t\\\n+\t\ttopic1:f3\t\t\\\n+\t\ttopic1:f3/f3\t\t\\\n+\t\ttopic1:f3/f3/data.txt | sort >expect_objects.txt\n+'\n+\n+test_expect_success 'non-sparse pack-objects' '\n+\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n+\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n+\ttest_cmp expect_objects.txt nonsparse_objects.txt\n+'\n+\n+test_expect_success 'sparse pack-objects' '\n+\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_objects.txt sparse_objects.txt\n+'\n+\n+test_expect_success 'duplicate a folder from f1 into f3' '\n+\tmkdir f3/f4 &&\n+\tcp -r f1/f1/* f3/f4 &&\n+\tgit add f3/f4 &&\n+\tgit commit -m \"Copied f1/f1 to f3/f4\" &&\n+\tcat >packinput.txt <<-EOF &&\n+\ttopic1\n+\t^topic1~1\n+\tEOF\n+\tgit rev-parse\t\t\\\n+\t\ttopic1\t\t\\\n+\t\ttopic1^{tree}\t\\\n+\t\ttopic1:f3 | sort >expect_objects.txt\n+'\n+\n+test_expect_success 'non-sparse pack-objects' '\n+\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n+\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n+\ttest_cmp expect_objects.txt nonsparse_objects.txt\n+'\n+\n+test_expect_success 'sparse pack-objects' '\n+\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_objects.txt sparse_objects.txt\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"364321","messageId":"c99957d06f7fa47927a1b57060d95678f2a18125.1543501438.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 4/6] revision: implement sparse algorithm","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-11-29T14:24:06Z","receivedAt":"2018-11-29T14:24:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen enumerating objects to place in a pack-file during 'git\npack-objects --revs', we discover the \"frontier\" of commits\nthat we care about and the boundary with commit we find\nuninteresting. From that point, we walk trees to discover which\ntrees and blobs are uninteresting. Finally, we walk trees to find\nthe interesting trees.\n\nThis commit introduces a new, \"sparse\" way to discover the\nuninteresting trees. We use the perspective of a single user trying\nto push their topic to a large repository. That user likely changed\na very small fraction of the paths in their working directory, but\nwe spend a lot of time walking all reachable trees.\n\nThe way to switch the logic to work in this sparse way is to start\ncaring about which paths introduce new trees. While it is not\npossible to generate a diff between the frontier boundary and all\nof the interesting commits, we can simulate that behavior by\ninspecting all of the root trees as a whole, then recursing down\nto the set of trees at each path.\n\nWe already had taken the first step by passing an oidset to\nmark_trees_uninteresting_sparse(). We now create a dictionary\nwhose keys are paths and values are oidsets. We consider the set\nof trees that appear at each path. While we inspect a tree, we\nadd its subtrees to the oidsets corresponding to the tree entry's\npath. We also mark trees as UNINTERESTING if the tree we are\nparsing is UNINTERESTING.\n\nTo actually improve the peformance, we need to terminate our\nrecursion unless the oidset contains some intersting trees and\nsome uninteresting trees. Technically, we only need one interesting\ntree for this to speed up in most cases, but we also will not mark\nanything UNINTERESTING if there are no uninteresting trees, so\nthat would be wasted effort.\n\nThere are a few ways that this is not a universally better option.\n\nFirst, we can pack extra objects. If someone copies a subtree\nfrom one tree to another, the first tree will appear UNINTERESTING\nand we will not recurse to see that the subtree should also be\nUNINTERESTING. We will walk the new tree and see the subtree as\na \"new\" object and add it to the pack. We add a test case that\ndemonstrates this as a way to prove that the --sparse option is\nactually working.\n\nSecond, we can have extra memory pressure. If instead of being a\nsingle user pushing a small topic we are a server sending new\nobjects from across the entire working directory, then we will\ngain very little (the recursion will rarely terminate early) but\nwill spend extra time maintaining the path-oidset dictionaries.\n\nDespite these potential drawbacks, the benefits of the algorithm\nare clear. By adding a counter to 'add_children_by_path' and\n'mark_tree_contents_uninteresting', I measured the number of\nparsed trees for the two algorithms in a variety of repos.\n\nFor git.git, I used the following input:\n\n\tv2.19.0\n\t^v2.19.0~10\n\n Objects to pack: 550\nWalked (old alg): 282\nWalked (new alg): 130\n\nFor the Linux repo, I used the following input:\n\n\tv4.18\n\t^v4.18~10\n\n Objects to pack:   518\nWalked (old alg): 4,836\nWalked (new alg):   188\n\nThe two repos above are rather \"wide and flat\" compared to\nother repos that I have used in the past. As a comparison,\nI tested an old topic branch in the Azure DevOps repo, which\nhas a much deeper folder structure than the Linux repo.\n\n Objects to pack:    220\nWalked (old alg): 22,804\nWalked (new alg):    129\n\nI used the number of walked trees the main metric above because\nit is consistent across multiple runs. When I ran my tests, the\nperformance of the pack-objects command with the same options\ncould change the end-to-end time by 10x depending on the file\nsystem being warm. However, by repeating the same test on repeat\nI could get more consistent timing results. The git.git and\nLinux tests were too fast overall (less than 0.5s) to measure\nan end-to-end difference. The Azure DevOps case was slow enough\nto see the time improve from 15s to 1s in the warm case. The\ncold case was 90s to 9s in my testing.\n\nThese improvements will have even larger benefits in the super-\nlarge Windows repository. In our experiments, we see the\n\"Enumerate objects\" phase of pack-objects taking 60-80% of the\nend-to-end time of non-trivial pushes, taking longer than the\nnetwork time to send the pack and the server time to verify the\npack.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n revision.c                     | 116 ++++++++++++++++++++++++++++++---\n t/t5322-pack-objects-sparse.sh |  21 ++++--\n 2 files changed, 121 insertions(+), 16 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex f9eb6400f1..971f1bb095 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -99,29 +99,125 @@ void mark_tree_uninteresting(struct repository *r, struct tree *tree)\n \tmark_tree_contents_uninteresting(r, tree);\n }\n \n+struct paths_and_oids {\n+\tstruct string_list list;\n+};\n+\n+static void paths_and_oids_init(struct paths_and_oids *po)\n+{\n+\tstring_list_init(&po->list, 1);\n+}\n+\n+static void paths_and_oids_clear(struct paths_and_oids *po)\n+{\n+\tint i;\n+\tfor (i = 0; i < po->list.nr; i++) {\n+\t\toidset_clear(po->list.items[i].util);\n+\t\tfree(po->list.items[i].util);\n+\t}\n+\n+\tstring_list_clear(&po->list, 0);\n+}\n+\n+static void paths_and_oids_insert(struct paths_and_oids *po,\n+\t\t\t\t  const char *path,\n+\t\t\t\t  const struct object_id *oid)\n+{\n+\tstruct string_list_item *item = string_list_insert(&po->list, path);\n+\tstruct oidset *set;\n+\n+\tif (!item->util) {\n+\t\tset = xcalloc(1, sizeof(struct oidset));\n+\t\toidset_init(set, 16);\n+\t\titem->util = set;\n+\t} else {\n+\t\tset = item->util;\n+\t}\n+\n+\toidset_insert(set, oid);\n+}\n+\n+static void add_children_by_path(struct repository *r,\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct paths_and_oids *po)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\n+\tif (!tree)\n+\t\treturn;\n+\n+\tif (parse_tree_gently(tree, 1) < 0)\n+\t\treturn;\n+\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tswitch (object_type(entry.mode)) {\n+\t\tcase OBJ_TREE:\n+\t\t\tpaths_and_oids_insert(po, entry.path, entry.oid);\n+\n+\t\t\tif (tree->object.flags & UNINTERESTING) {\n+\t\t\t\tstruct tree *child = lookup_tree(r, entry.oid);\n+\t\t\t\tif (child)\n+\t\t\t\t\tchild->object.flags |= UNINTERESTING;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\tcase OBJ_BLOB:\n+\t\t\tif (tree->object.flags & UNINTERESTING) {\n+\t\t\t\tstruct blob *child = lookup_blob(r, entry.oid);\n+\t\t\t\tif (child)\n+\t\t\t\t\tchild->object.flags |= UNINTERESTING;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\t/* Subproject commit - not in this repository */\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfree_tree_buffer(tree);\n+}\n+\n void mark_trees_uninteresting_sparse(struct repository *r,\n \t\t\t\t     struct oidset *set)\n {\n+\tint i;\n+\tunsigned has_interesting = 0, has_uninteresting = 0;\n+\tstruct paths_and_oids po;\n \tstruct object_id *oid;\n \tstruct oidset_iter iter;\n \n \toidset_iter_init(set, &iter);\n-\twhile ((oid = oidset_iter_next(&iter))) {\n+\twhile ((!has_interesting || !has_uninteresting) &&\n+\t       (oid = oidset_iter_next(&iter))) {\n \t\tstruct tree *tree = lookup_tree(r, oid);\n \n \t\tif (!tree)\n \t\t\tcontinue;\n \n-\t\tif (tree->object.flags & UNINTERESTING) {\n-\t\t\t/*\n-\t\t\t * Remove the flag so the next call\n-\t\t\t * is not a no-op. The flag is added\n-\t\t\t * in mark_tree_unintersting().\n-\t\t\t */\n-\t\t\ttree->object.flags ^= UNINTERESTING;\n-\t\t\tmark_tree_uninteresting(r, tree);\n-\t\t}\n+\t\tif (tree->object.flags & UNINTERESTING)\n+\t\t\thas_uninteresting = 1;\n+\t\telse\n+\t\t\thas_interesting = 1;\n \t}\n+\n+\t/* Do not walk unless we have both types of trees. */\n+\tif (!has_uninteresting || !has_interesting)\n+\t\treturn;\n+\n+\tpaths_and_oids_init(&po);\n+\n+\toidset_iter_init(set, &iter);\n+\twhile ((oid = oidset_iter_next(&iter))) {\n+\t\tstruct tree *tree = lookup_tree(r, oid);\n+\t\tadd_children_by_path(r, tree, &po);\n+\t}\n+\n+\tfor (i = 0; i < po.list.nr; i++)\n+\t\tmark_trees_uninteresting_sparse(\n+\t\t\tr, (struct oidset *)po.list.items[i].util);\n+\n+\tpaths_and_oids_clear(&po);\n }\n \n struct commit_stack {\ndiff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\nindex 81f6805bc3..45dba6e014 100755\n--- a/t/t5322-pack-objects-sparse.sh\n+++ b/t/t5322-pack-objects-sparse.sh\n@@ -83,22 +83,25 @@ test_expect_success 'sparse pack-objects' '\n \ttest_cmp expect_objects.txt sparse_objects.txt\n '\n \n+# Demonstrate that the algorithms differ when we copy a tree wholesale\n+# from one folder to another.\n+\n test_expect_success 'duplicate a folder from f1 into f3' '\n \tmkdir f3/f4 &&\n \tcp -r f1/f1/* f3/f4 &&\n \tgit add f3/f4 &&\n \tgit commit -m \"Copied f1/f1 to f3/f4\" &&\n-\tcat >packinput.txt <<-EOF &&\n+\tcat >packinput.txt <<-EOF\n \ttopic1\n \t^topic1~1\n \tEOF\n-\tgit rev-parse\t\t\\\n-\t\ttopic1\t\t\\\n-\t\ttopic1^{tree}\t\\\n-\t\ttopic1:f3 | sort >expect_objects.txt\n '\n \n test_expect_success 'non-sparse pack-objects' '\n+\tgit rev-parse\t\t\t\\\n+\t\ttopic1\t\t\t\\\n+\t\ttopic1^{tree}\t\t\\\n+\t\ttopic1:f3 | sort >expect_objects.txt &&\n \tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n \tgit index-pack -o nonsparse.idx nonsparse.pack &&\n \tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n@@ -106,10 +109,16 @@ test_expect_success 'non-sparse pack-objects' '\n '\n \n test_expect_success 'sparse pack-objects' '\n+\tgit rev-parse\t\t\t\\\n+\t\ttopic1\t\t\t\\\n+\t\ttopic1^{tree}\t\t\\\n+\t\ttopic1:f3\t\t\\\n+\t\ttopic1:f3/f4\t\t\\\n+\t\ttopic1:f3/f4/data.txt | sort >expect_sparse_objects.txt &&\n \tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n \tgit index-pack -o sparse.idx sparse.pack &&\n \tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n-\ttest_cmp expect_objects.txt sparse_objects.txt\n+\ttest_cmp expect_sparse_objects.txt sparse_objects.txt\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"364322","messageId":"3d394a91367698c1c9bf7a0fccacc56446f39c74.1543501438.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 6/6] pack-objects: create GIT_TEST_PACK_SPARSE","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-11-29T14:24:08Z","receivedAt":"2018-11-29T14:24:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nCreate a test variable GIT_TEST_PACK_SPARSE to enable the sparse\nobject walk algorithm by default during the test suite. Enabling\nthis variable ensures coverage in many interesting cases, such as\nshallow clones, partial clones, and missing objects.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n builtin/pack-objects.c         | 1 +\n t/README                       | 4 ++++\n t/t5322-pack-objects-sparse.sh | 6 +++---\n 3 files changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 124b1bafc4..507d381153 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3331,6 +3331,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \n \tread_replace_refs = 0;\n \n+\tsparse = git_env_bool(\"GIT_TEST_PACK_SPARSE\", 0);\n \treset_pack_idx_option(&pack_idx_opts);\n \tgit_config(git_pack_config, NULL);\n \ndiff --git a/t/README b/t/README\nindex 28711cc508..8b6dfe1864 100644\n--- a/t/README\n+++ b/t/README\n@@ -342,6 +342,10 @@ GIT_TEST_INDEX_VERSION=<n> exercises the index read/write code path\n for the index version specified.  Can be set to any valid version\n (currently 2, 3, or 4).\n \n+GIT_TEST_PACK_SPARSE=<boolean> if enabled will default the pack-objects\n+builtin to use the sparse object walk. This can still be overridden by\n+the --no-sparse command-line argument.\n+\n GIT_TEST_PRELOAD_INDEX=<boolean> exercises the preload-index code path\n by overriding the minimum number of cache entries required per thread.\n \ndiff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\nindex 8f5699bd91..e8cf41d1c6 100755\n--- a/t/t5322-pack-objects-sparse.sh\n+++ b/t/t5322-pack-objects-sparse.sh\n@@ -36,7 +36,7 @@ test_expect_success 'setup repo' '\n '\n \n test_expect_success 'non-sparse pack-objects' '\n-\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >nonsparse.pack &&\n \tgit index-pack -o nonsparse.idx nonsparse.pack &&\n \tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n \ttest_cmp expect_objects.txt nonsparse_objects.txt\n@@ -70,7 +70,7 @@ test_expect_success 'duplicate a folder from f3 and commit to topic1' '\n '\n \n test_expect_success 'non-sparse pack-objects' '\n-\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >nonsparse.pack &&\n \tgit index-pack -o nonsparse.idx nonsparse.pack &&\n \tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n \ttest_cmp expect_objects.txt nonsparse_objects.txt\n@@ -102,7 +102,7 @@ test_expect_success 'non-sparse pack-objects' '\n \t\ttopic1\t\t\t\\\n \t\ttopic1^{tree}\t\t\\\n \t\ttopic1:f3 | sort >expect_objects.txt &&\n-\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >nonsparse.pack &&\n \tgit index-pack -o nonsparse.idx nonsparse.pack &&\n \tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n \ttest_cmp expect_objects.txt nonsparse_objects.txt\n-- \ngitgitgadget\n"},{"id":"364324","messageId":"d6912188bee43f2558b7b42395e8687073d50720.1543501438.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 5/6] pack-objects: create pack.useSparse setting","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-11-29T14:24:07Z","receivedAt":"2018-11-29T14:51:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nThe '--sparse' flag in 'git pack-objects' changes the algorithm\nused to enumerate objects to one that is faster for individual\nusers pushing new objects that change only a small cone of the\nworking directory. The sparse algorithm is not recommended for a\nserver, which likely sends new objects that appear across the\nentire working directory.\n\nCreate a 'pack.useSparse' setting that enables this new algorithm.\nThis allows 'git push' to use this algorithm without passing a\n'--sparse' flag all the way through four levels of run_command()\ncalls.\n\nIf the '--no-sparse' flag is set, then this config setting is\noverridden.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n builtin/pack-objects.c         |  4 ++++\n t/t5322-pack-objects-sparse.sh | 15 +++++++++++++++\n 2 files changed, 19 insertions(+)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7d5b0735e3..124b1bafc4 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2711,6 +2711,10 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(k, \"pack.usesparse\")) {\n+\t\tsparse = git_config_bool(k, v);\n+\t\treturn 0;\n+\t}\n \tif (!strcmp(k, \"pack.threads\")) {\n \t\tdelta_search_threads = git_config_int(k, v);\n \t\tif (delta_search_threads < 0)\ndiff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\nindex 45dba6e014..8f5699bd91 100755\n--- a/t/t5322-pack-objects-sparse.sh\n+++ b/t/t5322-pack-objects-sparse.sh\n@@ -121,4 +121,19 @@ test_expect_success 'sparse pack-objects' '\n \ttest_cmp expect_sparse_objects.txt sparse_objects.txt\n '\n \n+test_expect_success 'pack.useSparse enables algorithm' '\n+\tgit config pack.useSparse true &&\n+\tgit pack-objects --stdout --revs <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_sparse_objects.txt sparse_objects.txt\n+'\n+\n+test_expect_success 'pack.useSparse overridden' '\n+\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_objects.txt sparse_objects.txt\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"364325","messageId":"4527addacb6cb663e463859c6661918ea47768cd.1543501438.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 2/6] list-objects: consume sparse tree walk","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-11-29T14:24:03Z","receivedAt":"2018-11-29T14:51:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen creating a pack-file using 'git pack-objects --revs' we provide\na list of interesting and uninteresting commits. For example, a push\noperation would make the local topic branch be interesting and the\nknown remote refs as uninteresting. We want to discover the set of\nnew objects to send to the server as a thin pack.\n\nWe walk these commits until we discover a frontier of commits such\nthat every commit walk starting at interesting commits ends in a root\ncommit or unintersting commit. We then need to discover which\nnon-commit objects are reachable from  uninteresting commits.\n\nThe mark_edges_unintersting() method in list-objects.c iterates on\nthe commit list and does the following:\n\n* If the commit is UNINTERSTING, then mark its root tree and every\n  object it can reach as UNINTERESTING.\n\n* If the commit is interesting, then mark the root tree of every\n  UNINTERSTING parent (and all objects that tree can reach) as\n  UNINTERSTING.\n\nAt the very end, we repeat the process on every commit directly\ngiven to the revision walk from stdin. This helps ensure we properly\ncover shallow commits that otherwise were not included in the\nfrontier.\n\nThe logic to recursively follow trees is in the\nmark_tree_uninteresting() method in revision.c. The algorithm avoids\nduplicate work by not recursing into trees that are already marked\nUNINTERSTING.\n\nAdd a new 'sparse' option to the mark_edges_uninteresting() method\nthat performs this logic in a slightly new way. As we iterate over\nthe commits, we add all of the root trees to an oidset. Then, call\nmark_trees_uninteresting_sparse() on that oidset. Note that we\ninclude interesting trees in this process. The current implementation\nof mark_trees_unintersting_sparse() will walk the same trees as\nthe old logic, but this will be replaced in a later change.\n\nThe sparse option is not used by any callers at the moment, but\nwill be wired to 'git pack-objects' in the next change.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n bisect.c               |  2 +-\n builtin/pack-objects.c |  2 +-\n builtin/rev-list.c     |  2 +-\n http-push.c            |  2 +-\n list-objects.c         | 55 +++++++++++++++++++++++++++++++++++++++---\n list-objects.h         |  4 ++-\n revision.c             |  3 +++\n 7 files changed, 61 insertions(+), 9 deletions(-)\n\ndiff --git a/bisect.c b/bisect.c\nindex 487675c672..842f8b4b8f 100644\n--- a/bisect.c\n+++ b/bisect.c\n@@ -656,7 +656,7 @@ static void bisect_common(struct rev_info *revs)\n \tif (prepare_revision_walk(revs))\n \t\tdie(\"revision walk setup failed\");\n \tif (revs->tree_objects)\n-\t\tmark_edges_uninteresting(revs, NULL);\n+\t\tmark_edges_uninteresting(revs, NULL, 0);\n }\n \n static void exit_if_skipped_commits(struct commit_list *tried,\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 411aefd687..5f70d840a7 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3135,7 +3135,7 @@ static void get_object_list(int ac, const char **av)\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n-\tmark_edges_uninteresting(&revs, show_edge);\n+\tmark_edges_uninteresting(&revs, show_edge, 0);\n \n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 2880ed37e3..9663cbfae0 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -543,7 +543,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \tif (revs.tree_objects)\n-\t\tmark_edges_uninteresting(&revs, show_edge);\n+\t\tmark_edges_uninteresting(&revs, show_edge, 0);\n \n \tif (bisect_list) {\n \t\tint reaches, all;\ndiff --git a/http-push.c b/http-push.c\nindex cd48590912..ea52d6f9f6 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -1933,7 +1933,7 @@ int cmd_main(int argc, const char **argv)\n \t\tpushing = 0;\n \t\tif (prepare_revision_walk(&revs))\n \t\t\tdie(\"revision walk setup failed\");\n-\t\tmark_edges_uninteresting(&revs, NULL);\n+\t\tmark_edges_uninteresting(&revs, NULL, 0);\n \t\tobjects_to_send = get_delta(&revs, ref_lock);\n \t\tfinish_all_active_slots();\n \ndiff --git a/list-objects.c b/list-objects.c\nindex c41cc80db5..4fbdeca0a4 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -222,25 +222,72 @@ static void mark_edge_parents_uninteresting(struct commit *commit,\n \t}\n }\n \n-void mark_edges_uninteresting(struct rev_info *revs, show_edge_fn show_edge)\n+static void add_edge_parents(struct commit *commit,\n+\t\t\t     struct rev_info *revs,\n+\t\t\t     show_edge_fn show_edge,\n+\t\t\t     struct oidset *set)\n+{\n+\tstruct commit_list *parents;\n+\n+\tfor (parents = commit->parents; parents; parents = parents->next) {\n+\t\tstruct commit *parent = parents->item;\n+\t\tstruct tree *tree = get_commit_tree(parent);\n+\n+\t\tif (!tree)\n+\t\t\tcontinue;\n+\n+\t\toidset_insert(set, &tree->object.oid);\n+\n+\t\tif (!(parent->object.flags & UNINTERESTING))\n+\t\t\tcontinue;\n+\t\ttree->object.flags |= UNINTERESTING;\n+\n+\t\tif (revs->edge_hint && !(parent->object.flags & SHOWN)) {\n+\t\t\tparent->object.flags |= SHOWN;\n+\t\t\tshow_edge(parent);\n+\t\t}\n+\t}\n+}\n+\n+void mark_edges_uninteresting(struct rev_info *revs,\n+\t\t\t      show_edge_fn show_edge,\n+\t\t\t      int sparse)\n {\n \tstruct commit_list *list;\n+\tstruct oidset set;\n \tint i;\n \n+\tif (sparse)\n+\t\toidset_init(&set, 16);\n+\n \tfor (list = revs->commits; list; list = list->next) {\n \t\tstruct commit *commit = list->item;\n \n-\t\tif (commit->object.flags & UNINTERESTING) {\n+\t\tif (sparse) {\n+\t\t\tstruct tree *tree = get_commit_tree(commit);\n+\n+\t\t\tif (commit->object.flags & UNINTERESTING)\n+\t\t\t\ttree->object.flags |= UNINTERESTING;\n+\n+\t\t\toidset_insert(&set, &tree->object.oid);\n+\t\t\tadd_edge_parents(commit, revs, show_edge, &set);\n+\t\t} else if (commit->object.flags & UNINTERESTING) {\n \t\t\tmark_tree_uninteresting(revs->repo,\n \t\t\t\t\t\tget_commit_tree(commit));\n \t\t\tif (revs->edge_hint_aggressive && !(commit->object.flags & SHOWN)) {\n \t\t\t\tcommit->object.flags |= SHOWN;\n \t\t\t\tshow_edge(commit);\n \t\t\t}\n-\t\t\tcontinue;\n+\t\t} else {\n+\t\t\tmark_edge_parents_uninteresting(commit, revs, show_edge);\n \t\t}\n-\t\tmark_edge_parents_uninteresting(commit, revs, show_edge);\n \t}\n+\n+\tif (sparse) {\n+\t\tmark_trees_uninteresting_sparse(revs->repo, &set);\n+\t\toidset_clear(&set);\n+\t}\n+\n \tif (revs->edge_hint_aggressive) {\n \t\tfor (i = 0; i < revs->cmdline.nr; i++) {\n \t\t\tstruct object *obj = revs->cmdline.rev[i].item;\ndiff --git a/list-objects.h b/list-objects.h\nindex ad40762926..a952680e46 100644\n--- a/list-objects.h\n+++ b/list-objects.h\n@@ -10,7 +10,9 @@ typedef void (*show_object_fn)(struct object *, const char *, void *);\n void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);\n \n typedef void (*show_edge_fn)(struct commit *);\n-void mark_edges_uninteresting(struct rev_info *, show_edge_fn);\n+void mark_edges_uninteresting(struct rev_info *revs,\n+\t\t\t      show_edge_fn show_edge,\n+\t\t\t      int sparse);\n \n struct oidset;\n struct list_objects_filter_options;\ndiff --git a/revision.c b/revision.c\nindex 3a62c7c187..f9eb6400f1 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -109,6 +109,9 @@ void mark_trees_uninteresting_sparse(struct repository *r,\n \twhile ((oid = oidset_iter_next(&iter))) {\n \t\tstruct tree *tree = lookup_tree(r, oid);\n \n+\t\tif (!tree)\n+\t\t\tcontinue;\n+\n \t\tif (tree->object.flags & UNINTERESTING) {\n \t\t\t/*\n \t\t\t * Remove the flag so the next call\n-- \ngitgitgadget\n\n"},{"id":"364326","messageId":"60617681f7daee2a94ff23f91b09b02d58fbc3df.1543501438.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 1/6] revision: add mark_tree_uninteresting_sparse","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-11-29T14:24:01Z","receivedAt":"2018-11-29T14:51:39Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nIn preparation for a new algorithm that walks fewer trees when\ncreating a pack from a set of revisions, create a method that\ntakes an oidset of tree oids and marks reachable objects as\nUNINTERESTING.\n\nThe current implementation uses the existing\nmark_tree_uninteresting to recursively walk the trees and blobs.\nThis will walk the same number of trees as the old mechanism.\n\nThere is one new assumption in this approach: we are also given\nthe oids of the interesting trees. This implementation does not\nuse those trees at the moment, but we will use them in a later\nrewrite of this method.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n revision.c | 22 ++++++++++++++++++++++\n revision.h |  2 ++\n 2 files changed, 24 insertions(+)\n\ndiff --git a/revision.c b/revision.c\nindex 13e0519c02..3a62c7c187 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -99,6 +99,28 @@ void mark_tree_uninteresting(struct repository *r, struct tree *tree)\n \tmark_tree_contents_uninteresting(r, tree);\n }\n \n+void mark_trees_uninteresting_sparse(struct repository *r,\n+\t\t\t\t     struct oidset *set)\n+{\n+\tstruct object_id *oid;\n+\tstruct oidset_iter iter;\n+\n+\toidset_iter_init(set, &iter);\n+\twhile ((oid = oidset_iter_next(&iter))) {\n+\t\tstruct tree *tree = lookup_tree(r, oid);\n+\n+\t\tif (tree->object.flags & UNINTERESTING) {\n+\t\t\t/*\n+\t\t\t * Remove the flag so the next call\n+\t\t\t * is not a no-op. The flag is added\n+\t\t\t * in mark_tree_unintersting().\n+\t\t\t */\n+\t\t\ttree->object.flags ^= UNINTERESTING;\n+\t\t\tmark_tree_uninteresting(r, tree);\n+\t\t}\n+\t}\n+}\n+\n struct commit_stack {\n \tstruct commit **items;\n \tsize_t nr, alloc;\ndiff --git a/revision.h b/revision.h\nindex 7987bfcd2e..f828e91ae9 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -67,6 +67,7 @@ struct rev_cmdline_info {\n #define REVISION_WALK_NO_WALK_SORTED 1\n #define REVISION_WALK_NO_WALK_UNSORTED 2\n \n+struct oidset;\n struct topo_walk_info;\n \n struct rev_info {\n@@ -327,6 +328,7 @@ void put_revision_mark(const struct rev_info *revs,\n \n void mark_parents_uninteresting(struct commit *commit);\n void mark_tree_uninteresting(struct repository *r, struct tree *tree);\n+void mark_trees_uninteresting_sparse(struct repository *r, struct oidset *set);\n \n void show_object_with_name(FILE *, struct object *, const char *);\n \n-- \ngitgitgadget\n\n"},{"id":"364327","messageId":"pull.89.v2.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.git.gitgitgadget@gmail.com","subject":"[PATCH v2 0/6] Add a new \"sparse\" tree walk algorithm","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-11-29T14:24:00Z","receivedAt":"2018-11-29T14:51:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"One of the biggest remaining pain points for users of very large\nrepositories is the time it takes to run 'git push'. We inspected some slow\npushes by our developers and found that the \"Enumerating Objects\" phase of a\npush was very slow. This is unsurprising, because this is why reachability\nbitmaps exist. However, reachability bitmaps are not available to us because\nof the single pack-file requirement. The bitmap approach is intended for\nservers anyway, and clients have a much different behavior pattern.\n\nSpecifically, clients are normally pushing a very small number of objects\ncompared to the entire working directory. A typical user changes only a\nsmall cone of the working directory, so let's use that to our benefit.\n\nCreate a new \"sparse\" mode for 'git pack-objects' that uses the paths that\nintroduce new objects to direct our search into the reachable trees. By\ncollecting trees at each path, we can then recurse into a path only when\nthere are uninteresting and interesting trees at that path. This gains a\nsignificant performance boost for small topics while presenting a\npossibility of packing extra objects.\n\nThe main algorithm change is in patch 4, but is set up a little bit in\npatches 1 and 2.\n\nAs demonstrated in the included test script, we see that the existing\nalgorithm can send extra objects due to the way we specify the \"frontier\".\nBut we can send even more objects if a user copies objects from one folder\nto another. I say \"copy\" because a rename would (usually) change the\noriginal folder and trigger a walk into that path, discovering the objects.\n\nIn order to benefit from this approach, the user can opt-in using the\npack.useSparse config setting. This setting can be overridden using the\n'--no-sparse' option.\n\nUpdate in V2: \n\n * Added GIT_TEST_PACK_SPARSE test option.\n * Fixed test breakages when GIT_TEST_PACK_SPARSE is enabled by adding null\n   checks.\n\nDerrick Stolee (6):\n  revision: add mark_tree_uninteresting_sparse\n  list-objects: consume sparse tree walk\n  pack-objects: add --sparse option\n  revision: implement sparse algorithm\n  pack-objects: create pack.useSparse setting\n  pack-objects: create GIT_TEST_PACK_SPARSE\n\n Documentation/git-pack-objects.txt |   9 +-\n bisect.c                           |   2 +-\n builtin/pack-objects.c             |  10 ++-\n builtin/rev-list.c                 |   2 +-\n http-push.c                        |   2 +-\n list-objects.c                     |  55 +++++++++++-\n list-objects.h                     |   4 +-\n revision.c                         | 121 +++++++++++++++++++++++++\n revision.h                         |   2 +\n t/README                           |   4 +\n t/t5322-pack-objects-sparse.sh     | 139 +++++++++++++++++++++++++++++\n 11 files changed, 340 insertions(+), 10 deletions(-)\n create mode 100755 t/t5322-pack-objects-sparse.sh\n\n\nbase-commit: a1598010f775d82b5adf12c29d0f5bc9b41434c6\nPublished-As: https://github.com/gitgitgadget/git/releases/tags/pr-89%2Fderrickstolee%2Fpush%2Fsparse-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-89/derrickstolee/push/sparse-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/89\n\nRange-diff vs v1:\n\n 1:  b73b8de98c = 1:  60617681f7 revision: add mark_tree_uninteresting_sparse\n 2:  9bf04c748b ! 2:  4527addacb list-objects: consume sparse tree walk\n     @@ -116,6 +116,10 @@\n      +\tfor (parents = commit->parents; parents; parents = parents->next) {\n      +\t\tstruct commit *parent = parents->item;\n      +\t\tstruct tree *tree = get_commit_tree(parent);\n     ++\n     ++\t\tif (!tree)\n     ++\t\t\tcontinue;\n     ++\n      +\t\toidset_insert(set, &tree->object.oid);\n      +\n      +\t\tif (!(parent->object.flags & UNINTERESTING))\n     @@ -142,14 +146,14 @@\n      +\n       \tfor (list = revs->commits; list; list = list->next) {\n       \t\tstruct commit *commit = list->item;\n     -+\t\t\n     + \n     +-\t\tif (commit->object.flags & UNINTERESTING) {\n      +\t\tif (sparse) {\n      +\t\t\tstruct tree *tree = get_commit_tree(commit);\n     -+\t\t\t\n     ++\n      +\t\t\tif (commit->object.flags & UNINTERESTING)\n      +\t\t\t\ttree->object.flags |= UNINTERESTING;\n     - \n     --\t\tif (commit->object.flags & UNINTERESTING) {\n     ++\n      +\t\t\toidset_insert(&set, &tree->object.oid);\n      +\t\t\tadd_edge_parents(commit, revs, show_edge, &set);\n      +\t\t} else if (commit->object.flags & UNINTERESTING) {\n     @@ -189,3 +193,17 @@\n       \n       struct oidset;\n       struct list_objects_filter_options;\n     +\n     +diff --git a/revision.c b/revision.c\n     +--- a/revision.c\n     ++++ b/revision.c\n     +@@\n     + \twhile ((oid = oidset_iter_next(&iter))) {\n     + \t\tstruct tree *tree = lookup_tree(r, oid);\n     + \n     ++\t\tif (!tree)\n     ++\t\t\tcontinue;\n     ++\n     + \t\tif (tree->object.flags & UNINTERESTING) {\n     + \t\t\t/*\n     + \t\t\t * Remove the flag so the next call\n 3:  9d6b8f6d06 = 3:  9644f6ff04 pack-objects: add --sparse option\n 4:  0725aac4bb ! 4:  c99957d06f revision: implement sparse algorithm\n     @@ -157,6 +157,9 @@\n      +\tstruct tree_desc desc;\n      +\tstruct name_entry entry;\n      +\n     ++\tif (!tree)\n     ++\t\treturn;\n     ++\n      +\tif (parse_tree_gently(tree, 1) < 0)\n      +\t\treturn;\n      +\n     @@ -168,13 +171,15 @@\n      +\n      +\t\t\tif (tree->object.flags & UNINTERESTING) {\n      +\t\t\t\tstruct tree *child = lookup_tree(r, entry.oid);\n     -+\t\t\t\tchild->object.flags |= UNINTERESTING;\n     ++\t\t\t\tif (child)\n     ++\t\t\t\t\tchild->object.flags |= UNINTERESTING;\n      +\t\t\t}\n      +\t\t\tbreak;\n      +\t\tcase OBJ_BLOB:\n      +\t\t\tif (tree->object.flags & UNINTERESTING) {\n      +\t\t\t\tstruct blob *child = lookup_blob(r, entry.oid);\n     -+\t\t\t\tchild->object.flags |= UNINTERESTING;\n     ++\t\t\t\tif (child)\n     ++\t\t\t\t\tchild->object.flags |= UNINTERESTING;\n      +\t\t\t}\n      +\t\t\tbreak;\n      +\t\tdefault:\n     @@ -201,6 +206,9 @@\n      +\t       (oid = oidset_iter_next(&iter))) {\n       \t\tstruct tree *tree = lookup_tree(r, oid);\n       \n     + \t\tif (!tree)\n     + \t\t\tcontinue;\n     + \n      -\t\tif (tree->object.flags & UNINTERESTING) {\n      -\t\t\t/*\n      -\t\t\t * Remove the flag so the next call\n 5:  bbc3f78182 = 5:  d6912188be pack-objects: create pack.useSparse setting\n -:  ---------- > 6:  3d394a9136 pack-objects: create GIT_TEST_PACK_SPARSE\n\n-- \ngitgitgadget\n"},{"id":"364379","messageId":"xmqq36rjmgqf.fsf@gitster-ct.c.googlers.com","threadId":"49914","inReplyTo":"1e9d4d2d-561e-fcbc-48cf-374dcb9ce009@gmail.com","subject":"Re: [PATCH 3/5] pack-objects: add --sparse option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-11-30T02:39:04Z","receivedAt":"2018-11-30T02:39:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee <stolee@gmail.com> writes:\n\n> You're right that having this hidden as an opt-in config variable\n> makes it hard to discover as a typical user.\n>\n> I would argue that we should actually make the config setting true by\n> default, and recommend that servers opt-out. Here are my reasons:\n>\n> 1. The vast majority of users are clients.\n>\n> 2. Client users are not likely to know about and tweak these settings.\n>\n> 3. Server users are more likely to keep an eye on the different knobs\n> they can tweak.\n>\n> 4. Servers should use the reachability bitmaps, which don't use this\n> logic anyway.\n>\n> While _eventually_ we should make this opt-out, we shouldn't do that\n> until it has cooked a while.\n\nI actually do not agree.  If the knob gives enough benefit, the\nusers will learn about it viva voce, and in a few more releases\nwe'll hear \"enough users complain that they have to turn it on,\nlet's make it the default\".  If that does not happen, the knob\ndoes not deserve to be turned on in the first place.\n\nThe same applies to many shiny new toys people are discussing\nrecently on this list (e.g. precious vs expendable and non-overlay\ncheckout are the ones that immediately come to my mind).\n\nHaving said that, I won't be commenting on this shiny new toy before\nthe final.  I want to see more people help tying the loose ends and\ngive it final varnish to the upcoming release, as it seems to have\nbecome rockier and larger release than we originally anticipated.\n"},{"id":"364399","messageId":"f9421d46-9cdd-5399-d55e-d5351f149630@gmail.com","threadId":"49914","inReplyTo":"xmqq36rjmgqf.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 3/5] pack-objects: add --sparse option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-30T15:53:15Z","receivedAt":"2018-11-30T15:53:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/29/2018 9:39 PM, Junio C Hamano wrote:\n> Derrick Stolee <stolee@gmail.com> writes:\n>\n>> While _eventually_ we should make this opt-out, we shouldn't do that\n>> until it has cooked a while.\n> I actually do not agree.  If the knob gives enough benefit, the\n> users will learn about it viva voce, and in a few more releases\n> we'll hear \"enough users complain that they have to turn it on,\n> let's make it the default\".  If that does not happen, the knob\n> does not deserve to be turned on in the first place.\n\nThat's a good philosophy. I'll keep it in mind.\n> Having said that, I won't be commenting on this shiny new toy before\n> the final.  I want to see more people help tying the loose ends and\n> give it final varnish to the upcoming release, as it seems to have\n> become rockier and larger release than we originally anticipated.\n\nYeah, no rush on this one. I just wanted to get some initial feedback \nabout the idea.\n\nThanks,\n-Stolee\n"},{"id":"364932","messageId":"pull.89.v3.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v3 0/6] Add a new \"sparse\" tree walk algorithm","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-10T16:42:40Z","receivedAt":"2018-12-10T16:42:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"One of the biggest remaining pain points for users of very large\nrepositories is the time it takes to run 'git push'. We inspected some slow\npushes by our developers and found that the \"Enumerating Objects\" phase of a\npush was very slow. This is unsurprising, because this is why reachability\nbitmaps exist. However, reachability bitmaps are not available to us because\nof the single pack-file requirement. The bitmap approach is intended for\nservers anyway, and clients have a much different behavior pattern.\n\nSpecifically, clients are normally pushing a very small number of objects\ncompared to the entire working directory. A typical user changes only a\nsmall cone of the working directory, so let's use that to our benefit.\n\nCreate a new \"sparse\" mode for 'git pack-objects' that uses the paths that\nintroduce new objects to direct our search into the reachable trees. By\ncollecting trees at each path, we can then recurse into a path only when\nthere are uninteresting and interesting trees at that path. This gains a\nsignificant performance boost for small topics while presenting a\npossibility of packing extra objects.\n\nThe main algorithm change is in patch 4, but is set up a little bit in\npatches 1 and 2.\n\nAs demonstrated in the included test script, we see that the existing\nalgorithm can send extra objects due to the way we specify the \"frontier\".\nBut we can send even more objects if a user copies objects from one folder\nto another. I say \"copy\" because a rename would (usually) change the\noriginal folder and trigger a walk into that path, discovering the objects.\n\nIn order to benefit from this approach, the user can opt-in using the\npack.useSparse config setting. This setting can be overridden using the\n'--no-sparse' option.\n\nUpdate in V2: \n\n * Added GIT_TEST_PACK_SPARSE test option.\n * Fixed test breakages when GIT_TEST_PACK_SPARSE is enabled by adding null\n   checks.\n\nUpdate in V3:\n\n * Change documentation around 'pack.useSparse' config setting to better\n   advertise to users.\n * Mostly a ping now that v2.20.0 is out.\n\nDerrick Stolee (6):\n  revision: add mark_tree_uninteresting_sparse\n  list-objects: consume sparse tree walk\n  pack-objects: add --sparse option\n  revision: implement sparse algorithm\n  pack-objects: create pack.useSparse setting\n  pack-objects: create GIT_TEST_PACK_SPARSE\n\n Documentation/config/pack.txt      |   9 ++\n Documentation/git-pack-objects.txt |  11 ++-\n bisect.c                           |   2 +-\n builtin/pack-objects.c             |  10 ++-\n builtin/rev-list.c                 |   2 +-\n http-push.c                        |   2 +-\n list-objects.c                     |  55 +++++++++++-\n list-objects.h                     |   4 +-\n revision.c                         | 121 +++++++++++++++++++++++++\n revision.h                         |   2 +\n t/README                           |   4 +\n t/t5322-pack-objects-sparse.sh     | 139 +++++++++++++++++++++++++++++\n 12 files changed, 351 insertions(+), 10 deletions(-)\n create mode 100755 t/t5322-pack-objects-sparse.sh\n\n\nbase-commit: a1598010f775d82b5adf12c29d0f5bc9b41434c6\nPublished-As: https://github.com/gitgitgadget/git/releases/tags/pr-89%2Fderrickstolee%2Fpush%2Fsparse-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-89/derrickstolee/push/sparse-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/89\n\nRange-diff vs v2:\n\n 1:  60617681f7 = 1:  60617681f7 revision: add mark_tree_uninteresting_sparse\n 2:  4527addacb = 2:  4527addacb list-objects: consume sparse tree walk\n 3:  9644f6ff04 ! 3:  4ef318bdb2 pack-objects: add --sparse option\n     @@ -32,7 +32,9 @@\n       \n      +--sparse::\n      +\tUse the \"sparse\" algorithm to determine which objects to include in\n     -+\tthe pack. This can have significant performance benefits when computing\n     ++\tthe pack, when combined with the \"--revs\" option. This algorithm\n     ++\tonly walks trees that appear in paths that introduce new objects.\n     ++\tThis can have significant performance benefits when computing\n      +\ta pack to send a small change. However, it is possible that extra\n      +\tobjects are added to the pack-file if the included commits contain\n      +\tcertain types of direct renames.\n 4:  c99957d06f = 4:  571b2e2784 revision: implement sparse algorithm\n 5:  d6912188be ! 5:  33d2c04dd6 pack-objects: create pack.useSparse setting\n     @@ -19,6 +19,26 @@\n      \n          Signed-off-by: Derrick Stolee <dstolee@microsoft.com>\n      \n     +diff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt\n     +--- a/Documentation/config/pack.txt\n     ++++ b/Documentation/config/pack.txt\n     +@@\n     + \ttrue. You should not generally need to turn this off unless\n     + \tyou are debugging pack bitmaps.\n     + \n     ++pack.useSparse::\n     ++\tWhen true, git will default to using the '--sparse' option in\n     ++\t'git pack-objects' when the '--revs' option is present. This\n     ++\talgorithm only walks trees that appear in paths that introduce new\n     ++\tobjects. This can have significant performance benefits when\n     ++\tcomputing a pack to send a small change. However, it is possible\n     ++\tthat extra objects are added to the pack-file if the included\n     ++\tcommits contain certain types of direct renames.\n     ++\n     + pack.writeBitmaps (deprecated)::\n     + \tThis is a deprecated synonym for `repack.writeBitmaps`.\n     + \n     +\n      diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n      --- a/builtin/pack-objects.c\n      +++ b/builtin/pack-objects.c\n 6:  3d394a9136 = 6:  e4f29543ee pack-objects: create GIT_TEST_PACK_SPARSE\n\n-- \ngitgitgadget\n"},{"id":"364933","messageId":"60617681f7daee2a94ff23f91b09b02d58fbc3df.1544460159.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 1/6] revision: add mark_tree_uninteresting_sparse","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-10T16:42:42Z","receivedAt":"2018-12-10T16:42:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nIn preparation for a new algorithm that walks fewer trees when\ncreating a pack from a set of revisions, create a method that\ntakes an oidset of tree oids and marks reachable objects as\nUNINTERESTING.\n\nThe current implementation uses the existing\nmark_tree_uninteresting to recursively walk the trees and blobs.\nThis will walk the same number of trees as the old mechanism.\n\nThere is one new assumption in this approach: we are also given\nthe oids of the interesting trees. This implementation does not\nuse those trees at the moment, but we will use them in a later\nrewrite of this method.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n revision.c | 22 ++++++++++++++++++++++\n revision.h |  2 ++\n 2 files changed, 24 insertions(+)\n\ndiff --git a/revision.c b/revision.c\nindex 13e0519c02..3a62c7c187 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -99,6 +99,28 @@ void mark_tree_uninteresting(struct repository *r, struct tree *tree)\n \tmark_tree_contents_uninteresting(r, tree);\n }\n \n+void mark_trees_uninteresting_sparse(struct repository *r,\n+\t\t\t\t     struct oidset *set)\n+{\n+\tstruct object_id *oid;\n+\tstruct oidset_iter iter;\n+\n+\toidset_iter_init(set, &iter);\n+\twhile ((oid = oidset_iter_next(&iter))) {\n+\t\tstruct tree *tree = lookup_tree(r, oid);\n+\n+\t\tif (tree->object.flags & UNINTERESTING) {\n+\t\t\t/*\n+\t\t\t * Remove the flag so the next call\n+\t\t\t * is not a no-op. The flag is added\n+\t\t\t * in mark_tree_unintersting().\n+\t\t\t */\n+\t\t\ttree->object.flags ^= UNINTERESTING;\n+\t\t\tmark_tree_uninteresting(r, tree);\n+\t\t}\n+\t}\n+}\n+\n struct commit_stack {\n \tstruct commit **items;\n \tsize_t nr, alloc;\ndiff --git a/revision.h b/revision.h\nindex 7987bfcd2e..f828e91ae9 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -67,6 +67,7 @@ struct rev_cmdline_info {\n #define REVISION_WALK_NO_WALK_SORTED 1\n #define REVISION_WALK_NO_WALK_UNSORTED 2\n \n+struct oidset;\n struct topo_walk_info;\n \n struct rev_info {\n@@ -327,6 +328,7 @@ void put_revision_mark(const struct rev_info *revs,\n \n void mark_parents_uninteresting(struct commit *commit);\n void mark_tree_uninteresting(struct repository *r, struct tree *tree);\n+void mark_trees_uninteresting_sparse(struct repository *r, struct oidset *set);\n \n void show_object_with_name(FILE *, struct object *, const char *);\n \n-- \ngitgitgadget\n\n"},{"id":"364934","messageId":"4ef318bdb2f1f7c55130f875280604e816cc183f.1544460159.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 3/6] pack-objects: add --sparse option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-10T16:42:44Z","receivedAt":"2018-12-10T16:42:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nAdd a '--sparse' option flag to the pack-objects builtin. This\nallows the user to specify that they want to use the new logic\nfor walking trees. This logic currently does not differ from the\nexisting output, but will in a later change.\n\nCreate a new test script, t5322-pack-objects-sparse.sh, to ensure\nthe object list that is selected matches what we expect. When we\nupdate the logic to walk in a sparse fashion, the final test will\nbe updated to show the extra objects that are added.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n Documentation/git-pack-objects.txt |  11 ++-\n builtin/pack-objects.c             |   5 +-\n t/t5322-pack-objects-sparse.sh     | 115 +++++++++++++++++++++++++++++\n 3 files changed, 129 insertions(+), 2 deletions(-)\n create mode 100755 t/t5322-pack-objects-sparse.sh\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 40c825c381..e45f3e680d 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -14,7 +14,7 @@ SYNOPSIS\n \t[--local] [--incremental] [--window=<n>] [--depth=<n>]\n \t[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\n \t[--stdout [--filter=<filter-spec>] | base-name]\n-\t[--shallow] [--keep-true-parents] < object-list\n+\t[--shallow] [--keep-true-parents] [--sparse] < object-list\n \n \n DESCRIPTION\n@@ -196,6 +196,15 @@ depth is 4095.\n \tAdd --no-reuse-object if you want to force a uniform compression\n \tlevel on all data no matter the source.\n \n+--sparse::\n+\tUse the \"sparse\" algorithm to determine which objects to include in\n+\tthe pack, when combined with the \"--revs\" option. This algorithm\n+\tonly walks trees that appear in paths that introduce new objects.\n+\tThis can have significant performance benefits when computing\n+\ta pack to send a small change. However, it is possible that extra\n+\tobjects are added to the pack-file if the included commits contain\n+\tcertain types of direct renames.\n+\n --thin::\n \tCreate a \"thin\" pack by omitting the common objects between a\n \tsender and a receiver in order to reduce network transfer. This\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5f70d840a7..7d5b0735e3 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -84,6 +84,7 @@ static unsigned long pack_size_limit;\n static int depth = 50;\n static int delta_search_threads;\n static int pack_to_stdout;\n+static int sparse;\n static int thin;\n static int num_preferred_base;\n static struct progress *progress_state;\n@@ -3135,7 +3136,7 @@ static void get_object_list(int ac, const char **av)\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n-\tmark_edges_uninteresting(&revs, show_edge, 0);\n+\tmark_edges_uninteresting(&revs, show_edge, sparse);\n \n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\n@@ -3292,6 +3293,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t{ OPTION_CALLBACK, 0, \"unpack-unreachable\", NULL, N_(\"time\"),\n \t\t  N_(\"unpack unreachable objects newer than <time>\"),\n \t\t  PARSE_OPT_OPTARG, option_parse_unpack_unreachable },\n+\t\tOPT_BOOL(0, \"sparse\", &sparse,\n+\t\t\t N_(\"use the sparse reachability algorithm\")),\n \t\tOPT_BOOL(0, \"thin\", &thin,\n \t\t\t N_(\"create thin packs\")),\n \t\tOPT_BOOL(0, \"shallow\", &shallow,\ndiff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\nnew file mode 100755\nindex 0000000000..81f6805bc3\n--- /dev/null\n+++ b/t/t5322-pack-objects-sparse.sh\n@@ -0,0 +1,115 @@\n+#!/bin/sh\n+\n+test_description='pack-objects object selection using sparse algorithm'\n+. ./test-lib.sh\n+\n+test_expect_success 'setup repo' '\n+\ttest_commit initial &&\n+\tfor i in $(test_seq 1 3)\n+\tdo\n+\t\tmkdir f$i &&\n+\t\tfor j in $(test_seq 1 3)\n+\t\tdo\n+\t\t\tmkdir f$i/f$j &&\n+\t\t\techo $j >f$i/f$j/data.txt\n+\t\tdone\n+\tdone &&\n+\tgit add . &&\n+\tgit commit -m \"Initialized trees\" &&\n+\tfor i in $(test_seq 1 3)\n+\tdo\n+\t\tgit checkout -b topic$i master &&\n+\t\techo change-$i >f$i/f$i/data.txt &&\n+\t\tgit commit -a -m \"Changed f$i/f$i/data.txt\"\n+\tdone &&\n+\tcat >packinput.txt <<-EOF &&\n+\ttopic1\n+\t^topic2\n+\t^topic3\n+\tEOF\n+\tgit rev-parse\t\t\t\\\n+\t\ttopic1\t\t\t\\\n+\t\ttopic1^{tree}\t\t\\\n+\t\ttopic1:f1\t\t\\\n+\t\ttopic1:f1/f1\t\t\\\n+\t\ttopic1:f1/f1/data.txt | sort >expect_objects.txt\n+'\n+\n+test_expect_success 'non-sparse pack-objects' '\n+\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n+\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n+\ttest_cmp expect_objects.txt nonsparse_objects.txt\n+'\n+\n+test_expect_success 'sparse pack-objects' '\n+\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_objects.txt sparse_objects.txt\n+'\n+\n+# Demonstrate that both algorithms send \"extra\" objects because\n+# they are not in the frontier.\n+\n+test_expect_success 'duplicate a folder from f3 and commit to topic1' '\n+\tgit checkout topic1 &&\n+\techo change-3 >f3/f3/data.txt &&\n+\tgit commit -a -m \"Changed f3/f3/data.txt\" &&\n+\tgit rev-parse\t\t\t\\\n+\t\ttopic1~1\t\t\\\n+\t\ttopic1~1^{tree}\t\t\\\n+\t\ttopic1^{tree}\t\t\\\n+\t\ttopic1\t\t\t\\\n+\t\ttopic1:f1\t\t\\\n+\t\ttopic1:f1/f1\t\t\\\n+\t\ttopic1:f1/f1/data.txt\t\\\n+\t\ttopic1:f3\t\t\\\n+\t\ttopic1:f3/f3\t\t\\\n+\t\ttopic1:f3/f3/data.txt | sort >expect_objects.txt\n+'\n+\n+test_expect_success 'non-sparse pack-objects' '\n+\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n+\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n+\ttest_cmp expect_objects.txt nonsparse_objects.txt\n+'\n+\n+test_expect_success 'sparse pack-objects' '\n+\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_objects.txt sparse_objects.txt\n+'\n+\n+test_expect_success 'duplicate a folder from f1 into f3' '\n+\tmkdir f3/f4 &&\n+\tcp -r f1/f1/* f3/f4 &&\n+\tgit add f3/f4 &&\n+\tgit commit -m \"Copied f1/f1 to f3/f4\" &&\n+\tcat >packinput.txt <<-EOF &&\n+\ttopic1\n+\t^topic1~1\n+\tEOF\n+\tgit rev-parse\t\t\\\n+\t\ttopic1\t\t\\\n+\t\ttopic1^{tree}\t\\\n+\t\ttopic1:f3 | sort >expect_objects.txt\n+'\n+\n+test_expect_success 'non-sparse pack-objects' '\n+\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n+\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n+\ttest_cmp expect_objects.txt nonsparse_objects.txt\n+'\n+\n+test_expect_success 'sparse pack-objects' '\n+\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_objects.txt sparse_objects.txt\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"364935","messageId":"4527addacb6cb663e463859c6661918ea47768cd.1544460159.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 2/6] list-objects: consume sparse tree walk","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-10T16:42:43Z","receivedAt":"2018-12-10T16:42:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen creating a pack-file using 'git pack-objects --revs' we provide\na list of interesting and uninteresting commits. For example, a push\noperation would make the local topic branch be interesting and the\nknown remote refs as uninteresting. We want to discover the set of\nnew objects to send to the server as a thin pack.\n\nWe walk these commits until we discover a frontier of commits such\nthat every commit walk starting at interesting commits ends in a root\ncommit or unintersting commit. We then need to discover which\nnon-commit objects are reachable from  uninteresting commits.\n\nThe mark_edges_unintersting() method in list-objects.c iterates on\nthe commit list and does the following:\n\n* If the commit is UNINTERSTING, then mark its root tree and every\n  object it can reach as UNINTERESTING.\n\n* If the commit is interesting, then mark the root tree of every\n  UNINTERSTING parent (and all objects that tree can reach) as\n  UNINTERSTING.\n\nAt the very end, we repeat the process on every commit directly\ngiven to the revision walk from stdin. This helps ensure we properly\ncover shallow commits that otherwise were not included in the\nfrontier.\n\nThe logic to recursively follow trees is in the\nmark_tree_uninteresting() method in revision.c. The algorithm avoids\nduplicate work by not recursing into trees that are already marked\nUNINTERSTING.\n\nAdd a new 'sparse' option to the mark_edges_uninteresting() method\nthat performs this logic in a slightly new way. As we iterate over\nthe commits, we add all of the root trees to an oidset. Then, call\nmark_trees_uninteresting_sparse() on that oidset. Note that we\ninclude interesting trees in this process. The current implementation\nof mark_trees_unintersting_sparse() will walk the same trees as\nthe old logic, but this will be replaced in a later change.\n\nThe sparse option is not used by any callers at the moment, but\nwill be wired to 'git pack-objects' in the next change.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n bisect.c               |  2 +-\n builtin/pack-objects.c |  2 +-\n builtin/rev-list.c     |  2 +-\n http-push.c            |  2 +-\n list-objects.c         | 55 +++++++++++++++++++++++++++++++++++++++---\n list-objects.h         |  4 ++-\n revision.c             |  3 +++\n 7 files changed, 61 insertions(+), 9 deletions(-)\n\ndiff --git a/bisect.c b/bisect.c\nindex 487675c672..842f8b4b8f 100644\n--- a/bisect.c\n+++ b/bisect.c\n@@ -656,7 +656,7 @@ static void bisect_common(struct rev_info *revs)\n \tif (prepare_revision_walk(revs))\n \t\tdie(\"revision walk setup failed\");\n \tif (revs->tree_objects)\n-\t\tmark_edges_uninteresting(revs, NULL);\n+\t\tmark_edges_uninteresting(revs, NULL, 0);\n }\n \n static void exit_if_skipped_commits(struct commit_list *tried,\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 411aefd687..5f70d840a7 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3135,7 +3135,7 @@ static void get_object_list(int ac, const char **av)\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n-\tmark_edges_uninteresting(&revs, show_edge);\n+\tmark_edges_uninteresting(&revs, show_edge, 0);\n \n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 2880ed37e3..9663cbfae0 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -543,7 +543,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \tif (revs.tree_objects)\n-\t\tmark_edges_uninteresting(&revs, show_edge);\n+\t\tmark_edges_uninteresting(&revs, show_edge, 0);\n \n \tif (bisect_list) {\n \t\tint reaches, all;\ndiff --git a/http-push.c b/http-push.c\nindex cd48590912..ea52d6f9f6 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -1933,7 +1933,7 @@ int cmd_main(int argc, const char **argv)\n \t\tpushing = 0;\n \t\tif (prepare_revision_walk(&revs))\n \t\t\tdie(\"revision walk setup failed\");\n-\t\tmark_edges_uninteresting(&revs, NULL);\n+\t\tmark_edges_uninteresting(&revs, NULL, 0);\n \t\tobjects_to_send = get_delta(&revs, ref_lock);\n \t\tfinish_all_active_slots();\n \ndiff --git a/list-objects.c b/list-objects.c\nindex c41cc80db5..4fbdeca0a4 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -222,25 +222,72 @@ static void mark_edge_parents_uninteresting(struct commit *commit,\n \t}\n }\n \n-void mark_edges_uninteresting(struct rev_info *revs, show_edge_fn show_edge)\n+static void add_edge_parents(struct commit *commit,\n+\t\t\t     struct rev_info *revs,\n+\t\t\t     show_edge_fn show_edge,\n+\t\t\t     struct oidset *set)\n+{\n+\tstruct commit_list *parents;\n+\n+\tfor (parents = commit->parents; parents; parents = parents->next) {\n+\t\tstruct commit *parent = parents->item;\n+\t\tstruct tree *tree = get_commit_tree(parent);\n+\n+\t\tif (!tree)\n+\t\t\tcontinue;\n+\n+\t\toidset_insert(set, &tree->object.oid);\n+\n+\t\tif (!(parent->object.flags & UNINTERESTING))\n+\t\t\tcontinue;\n+\t\ttree->object.flags |= UNINTERESTING;\n+\n+\t\tif (revs->edge_hint && !(parent->object.flags & SHOWN)) {\n+\t\t\tparent->object.flags |= SHOWN;\n+\t\t\tshow_edge(parent);\n+\t\t}\n+\t}\n+}\n+\n+void mark_edges_uninteresting(struct rev_info *revs,\n+\t\t\t      show_edge_fn show_edge,\n+\t\t\t      int sparse)\n {\n \tstruct commit_list *list;\n+\tstruct oidset set;\n \tint i;\n \n+\tif (sparse)\n+\t\toidset_init(&set, 16);\n+\n \tfor (list = revs->commits; list; list = list->next) {\n \t\tstruct commit *commit = list->item;\n \n-\t\tif (commit->object.flags & UNINTERESTING) {\n+\t\tif (sparse) {\n+\t\t\tstruct tree *tree = get_commit_tree(commit);\n+\n+\t\t\tif (commit->object.flags & UNINTERESTING)\n+\t\t\t\ttree->object.flags |= UNINTERESTING;\n+\n+\t\t\toidset_insert(&set, &tree->object.oid);\n+\t\t\tadd_edge_parents(commit, revs, show_edge, &set);\n+\t\t} else if (commit->object.flags & UNINTERESTING) {\n \t\t\tmark_tree_uninteresting(revs->repo,\n \t\t\t\t\t\tget_commit_tree(commit));\n \t\t\tif (revs->edge_hint_aggressive && !(commit->object.flags & SHOWN)) {\n \t\t\t\tcommit->object.flags |= SHOWN;\n \t\t\t\tshow_edge(commit);\n \t\t\t}\n-\t\t\tcontinue;\n+\t\t} else {\n+\t\t\tmark_edge_parents_uninteresting(commit, revs, show_edge);\n \t\t}\n-\t\tmark_edge_parents_uninteresting(commit, revs, show_edge);\n \t}\n+\n+\tif (sparse) {\n+\t\tmark_trees_uninteresting_sparse(revs->repo, &set);\n+\t\toidset_clear(&set);\n+\t}\n+\n \tif (revs->edge_hint_aggressive) {\n \t\tfor (i = 0; i < revs->cmdline.nr; i++) {\n \t\t\tstruct object *obj = revs->cmdline.rev[i].item;\ndiff --git a/list-objects.h b/list-objects.h\nindex ad40762926..a952680e46 100644\n--- a/list-objects.h\n+++ b/list-objects.h\n@@ -10,7 +10,9 @@ typedef void (*show_object_fn)(struct object *, const char *, void *);\n void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);\n \n typedef void (*show_edge_fn)(struct commit *);\n-void mark_edges_uninteresting(struct rev_info *, show_edge_fn);\n+void mark_edges_uninteresting(struct rev_info *revs,\n+\t\t\t      show_edge_fn show_edge,\n+\t\t\t      int sparse);\n \n struct oidset;\n struct list_objects_filter_options;\ndiff --git a/revision.c b/revision.c\nindex 3a62c7c187..f9eb6400f1 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -109,6 +109,9 @@ void mark_trees_uninteresting_sparse(struct repository *r,\n \twhile ((oid = oidset_iter_next(&iter))) {\n \t\tstruct tree *tree = lookup_tree(r, oid);\n \n+\t\tif (!tree)\n+\t\t\tcontinue;\n+\n \t\tif (tree->object.flags & UNINTERESTING) {\n \t\t\t/*\n \t\t\t * Remove the flag so the next call\n-- \ngitgitgadget\n\n"},{"id":"364936","messageId":"571b2e278439531b92ea6b80cb812ec0d176a8d4.1544460159.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 4/6] revision: implement sparse algorithm","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-10T16:42:45Z","receivedAt":"2018-12-10T16:42:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen enumerating objects to place in a pack-file during 'git\npack-objects --revs', we discover the \"frontier\" of commits\nthat we care about and the boundary with commit we find\nuninteresting. From that point, we walk trees to discover which\ntrees and blobs are uninteresting. Finally, we walk trees to find\nthe interesting trees.\n\nThis commit introduces a new, \"sparse\" way to discover the\nuninteresting trees. We use the perspective of a single user trying\nto push their topic to a large repository. That user likely changed\na very small fraction of the paths in their working directory, but\nwe spend a lot of time walking all reachable trees.\n\nThe way to switch the logic to work in this sparse way is to start\ncaring about which paths introduce new trees. While it is not\npossible to generate a diff between the frontier boundary and all\nof the interesting commits, we can simulate that behavior by\ninspecting all of the root trees as a whole, then recursing down\nto the set of trees at each path.\n\nWe already had taken the first step by passing an oidset to\nmark_trees_uninteresting_sparse(). We now create a dictionary\nwhose keys are paths and values are oidsets. We consider the set\nof trees that appear at each path. While we inspect a tree, we\nadd its subtrees to the oidsets corresponding to the tree entry's\npath. We also mark trees as UNINTERESTING if the tree we are\nparsing is UNINTERESTING.\n\nTo actually improve the peformance, we need to terminate our\nrecursion unless the oidset contains some intersting trees and\nsome uninteresting trees. Technically, we only need one interesting\ntree for this to speed up in most cases, but we also will not mark\nanything UNINTERESTING if there are no uninteresting trees, so\nthat would be wasted effort.\n\nThere are a few ways that this is not a universally better option.\n\nFirst, we can pack extra objects. If someone copies a subtree\nfrom one tree to another, the first tree will appear UNINTERESTING\nand we will not recurse to see that the subtree should also be\nUNINTERESTING. We will walk the new tree and see the subtree as\na \"new\" object and add it to the pack. We add a test case that\ndemonstrates this as a way to prove that the --sparse option is\nactually working.\n\nSecond, we can have extra memory pressure. If instead of being a\nsingle user pushing a small topic we are a server sending new\nobjects from across the entire working directory, then we will\ngain very little (the recursion will rarely terminate early) but\nwill spend extra time maintaining the path-oidset dictionaries.\n\nDespite these potential drawbacks, the benefits of the algorithm\nare clear. By adding a counter to 'add_children_by_path' and\n'mark_tree_contents_uninteresting', I measured the number of\nparsed trees for the two algorithms in a variety of repos.\n\nFor git.git, I used the following input:\n\n\tv2.19.0\n\t^v2.19.0~10\n\n Objects to pack: 550\nWalked (old alg): 282\nWalked (new alg): 130\n\nFor the Linux repo, I used the following input:\n\n\tv4.18\n\t^v4.18~10\n\n Objects to pack:   518\nWalked (old alg): 4,836\nWalked (new alg):   188\n\nThe two repos above are rather \"wide and flat\" compared to\nother repos that I have used in the past. As a comparison,\nI tested an old topic branch in the Azure DevOps repo, which\nhas a much deeper folder structure than the Linux repo.\n\n Objects to pack:    220\nWalked (old alg): 22,804\nWalked (new alg):    129\n\nI used the number of walked trees the main metric above because\nit is consistent across multiple runs. When I ran my tests, the\nperformance of the pack-objects command with the same options\ncould change the end-to-end time by 10x depending on the file\nsystem being warm. However, by repeating the same test on repeat\nI could get more consistent timing results. The git.git and\nLinux tests were too fast overall (less than 0.5s) to measure\nan end-to-end difference. The Azure DevOps case was slow enough\nto see the time improve from 15s to 1s in the warm case. The\ncold case was 90s to 9s in my testing.\n\nThese improvements will have even larger benefits in the super-\nlarge Windows repository. In our experiments, we see the\n\"Enumerate objects\" phase of pack-objects taking 60-80% of the\nend-to-end time of non-trivial pushes, taking longer than the\nnetwork time to send the pack and the server time to verify the\npack.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n revision.c                     | 116 ++++++++++++++++++++++++++++++---\n t/t5322-pack-objects-sparse.sh |  21 ++++--\n 2 files changed, 121 insertions(+), 16 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex f9eb6400f1..971f1bb095 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -99,29 +99,125 @@ void mark_tree_uninteresting(struct repository *r, struct tree *tree)\n \tmark_tree_contents_uninteresting(r, tree);\n }\n \n+struct paths_and_oids {\n+\tstruct string_list list;\n+};\n+\n+static void paths_and_oids_init(struct paths_and_oids *po)\n+{\n+\tstring_list_init(&po->list, 1);\n+}\n+\n+static void paths_and_oids_clear(struct paths_and_oids *po)\n+{\n+\tint i;\n+\tfor (i = 0; i < po->list.nr; i++) {\n+\t\toidset_clear(po->list.items[i].util);\n+\t\tfree(po->list.items[i].util);\n+\t}\n+\n+\tstring_list_clear(&po->list, 0);\n+}\n+\n+static void paths_and_oids_insert(struct paths_and_oids *po,\n+\t\t\t\t  const char *path,\n+\t\t\t\t  const struct object_id *oid)\n+{\n+\tstruct string_list_item *item = string_list_insert(&po->list, path);\n+\tstruct oidset *set;\n+\n+\tif (!item->util) {\n+\t\tset = xcalloc(1, sizeof(struct oidset));\n+\t\toidset_init(set, 16);\n+\t\titem->util = set;\n+\t} else {\n+\t\tset = item->util;\n+\t}\n+\n+\toidset_insert(set, oid);\n+}\n+\n+static void add_children_by_path(struct repository *r,\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct paths_and_oids *po)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\n+\tif (!tree)\n+\t\treturn;\n+\n+\tif (parse_tree_gently(tree, 1) < 0)\n+\t\treturn;\n+\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tswitch (object_type(entry.mode)) {\n+\t\tcase OBJ_TREE:\n+\t\t\tpaths_and_oids_insert(po, entry.path, entry.oid);\n+\n+\t\t\tif (tree->object.flags & UNINTERESTING) {\n+\t\t\t\tstruct tree *child = lookup_tree(r, entry.oid);\n+\t\t\t\tif (child)\n+\t\t\t\t\tchild->object.flags |= UNINTERESTING;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\tcase OBJ_BLOB:\n+\t\t\tif (tree->object.flags & UNINTERESTING) {\n+\t\t\t\tstruct blob *child = lookup_blob(r, entry.oid);\n+\t\t\t\tif (child)\n+\t\t\t\t\tchild->object.flags |= UNINTERESTING;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\t/* Subproject commit - not in this repository */\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfree_tree_buffer(tree);\n+}\n+\n void mark_trees_uninteresting_sparse(struct repository *r,\n \t\t\t\t     struct oidset *set)\n {\n+\tint i;\n+\tunsigned has_interesting = 0, has_uninteresting = 0;\n+\tstruct paths_and_oids po;\n \tstruct object_id *oid;\n \tstruct oidset_iter iter;\n \n \toidset_iter_init(set, &iter);\n-\twhile ((oid = oidset_iter_next(&iter))) {\n+\twhile ((!has_interesting || !has_uninteresting) &&\n+\t       (oid = oidset_iter_next(&iter))) {\n \t\tstruct tree *tree = lookup_tree(r, oid);\n \n \t\tif (!tree)\n \t\t\tcontinue;\n \n-\t\tif (tree->object.flags & UNINTERESTING) {\n-\t\t\t/*\n-\t\t\t * Remove the flag so the next call\n-\t\t\t * is not a no-op. The flag is added\n-\t\t\t * in mark_tree_unintersting().\n-\t\t\t */\n-\t\t\ttree->object.flags ^= UNINTERESTING;\n-\t\t\tmark_tree_uninteresting(r, tree);\n-\t\t}\n+\t\tif (tree->object.flags & UNINTERESTING)\n+\t\t\thas_uninteresting = 1;\n+\t\telse\n+\t\t\thas_interesting = 1;\n \t}\n+\n+\t/* Do not walk unless we have both types of trees. */\n+\tif (!has_uninteresting || !has_interesting)\n+\t\treturn;\n+\n+\tpaths_and_oids_init(&po);\n+\n+\toidset_iter_init(set, &iter);\n+\twhile ((oid = oidset_iter_next(&iter))) {\n+\t\tstruct tree *tree = lookup_tree(r, oid);\n+\t\tadd_children_by_path(r, tree, &po);\n+\t}\n+\n+\tfor (i = 0; i < po.list.nr; i++)\n+\t\tmark_trees_uninteresting_sparse(\n+\t\t\tr, (struct oidset *)po.list.items[i].util);\n+\n+\tpaths_and_oids_clear(&po);\n }\n \n struct commit_stack {\ndiff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\nindex 81f6805bc3..45dba6e014 100755\n--- a/t/t5322-pack-objects-sparse.sh\n+++ b/t/t5322-pack-objects-sparse.sh\n@@ -83,22 +83,25 @@ test_expect_success 'sparse pack-objects' '\n \ttest_cmp expect_objects.txt sparse_objects.txt\n '\n \n+# Demonstrate that the algorithms differ when we copy a tree wholesale\n+# from one folder to another.\n+\n test_expect_success 'duplicate a folder from f1 into f3' '\n \tmkdir f3/f4 &&\n \tcp -r f1/f1/* f3/f4 &&\n \tgit add f3/f4 &&\n \tgit commit -m \"Copied f1/f1 to f3/f4\" &&\n-\tcat >packinput.txt <<-EOF &&\n+\tcat >packinput.txt <<-EOF\n \ttopic1\n \t^topic1~1\n \tEOF\n-\tgit rev-parse\t\t\\\n-\t\ttopic1\t\t\\\n-\t\ttopic1^{tree}\t\\\n-\t\ttopic1:f3 | sort >expect_objects.txt\n '\n \n test_expect_success 'non-sparse pack-objects' '\n+\tgit rev-parse\t\t\t\\\n+\t\ttopic1\t\t\t\\\n+\t\ttopic1^{tree}\t\t\\\n+\t\ttopic1:f3 | sort >expect_objects.txt &&\n \tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n \tgit index-pack -o nonsparse.idx nonsparse.pack &&\n \tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n@@ -106,10 +109,16 @@ test_expect_success 'non-sparse pack-objects' '\n '\n \n test_expect_success 'sparse pack-objects' '\n+\tgit rev-parse\t\t\t\\\n+\t\ttopic1\t\t\t\\\n+\t\ttopic1^{tree}\t\t\\\n+\t\ttopic1:f3\t\t\\\n+\t\ttopic1:f3/f4\t\t\\\n+\t\ttopic1:f3/f4/data.txt | sort >expect_sparse_objects.txt &&\n \tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n \tgit index-pack -o sparse.idx sparse.pack &&\n \tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n-\ttest_cmp expect_objects.txt sparse_objects.txt\n+\ttest_cmp expect_sparse_objects.txt sparse_objects.txt\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"364938","messageId":"e4f29543ee3b03c55ecd99b624f06b9bdf96ef7e.1544460159.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 6/6] pack-objects: create GIT_TEST_PACK_SPARSE","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-10T16:42:48Z","receivedAt":"2018-12-10T16:42:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nCreate a test variable GIT_TEST_PACK_SPARSE to enable the sparse\nobject walk algorithm by default during the test suite. Enabling\nthis variable ensures coverage in many interesting cases, such as\nshallow clones, partial clones, and missing objects.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n builtin/pack-objects.c         | 1 +\n t/README                       | 4 ++++\n t/t5322-pack-objects-sparse.sh | 6 +++---\n 3 files changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 124b1bafc4..507d381153 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3331,6 +3331,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \n \tread_replace_refs = 0;\n \n+\tsparse = git_env_bool(\"GIT_TEST_PACK_SPARSE\", 0);\n \treset_pack_idx_option(&pack_idx_opts);\n \tgit_config(git_pack_config, NULL);\n \ndiff --git a/t/README b/t/README\nindex 28711cc508..8b6dfe1864 100644\n--- a/t/README\n+++ b/t/README\n@@ -342,6 +342,10 @@ GIT_TEST_INDEX_VERSION=<n> exercises the index read/write code path\n for the index version specified.  Can be set to any valid version\n (currently 2, 3, or 4).\n \n+GIT_TEST_PACK_SPARSE=<boolean> if enabled will default the pack-objects\n+builtin to use the sparse object walk. This can still be overridden by\n+the --no-sparse command-line argument.\n+\n GIT_TEST_PRELOAD_INDEX=<boolean> exercises the preload-index code path\n by overriding the minimum number of cache entries required per thread.\n \ndiff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\nindex 8f5699bd91..e8cf41d1c6 100755\n--- a/t/t5322-pack-objects-sparse.sh\n+++ b/t/t5322-pack-objects-sparse.sh\n@@ -36,7 +36,7 @@ test_expect_success 'setup repo' '\n '\n \n test_expect_success 'non-sparse pack-objects' '\n-\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >nonsparse.pack &&\n \tgit index-pack -o nonsparse.idx nonsparse.pack &&\n \tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n \ttest_cmp expect_objects.txt nonsparse_objects.txt\n@@ -70,7 +70,7 @@ test_expect_success 'duplicate a folder from f3 and commit to topic1' '\n '\n \n test_expect_success 'non-sparse pack-objects' '\n-\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >nonsparse.pack &&\n \tgit index-pack -o nonsparse.idx nonsparse.pack &&\n \tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n \ttest_cmp expect_objects.txt nonsparse_objects.txt\n@@ -102,7 +102,7 @@ test_expect_success 'non-sparse pack-objects' '\n \t\ttopic1\t\t\t\\\n \t\ttopic1^{tree}\t\t\\\n \t\ttopic1:f3 | sort >expect_objects.txt &&\n-\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >nonsparse.pack &&\n \tgit index-pack -o nonsparse.idx nonsparse.pack &&\n \tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n \ttest_cmp expect_objects.txt nonsparse_objects.txt\n-- \ngitgitgadget\n"},{"id":"364937","messageId":"33d2c04dd696d4a7f648343f3da5e48a9e1f1b5b.1544460159.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 5/6] pack-objects: create pack.useSparse setting","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-10T16:42:46Z","receivedAt":"2018-12-10T16:42:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nThe '--sparse' flag in 'git pack-objects' changes the algorithm\nused to enumerate objects to one that is faster for individual\nusers pushing new objects that change only a small cone of the\nworking directory. The sparse algorithm is not recommended for a\nserver, which likely sends new objects that appear across the\nentire working directory.\n\nCreate a 'pack.useSparse' setting that enables this new algorithm.\nThis allows 'git push' to use this algorithm without passing a\n'--sparse' flag all the way through four levels of run_command()\ncalls.\n\nIf the '--no-sparse' flag is set, then this config setting is\noverridden.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n Documentation/config/pack.txt  |  9 +++++++++\n builtin/pack-objects.c         |  4 ++++\n t/t5322-pack-objects-sparse.sh | 15 +++++++++++++++\n 3 files changed, 28 insertions(+)\n\ndiff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt\nindex edac75c83f..425c73aa52 100644\n--- a/Documentation/config/pack.txt\n+++ b/Documentation/config/pack.txt\n@@ -105,6 +105,15 @@ pack.useBitmaps::\n \ttrue. You should not generally need to turn this off unless\n \tyou are debugging pack bitmaps.\n \n+pack.useSparse::\n+\tWhen true, git will default to using the '--sparse' option in\n+\t'git pack-objects' when the '--revs' option is present. This\n+\talgorithm only walks trees that appear in paths that introduce new\n+\tobjects. This can have significant performance benefits when\n+\tcomputing a pack to send a small change. However, it is possible\n+\tthat extra objects are added to the pack-file if the included\n+\tcommits contain certain types of direct renames.\n+\n pack.writeBitmaps (deprecated)::\n \tThis is a deprecated synonym for `repack.writeBitmaps`.\n \ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7d5b0735e3..124b1bafc4 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2711,6 +2711,10 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(k, \"pack.usesparse\")) {\n+\t\tsparse = git_config_bool(k, v);\n+\t\treturn 0;\n+\t}\n \tif (!strcmp(k, \"pack.threads\")) {\n \t\tdelta_search_threads = git_config_int(k, v);\n \t\tif (delta_search_threads < 0)\ndiff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\nindex 45dba6e014..8f5699bd91 100755\n--- a/t/t5322-pack-objects-sparse.sh\n+++ b/t/t5322-pack-objects-sparse.sh\n@@ -121,4 +121,19 @@ test_expect_success 'sparse pack-objects' '\n \ttest_cmp expect_sparse_objects.txt sparse_objects.txt\n '\n \n+test_expect_success 'pack.useSparse enables algorithm' '\n+\tgit config pack.useSparse true &&\n+\tgit pack-objects --stdout --revs <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_sparse_objects.txt sparse_objects.txt\n+'\n+\n+test_expect_success 'pack.useSparse overridden' '\n+\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_objects.txt sparse_objects.txt\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"365372","messageId":"pull.89.v4.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v4 0/6] Add a new \"sparse\" tree walk algorithm","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-14T21:22:15Z","receivedAt":"2018-12-14T21:22:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"One of the biggest remaining pain points for users of very large\nrepositories is the time it takes to run 'git push'. We inspected some slow\npushes by our developers and found that the \"Enumerating Objects\" phase of a\npush was very slow. This is unsurprising, because this is why reachability\nbitmaps exist. However, reachability bitmaps are not available to us because\nof the single pack-file requirement. The bitmap approach is intended for\nservers anyway, and clients have a much different behavior pattern.\n\nSpecifically, clients are normally pushing a very small number of objects\ncompared to the entire working directory. A typical user changes only a\nsmall cone of the working directory, so let's use that to our benefit.\n\nCreate a new \"sparse\" mode for 'git pack-objects' that uses the paths that\nintroduce new objects to direct our search into the reachable trees. By\ncollecting trees at each path, we can then recurse into a path only when\nthere are uninteresting and interesting trees at that path. This gains a\nsignificant performance boost for small topics while presenting a\npossibility of packing extra objects.\n\nThe main algorithm change is in patch 4, but is set up a little bit in\npatches 1 and 2.\n\nAs demonstrated in the included test script, we see that the existing\nalgorithm can send extra objects due to the way we specify the \"frontier\".\nBut we can send even more objects if a user copies objects from one folder\nto another. I say \"copy\" because a rename would (usually) change the\noriginal folder and trigger a walk into that path, discovering the objects.\n\nIn order to benefit from this approach, the user can opt-in using the\npack.useSparse config setting. This setting can be overridden using the\n'--no-sparse' option.\n\nUpdate in V2: \n\n * Added GIT_TEST_PACK_SPARSE test option.\n * Fixed test breakages when GIT_TEST_PACK_SPARSE is enabled by adding null\n   checks.\n\nUpdate in V3:\n\n * Change documentation around 'pack.useSparse' config setting to better\n   advertise to users.\n * Mostly a ping now that v2.20.0 is out.\n\nUpdates in V4:\n\n * Switched to using hashmap instead of string_list for the path/oidset\n   dictionary. (This is due to some fear that the string_list performance\n   would degrade for a very wide tree, but I am unable to measure a\n   performance difference.)\n * Some cleanup of code snippets across commits.\n * Some grammar cleanup in the commit messages.\n\nDerrick Stolee (6):\n  revision: add mark_tree_uninteresting_sparse\n  list-objects: consume sparse tree walk\n  pack-objects: add --sparse option\n  revision: implement sparse algorithm\n  pack-objects: create pack.useSparse setting\n  pack-objects: create GIT_TEST_PACK_SPARSE\n\n Documentation/config/pack.txt      |   9 ++\n Documentation/git-pack-objects.txt |  11 ++-\n bisect.c                           |   2 +-\n builtin/pack-objects.c             |  10 +-\n builtin/rev-list.c                 |   2 +-\n http-push.c                        |   2 +-\n list-objects.c                     |  70 +++++++++++---\n list-objects.h                     |   4 +-\n revision.c                         | 144 +++++++++++++++++++++++++++++\n revision.h                         |   2 +\n t/README                           |   4 +\n t/t5322-pack-objects-sparse.sh     | 139 ++++++++++++++++++++++++++++\n 12 files changed, 382 insertions(+), 17 deletions(-)\n create mode 100755 t/t5322-pack-objects-sparse.sh\n\n\nbase-commit: a1598010f775d82b5adf12c29d0f5bc9b41434c6\nPublished-As: https://github.com/gitgitgadget/git/releases/tags/pr-89%2Fderrickstolee%2Fpush%2Fsparse-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-89/derrickstolee/push/sparse-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/89\n\nRange-diff vs v3:\n\n 1:  60617681f7 ! 1:  817e30a287 revision: add mark_tree_uninteresting_sparse\n     @@ -35,6 +35,9 @@\n      +\twhile ((oid = oidset_iter_next(&iter))) {\n      +\t\tstruct tree *tree = lookup_tree(r, oid);\n      +\n     ++\t\tif (!tree)\n     ++\t\t\tcontinue;\n     ++\n      +\t\tif (tree->object.flags & UNINTERESTING) {\n      +\t\t\t/*\n      +\t\t\t * Remove the flag so the next call\n 2:  4527addacb ! 2:  39dc89beb9 list-objects: consume sparse tree walk\n     @@ -11,9 +11,10 @@\n          We walk these commits until we discover a frontier of commits such\n          that every commit walk starting at interesting commits ends in a root\n          commit or unintersting commit. We then need to discover which\n     -    non-commit objects are reachable from  uninteresting commits.\n     +    non-commit objects are reachable from  uninteresting commits. This\n     +    commit walk is not changing during this series.\n      \n     -    The mark_edges_unintersting() method in list-objects.c iterates on\n     +    The mark_edges_uninteresting() method in list-objects.c iterates on\n          the commit list and does the following:\n      \n          * If the commit is UNINTERSTING, then mark its root tree and every\n     @@ -138,17 +139,22 @@\n      +\t\t\t      int sparse)\n       {\n       \tstruct commit_list *list;\n     -+\tstruct oidset set;\n       \tint i;\n       \n     -+\tif (sparse)\n     +-\tfor (list = revs->commits; list; list = list->next) {\n     +-\t\tstruct commit *commit = list->item;\n     ++\tif (sparse) {\n     ++\t\tstruct oidset set;\n      +\t\toidset_init(&set, 16);\n     -+\n     - \tfor (list = revs->commits; list; list = list->next) {\n     - \t\tstruct commit *commit = list->item;\n       \n      -\t\tif (commit->object.flags & UNINTERESTING) {\n     -+\t\tif (sparse) {\n     +-\t\t\tmark_tree_uninteresting(revs->repo,\n     +-\t\t\t\t\t\tget_commit_tree(commit));\n     +-\t\t\tif (revs->edge_hint_aggressive && !(commit->object.flags & SHOWN)) {\n     +-\t\t\t\tcommit->object.flags |= SHOWN;\n     +-\t\t\t\tshow_edge(commit);\n     ++\t\tfor (list = revs->commits; list; list = list->next) {\n     ++\t\t\tstruct commit *commit = list->item;\n      +\t\t\tstruct tree *tree = get_commit_tree(commit);\n      +\n      +\t\t\tif (commit->object.flags & UNINTERESTING)\n     @@ -156,24 +162,27 @@\n      +\n      +\t\t\toidset_insert(&set, &tree->object.oid);\n      +\t\t\tadd_edge_parents(commit, revs, show_edge, &set);\n     -+\t\t} else if (commit->object.flags & UNINTERESTING) {\n     - \t\t\tmark_tree_uninteresting(revs->repo,\n     - \t\t\t\t\t\tget_commit_tree(commit));\n     - \t\t\tif (revs->edge_hint_aggressive && !(commit->object.flags & SHOWN)) {\n     - \t\t\t\tcommit->object.flags |= SHOWN;\n     - \t\t\t\tshow_edge(commit);\n     ++\t\t}\n     ++\n     ++\t\tmark_trees_uninteresting_sparse(revs->repo, &set);\n     ++\t\toidset_clear(&set);\n     ++\t} else {\n     ++\t\tfor (list = revs->commits; list; list = list->next) {\n     ++\t\t\tstruct commit *commit = list->item;\n     ++\t\t\tif (commit->object.flags & UNINTERESTING) {\n     ++\t\t\t\tmark_tree_uninteresting(revs->repo,\n     ++\t\t\t\t\t\t\tget_commit_tree(commit));\n     ++\t\t\t\tif (revs->edge_hint_aggressive && !(commit->object.flags & SHOWN)) {\n     ++\t\t\t\t\tcommit->object.flags |= SHOWN;\n     ++\t\t\t\t\tshow_edge(commit);\n     ++\t\t\t\t}\n     ++\t\t\t\tcontinue;\n       \t\t\t}\n      -\t\t\tcontinue;\n     -+\t\t} else {\n      +\t\t\tmark_edge_parents_uninteresting(commit, revs, show_edge);\n       \t\t}\n      -\t\tmark_edge_parents_uninteresting(commit, revs, show_edge);\n       \t}\n     -+\n     -+\tif (sparse) {\n     -+\t\tmark_trees_uninteresting_sparse(revs->repo, &set);\n     -+\t\toidset_clear(&set);\n     -+\t}\n      +\n       \tif (revs->edge_hint_aggressive) {\n       \t\tfor (i = 0; i < revs->cmdline.nr; i++) {\n     @@ -193,17 +202,3 @@\n       \n       struct oidset;\n       struct list_objects_filter_options;\n     -\n     -diff --git a/revision.c b/revision.c\n     ---- a/revision.c\n     -+++ b/revision.c\n     -@@\n     - \twhile ((oid = oidset_iter_next(&iter))) {\n     - \t\tstruct tree *tree = lookup_tree(r, oid);\n     - \n     -+\t\tif (!tree)\n     -+\t\t\tcontinue;\n     -+\n     - \t\tif (tree->object.flags & UNINTERESTING) {\n     - \t\t\t/*\n     - \t\t\t * Remove the flag so the next call\n 3:  4ef318bdb2 = 3:  ab733daff5 pack-objects: add --sparse option\n 4:  571b2e2784 ! 4:  c44172c35e revision: implement sparse algorithm\n     @@ -6,8 +6,9 @@\n          pack-objects --revs', we discover the \"frontier\" of commits\n          that we care about and the boundary with commit we find\n          uninteresting. From that point, we walk trees to discover which\n     -    trees and blobs are uninteresting. Finally, we walk trees to find\n     -    the interesting trees.\n     +    trees and blobs are uninteresting. Finally, we walk trees from the\n     +    interesting commits to find the interesting objects that are\n     +    placed in the pack.\n      \n          This commit introduces a new, \"sparse\" way to discover the\n          uninteresting trees. We use the perspective of a single user trying\n     @@ -31,11 +32,13 @@\n          parsing is UNINTERESTING.\n      \n          To actually improve the peformance, we need to terminate our\n     -    recursion unless the oidset contains some intersting trees and\n     -    some uninteresting trees. Technically, we only need one interesting\n     -    tree for this to speed up in most cases, but we also will not mark\n     -    anything UNINTERESTING if there are no uninteresting trees, so\n     -    that would be wasted effort.\n     +    recursion. If the oidset contains only UNINTERESTING trees, then\n     +    we do not continue the recursion. This avoids walking trees that\n     +    are likely to not be reachable from interesting trees. If the\n     +    oidset contains only interesting trees, then we will walk these\n     +    trees in the final stage that collects the intersting objects to\n     +    place in the pack. Thus, we only recurse if the oidset contains\n     +    both interesting and UNINITERESTING trees.\n      \n          There are a few ways that this is not a universally better option.\n      \n     @@ -108,51 +111,80 @@\n      diff --git a/revision.c b/revision.c\n      --- a/revision.c\n      +++ b/revision.c\n     +@@\n     + #include \"commit-reach.h\"\n     + #include \"commit-graph.h\"\n     + #include \"prio-queue.h\"\n     ++#include \"hashmap.h\"\n     + \n     + volatile show_early_output_fn_t show_early_output;\n     + \n      @@\n       \tmark_tree_contents_uninteresting(r, tree);\n       }\n       \n     -+struct paths_and_oids {\n     -+\tstruct string_list list;\n     ++struct path_and_oids_entry {\n     ++\tstruct hashmap_entry ent;\n     ++\tchar *path;\n     ++\tstruct oidset set;\n      +};\n      +\n     -+static void paths_and_oids_init(struct paths_and_oids *po)\n     ++static int path_and_oids_cmp(const void *hashmap_cmp_fn_data,\n     ++\t\t\t     const struct path_and_oids_entry *e1,\n     ++\t\t\t     const struct path_and_oids_entry *e2,\n     ++\t\t\t     const void *keydata)\n     ++{\n     ++\treturn strcmp(e1->path, e2->path);\n     ++}\n     ++\n     ++int map_flags = 0;\n     ++static void paths_and_oids_init(struct hashmap *map)\n      +{\n     -+\tstring_list_init(&po->list, 1);\n     ++\thashmap_init(map, (hashmap_cmp_fn) path_and_oids_cmp, &map_flags, 0);\n      +}\n      +\n     -+static void paths_and_oids_clear(struct paths_and_oids *po)\n     ++static void paths_and_oids_clear(struct hashmap *map)\n      +{\n     -+\tint i;\n     -+\tfor (i = 0; i < po->list.nr; i++) {\n     -+\t\toidset_clear(po->list.items[i].util);\n     -+\t\tfree(po->list.items[i].util);\n     ++\tstruct hashmap_iter iter;\n     ++\tstruct path_and_oids_entry *entry;\n     ++\thashmap_iter_init(map, &iter);\n     ++\n     ++\twhile ((entry = (struct path_and_oids_entry *)hashmap_iter_next(&iter))) {\n     ++\t\toidset_clear(&entry->set);\n     ++\t\tfree(entry->path);\n      +\t}\n      +\n     -+\tstring_list_clear(&po->list, 0);\n     ++\thashmap_free(map, 1);\n      +}\n      +\n     -+static void paths_and_oids_insert(struct paths_and_oids *po,\n     ++static void paths_and_oids_insert(struct hashmap *map,\n      +\t\t\t\t  const char *path,\n      +\t\t\t\t  const struct object_id *oid)\n      +{\n     -+\tstruct string_list_item *item = string_list_insert(&po->list, path);\n     -+\tstruct oidset *set;\n     ++\tint hash = strhash(path);\n     ++\tstruct path_and_oids_entry key;\n     ++\tstruct path_and_oids_entry *entry;\n     ++\n     ++\thashmap_entry_init(&key, hash);\n     ++\tkey.path = xstrdup(path);\n     ++\toidset_init(&key.set, 0);\n      +\n     -+\tif (!item->util) {\n     -+\t\tset = xcalloc(1, sizeof(struct oidset));\n     -+\t\toidset_init(set, 16);\n     -+\t\titem->util = set;\n     ++\tif (!(entry = (struct path_and_oids_entry *)hashmap_get(map, &key, NULL))) {\n     ++\t\tentry = xcalloc(1, sizeof(struct path_and_oids_entry));\n     ++\t\thashmap_entry_init(entry, hash);\n     ++\t\tentry->path = key.path;\n     ++\t\toidset_init(&entry->set, 16);\n     ++\t\thashmap_put(map, entry);\n      +\t} else {\n     -+\t\tset = item->util;\n     ++\t\tfree(key.path);\n      +\t}\n      +\n     -+\toidset_insert(set, oid);\n     ++\toidset_insert(&entry->set, oid);\n      +}\n      +\n      +static void add_children_by_path(struct repository *r,\n      +\t\t\t\t struct tree *tree,\n     -+\t\t\t\t struct paths_and_oids *po)\n     ++\t\t\t\t struct hashmap *map)\n      +{\n      +\tstruct tree_desc desc;\n      +\tstruct name_entry entry;\n     @@ -167,7 +199,7 @@\n      +\twhile (tree_entry(&desc, &entry)) {\n      +\t\tswitch (object_type(entry.mode)) {\n      +\t\tcase OBJ_TREE:\n     -+\t\t\tpaths_and_oids_insert(po, entry.path, entry.oid);\n     ++\t\t\tpaths_and_oids_insert(map, entry.path, entry.oid);\n      +\n      +\t\t\tif (tree->object.flags & UNINTERESTING) {\n      +\t\t\t\tstruct tree *child = lookup_tree(r, entry.oid);\n     @@ -194,9 +226,10 @@\n       void mark_trees_uninteresting_sparse(struct repository *r,\n       \t\t\t\t     struct oidset *set)\n       {\n     -+\tint i;\n      +\tunsigned has_interesting = 0, has_uninteresting = 0;\n     -+\tstruct paths_and_oids po;\n     ++\tstruct hashmap map;\n     ++\tstruct hashmap_iter map_iter;\n     ++\tstruct path_and_oids_entry *entry;\n       \tstruct object_id *oid;\n       \tstruct oidset_iter iter;\n       \n     @@ -222,25 +255,25 @@\n      +\t\t\thas_uninteresting = 1;\n      +\t\telse\n      +\t\t\thas_interesting = 1;\n     - \t}\n     ++\t}\n      +\n      +\t/* Do not walk unless we have both types of trees. */\n      +\tif (!has_uninteresting || !has_interesting)\n      +\t\treturn;\n      +\n     -+\tpaths_and_oids_init(&po);\n     ++\tpaths_and_oids_init(&map);\n      +\n      +\toidset_iter_init(set, &iter);\n      +\twhile ((oid = oidset_iter_next(&iter))) {\n      +\t\tstruct tree *tree = lookup_tree(r, oid);\n     -+\t\tadd_children_by_path(r, tree, &po);\n     -+\t}\n     ++\t\tadd_children_by_path(r, tree, &map);\n     + \t}\n      +\n     -+\tfor (i = 0; i < po.list.nr; i++)\n     -+\t\tmark_trees_uninteresting_sparse(\n     -+\t\t\tr, (struct oidset *)po.list.items[i].util);\n     ++\thashmap_iter_init(&map, &map_iter);\n     ++\twhile ((entry = hashmap_iter_next(&map_iter)))\n     ++\t\tmark_trees_uninteresting_sparse(r, &entry->set);\n      +\n     -+\tpaths_and_oids_clear(&po);\n     ++\tpaths_and_oids_clear(&map);\n       }\n       \n       struct commit_stack {\n 5:  33d2c04dd6 = 5:  f386f6c3c9 pack-objects: create pack.useSparse setting\n 6:  e4f29543ee = 6:  d011a9c1b1 pack-objects: create GIT_TEST_PACK_SPARSE\n\n-- \ngitgitgadget\n"},{"id":"365373","messageId":"817e30a287e12ce8e94ce41fcb969dd8ae53b9ce.1544822533.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 1/6] revision: add mark_tree_uninteresting_sparse","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-14T21:22:17Z","receivedAt":"2018-12-14T21:22:21Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nIn preparation for a new algorithm that walks fewer trees when\ncreating a pack from a set of revisions, create a method that\ntakes an oidset of tree oids and marks reachable objects as\nUNINTERESTING.\n\nThe current implementation uses the existing\nmark_tree_uninteresting to recursively walk the trees and blobs.\nThis will walk the same number of trees as the old mechanism.\n\nThere is one new assumption in this approach: we are also given\nthe oids of the interesting trees. This implementation does not\nuse those trees at the moment, but we will use them in a later\nrewrite of this method.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n revision.c | 25 +++++++++++++++++++++++++\n revision.h |  2 ++\n 2 files changed, 27 insertions(+)\n\ndiff --git a/revision.c b/revision.c\nindex 13e0519c02..f9eb6400f1 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -99,6 +99,31 @@ void mark_tree_uninteresting(struct repository *r, struct tree *tree)\n \tmark_tree_contents_uninteresting(r, tree);\n }\n \n+void mark_trees_uninteresting_sparse(struct repository *r,\n+\t\t\t\t     struct oidset *set)\n+{\n+\tstruct object_id *oid;\n+\tstruct oidset_iter iter;\n+\n+\toidset_iter_init(set, &iter);\n+\twhile ((oid = oidset_iter_next(&iter))) {\n+\t\tstruct tree *tree = lookup_tree(r, oid);\n+\n+\t\tif (!tree)\n+\t\t\tcontinue;\n+\n+\t\tif (tree->object.flags & UNINTERESTING) {\n+\t\t\t/*\n+\t\t\t * Remove the flag so the next call\n+\t\t\t * is not a no-op. The flag is added\n+\t\t\t * in mark_tree_unintersting().\n+\t\t\t */\n+\t\t\ttree->object.flags ^= UNINTERESTING;\n+\t\t\tmark_tree_uninteresting(r, tree);\n+\t\t}\n+\t}\n+}\n+\n struct commit_stack {\n \tstruct commit **items;\n \tsize_t nr, alloc;\ndiff --git a/revision.h b/revision.h\nindex 7987bfcd2e..f828e91ae9 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -67,6 +67,7 @@ struct rev_cmdline_info {\n #define REVISION_WALK_NO_WALK_SORTED 1\n #define REVISION_WALK_NO_WALK_UNSORTED 2\n \n+struct oidset;\n struct topo_walk_info;\n \n struct rev_info {\n@@ -327,6 +328,7 @@ void put_revision_mark(const struct rev_info *revs,\n \n void mark_parents_uninteresting(struct commit *commit);\n void mark_tree_uninteresting(struct repository *r, struct tree *tree);\n+void mark_trees_uninteresting_sparse(struct repository *r, struct oidset *set);\n \n void show_object_with_name(FILE *, struct object *, const char *);\n \n-- \ngitgitgadget\n\n"},{"id":"365374","messageId":"39dc89beb91ac12c94d13f7931a4d9ebc602681f.1544822533.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 2/6] list-objects: consume sparse tree walk","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-14T21:22:18Z","receivedAt":"2018-12-14T21:22:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen creating a pack-file using 'git pack-objects --revs' we provide\na list of interesting and uninteresting commits. For example, a push\noperation would make the local topic branch be interesting and the\nknown remote refs as uninteresting. We want to discover the set of\nnew objects to send to the server as a thin pack.\n\nWe walk these commits until we discover a frontier of commits such\nthat every commit walk starting at interesting commits ends in a root\ncommit or unintersting commit. We then need to discover which\nnon-commit objects are reachable from  uninteresting commits. This\ncommit walk is not changing during this series.\n\nThe mark_edges_uninteresting() method in list-objects.c iterates on\nthe commit list and does the following:\n\n* If the commit is UNINTERSTING, then mark its root tree and every\n  object it can reach as UNINTERESTING.\n\n* If the commit is interesting, then mark the root tree of every\n  UNINTERSTING parent (and all objects that tree can reach) as\n  UNINTERSTING.\n\nAt the very end, we repeat the process on every commit directly\ngiven to the revision walk from stdin. This helps ensure we properly\ncover shallow commits that otherwise were not included in the\nfrontier.\n\nThe logic to recursively follow trees is in the\nmark_tree_uninteresting() method in revision.c. The algorithm avoids\nduplicate work by not recursing into trees that are already marked\nUNINTERSTING.\n\nAdd a new 'sparse' option to the mark_edges_uninteresting() method\nthat performs this logic in a slightly new way. As we iterate over\nthe commits, we add all of the root trees to an oidset. Then, call\nmark_trees_uninteresting_sparse() on that oidset. Note that we\ninclude interesting trees in this process. The current implementation\nof mark_trees_unintersting_sparse() will walk the same trees as\nthe old logic, but this will be replaced in a later change.\n\nThe sparse option is not used by any callers at the moment, but\nwill be wired to 'git pack-objects' in the next change.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n bisect.c               |  2 +-\n builtin/pack-objects.c |  2 +-\n builtin/rev-list.c     |  2 +-\n http-push.c            |  2 +-\n list-objects.c         | 70 +++++++++++++++++++++++++++++++++++-------\n list-objects.h         |  4 ++-\n 6 files changed, 66 insertions(+), 16 deletions(-)\n\ndiff --git a/bisect.c b/bisect.c\nindex 487675c672..842f8b4b8f 100644\n--- a/bisect.c\n+++ b/bisect.c\n@@ -656,7 +656,7 @@ static void bisect_common(struct rev_info *revs)\n \tif (prepare_revision_walk(revs))\n \t\tdie(\"revision walk setup failed\");\n \tif (revs->tree_objects)\n-\t\tmark_edges_uninteresting(revs, NULL);\n+\t\tmark_edges_uninteresting(revs, NULL, 0);\n }\n \n static void exit_if_skipped_commits(struct commit_list *tried,\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 411aefd687..5f70d840a7 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3135,7 +3135,7 @@ static void get_object_list(int ac, const char **av)\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n-\tmark_edges_uninteresting(&revs, show_edge);\n+\tmark_edges_uninteresting(&revs, show_edge, 0);\n \n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 2880ed37e3..9663cbfae0 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -543,7 +543,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \tif (revs.tree_objects)\n-\t\tmark_edges_uninteresting(&revs, show_edge);\n+\t\tmark_edges_uninteresting(&revs, show_edge, 0);\n \n \tif (bisect_list) {\n \t\tint reaches, all;\ndiff --git a/http-push.c b/http-push.c\nindex cd48590912..ea52d6f9f6 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -1933,7 +1933,7 @@ int cmd_main(int argc, const char **argv)\n \t\tpushing = 0;\n \t\tif (prepare_revision_walk(&revs))\n \t\t\tdie(\"revision walk setup failed\");\n-\t\tmark_edges_uninteresting(&revs, NULL);\n+\t\tmark_edges_uninteresting(&revs, NULL, 0);\n \t\tobjects_to_send = get_delta(&revs, ref_lock);\n \t\tfinish_all_active_slots();\n \ndiff --git a/list-objects.c b/list-objects.c\nindex c41cc80db5..fb728f7842 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -222,25 +222,73 @@ static void mark_edge_parents_uninteresting(struct commit *commit,\n \t}\n }\n \n-void mark_edges_uninteresting(struct rev_info *revs, show_edge_fn show_edge)\n+static void add_edge_parents(struct commit *commit,\n+\t\t\t     struct rev_info *revs,\n+\t\t\t     show_edge_fn show_edge,\n+\t\t\t     struct oidset *set)\n+{\n+\tstruct commit_list *parents;\n+\n+\tfor (parents = commit->parents; parents; parents = parents->next) {\n+\t\tstruct commit *parent = parents->item;\n+\t\tstruct tree *tree = get_commit_tree(parent);\n+\n+\t\tif (!tree)\n+\t\t\tcontinue;\n+\n+\t\toidset_insert(set, &tree->object.oid);\n+\n+\t\tif (!(parent->object.flags & UNINTERESTING))\n+\t\t\tcontinue;\n+\t\ttree->object.flags |= UNINTERESTING;\n+\n+\t\tif (revs->edge_hint && !(parent->object.flags & SHOWN)) {\n+\t\t\tparent->object.flags |= SHOWN;\n+\t\t\tshow_edge(parent);\n+\t\t}\n+\t}\n+}\n+\n+void mark_edges_uninteresting(struct rev_info *revs,\n+\t\t\t      show_edge_fn show_edge,\n+\t\t\t      int sparse)\n {\n \tstruct commit_list *list;\n \tint i;\n \n-\tfor (list = revs->commits; list; list = list->next) {\n-\t\tstruct commit *commit = list->item;\n+\tif (sparse) {\n+\t\tstruct oidset set;\n+\t\toidset_init(&set, 16);\n \n-\t\tif (commit->object.flags & UNINTERESTING) {\n-\t\t\tmark_tree_uninteresting(revs->repo,\n-\t\t\t\t\t\tget_commit_tree(commit));\n-\t\t\tif (revs->edge_hint_aggressive && !(commit->object.flags & SHOWN)) {\n-\t\t\t\tcommit->object.flags |= SHOWN;\n-\t\t\t\tshow_edge(commit);\n+\t\tfor (list = revs->commits; list; list = list->next) {\n+\t\t\tstruct commit *commit = list->item;\n+\t\t\tstruct tree *tree = get_commit_tree(commit);\n+\n+\t\t\tif (commit->object.flags & UNINTERESTING)\n+\t\t\t\ttree->object.flags |= UNINTERESTING;\n+\n+\t\t\toidset_insert(&set, &tree->object.oid);\n+\t\t\tadd_edge_parents(commit, revs, show_edge, &set);\n+\t\t}\n+\n+\t\tmark_trees_uninteresting_sparse(revs->repo, &set);\n+\t\toidset_clear(&set);\n+\t} else {\n+\t\tfor (list = revs->commits; list; list = list->next) {\n+\t\t\tstruct commit *commit = list->item;\n+\t\t\tif (commit->object.flags & UNINTERESTING) {\n+\t\t\t\tmark_tree_uninteresting(revs->repo,\n+\t\t\t\t\t\t\tget_commit_tree(commit));\n+\t\t\t\tif (revs->edge_hint_aggressive && !(commit->object.flags & SHOWN)) {\n+\t\t\t\t\tcommit->object.flags |= SHOWN;\n+\t\t\t\t\tshow_edge(commit);\n+\t\t\t\t}\n+\t\t\t\tcontinue;\n \t\t\t}\n-\t\t\tcontinue;\n+\t\t\tmark_edge_parents_uninteresting(commit, revs, show_edge);\n \t\t}\n-\t\tmark_edge_parents_uninteresting(commit, revs, show_edge);\n \t}\n+\n \tif (revs->edge_hint_aggressive) {\n \t\tfor (i = 0; i < revs->cmdline.nr; i++) {\n \t\t\tstruct object *obj = revs->cmdline.rev[i].item;\ndiff --git a/list-objects.h b/list-objects.h\nindex ad40762926..a952680e46 100644\n--- a/list-objects.h\n+++ b/list-objects.h\n@@ -10,7 +10,9 @@ typedef void (*show_object_fn)(struct object *, const char *, void *);\n void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);\n \n typedef void (*show_edge_fn)(struct commit *);\n-void mark_edges_uninteresting(struct rev_info *, show_edge_fn);\n+void mark_edges_uninteresting(struct rev_info *revs,\n+\t\t\t      show_edge_fn show_edge,\n+\t\t\t      int sparse);\n \n struct oidset;\n struct list_objects_filter_options;\n-- \ngitgitgadget\n\n"},{"id":"365375","messageId":"ab733daff5398fd07ff051c323f51b70efbc2e57.1544822533.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 3/6] pack-objects: add --sparse option","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-14T21:22:19Z","receivedAt":"2018-12-14T21:22:23Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nAdd a '--sparse' option flag to the pack-objects builtin. This\nallows the user to specify that they want to use the new logic\nfor walking trees. This logic currently does not differ from the\nexisting output, but will in a later change.\n\nCreate a new test script, t5322-pack-objects-sparse.sh, to ensure\nthe object list that is selected matches what we expect. When we\nupdate the logic to walk in a sparse fashion, the final test will\nbe updated to show the extra objects that are added.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n Documentation/git-pack-objects.txt |  11 ++-\n builtin/pack-objects.c             |   5 +-\n t/t5322-pack-objects-sparse.sh     | 115 +++++++++++++++++++++++++++++\n 3 files changed, 129 insertions(+), 2 deletions(-)\n create mode 100755 t/t5322-pack-objects-sparse.sh\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 40c825c381..e45f3e680d 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -14,7 +14,7 @@ SYNOPSIS\n \t[--local] [--incremental] [--window=<n>] [--depth=<n>]\n \t[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\n \t[--stdout [--filter=<filter-spec>] | base-name]\n-\t[--shallow] [--keep-true-parents] < object-list\n+\t[--shallow] [--keep-true-parents] [--sparse] < object-list\n \n \n DESCRIPTION\n@@ -196,6 +196,15 @@ depth is 4095.\n \tAdd --no-reuse-object if you want to force a uniform compression\n \tlevel on all data no matter the source.\n \n+--sparse::\n+\tUse the \"sparse\" algorithm to determine which objects to include in\n+\tthe pack, when combined with the \"--revs\" option. This algorithm\n+\tonly walks trees that appear in paths that introduce new objects.\n+\tThis can have significant performance benefits when computing\n+\ta pack to send a small change. However, it is possible that extra\n+\tobjects are added to the pack-file if the included commits contain\n+\tcertain types of direct renames.\n+\n --thin::\n \tCreate a \"thin\" pack by omitting the common objects between a\n \tsender and a receiver in order to reduce network transfer. This\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 5f70d840a7..7d5b0735e3 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -84,6 +84,7 @@ static unsigned long pack_size_limit;\n static int depth = 50;\n static int delta_search_threads;\n static int pack_to_stdout;\n+static int sparse;\n static int thin;\n static int num_preferred_base;\n static struct progress *progress_state;\n@@ -3135,7 +3136,7 @@ static void get_object_list(int ac, const char **av)\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n-\tmark_edges_uninteresting(&revs, show_edge, 0);\n+\tmark_edges_uninteresting(&revs, show_edge, sparse);\n \n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\n@@ -3292,6 +3293,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t{ OPTION_CALLBACK, 0, \"unpack-unreachable\", NULL, N_(\"time\"),\n \t\t  N_(\"unpack unreachable objects newer than <time>\"),\n \t\t  PARSE_OPT_OPTARG, option_parse_unpack_unreachable },\n+\t\tOPT_BOOL(0, \"sparse\", &sparse,\n+\t\t\t N_(\"use the sparse reachability algorithm\")),\n \t\tOPT_BOOL(0, \"thin\", &thin,\n \t\t\t N_(\"create thin packs\")),\n \t\tOPT_BOOL(0, \"shallow\", &shallow,\ndiff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\nnew file mode 100755\nindex 0000000000..81f6805bc3\n--- /dev/null\n+++ b/t/t5322-pack-objects-sparse.sh\n@@ -0,0 +1,115 @@\n+#!/bin/sh\n+\n+test_description='pack-objects object selection using sparse algorithm'\n+. ./test-lib.sh\n+\n+test_expect_success 'setup repo' '\n+\ttest_commit initial &&\n+\tfor i in $(test_seq 1 3)\n+\tdo\n+\t\tmkdir f$i &&\n+\t\tfor j in $(test_seq 1 3)\n+\t\tdo\n+\t\t\tmkdir f$i/f$j &&\n+\t\t\techo $j >f$i/f$j/data.txt\n+\t\tdone\n+\tdone &&\n+\tgit add . &&\n+\tgit commit -m \"Initialized trees\" &&\n+\tfor i in $(test_seq 1 3)\n+\tdo\n+\t\tgit checkout -b topic$i master &&\n+\t\techo change-$i >f$i/f$i/data.txt &&\n+\t\tgit commit -a -m \"Changed f$i/f$i/data.txt\"\n+\tdone &&\n+\tcat >packinput.txt <<-EOF &&\n+\ttopic1\n+\t^topic2\n+\t^topic3\n+\tEOF\n+\tgit rev-parse\t\t\t\\\n+\t\ttopic1\t\t\t\\\n+\t\ttopic1^{tree}\t\t\\\n+\t\ttopic1:f1\t\t\\\n+\t\ttopic1:f1/f1\t\t\\\n+\t\ttopic1:f1/f1/data.txt | sort >expect_objects.txt\n+'\n+\n+test_expect_success 'non-sparse pack-objects' '\n+\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n+\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n+\ttest_cmp expect_objects.txt nonsparse_objects.txt\n+'\n+\n+test_expect_success 'sparse pack-objects' '\n+\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_objects.txt sparse_objects.txt\n+'\n+\n+# Demonstrate that both algorithms send \"extra\" objects because\n+# they are not in the frontier.\n+\n+test_expect_success 'duplicate a folder from f3 and commit to topic1' '\n+\tgit checkout topic1 &&\n+\techo change-3 >f3/f3/data.txt &&\n+\tgit commit -a -m \"Changed f3/f3/data.txt\" &&\n+\tgit rev-parse\t\t\t\\\n+\t\ttopic1~1\t\t\\\n+\t\ttopic1~1^{tree}\t\t\\\n+\t\ttopic1^{tree}\t\t\\\n+\t\ttopic1\t\t\t\\\n+\t\ttopic1:f1\t\t\\\n+\t\ttopic1:f1/f1\t\t\\\n+\t\ttopic1:f1/f1/data.txt\t\\\n+\t\ttopic1:f3\t\t\\\n+\t\ttopic1:f3/f3\t\t\\\n+\t\ttopic1:f3/f3/data.txt | sort >expect_objects.txt\n+'\n+\n+test_expect_success 'non-sparse pack-objects' '\n+\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n+\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n+\ttest_cmp expect_objects.txt nonsparse_objects.txt\n+'\n+\n+test_expect_success 'sparse pack-objects' '\n+\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_objects.txt sparse_objects.txt\n+'\n+\n+test_expect_success 'duplicate a folder from f1 into f3' '\n+\tmkdir f3/f4 &&\n+\tcp -r f1/f1/* f3/f4 &&\n+\tgit add f3/f4 &&\n+\tgit commit -m \"Copied f1/f1 to f3/f4\" &&\n+\tcat >packinput.txt <<-EOF &&\n+\ttopic1\n+\t^topic1~1\n+\tEOF\n+\tgit rev-parse\t\t\\\n+\t\ttopic1\t\t\\\n+\t\ttopic1^{tree}\t\\\n+\t\ttopic1:f3 | sort >expect_objects.txt\n+'\n+\n+test_expect_success 'non-sparse pack-objects' '\n+\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n+\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n+\ttest_cmp expect_objects.txt nonsparse_objects.txt\n+'\n+\n+test_expect_success 'sparse pack-objects' '\n+\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_objects.txt sparse_objects.txt\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"365376","messageId":"c44172c35ece7aafec02c7f3c8438ccca4f69023.1544822533.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 4/6] revision: implement sparse algorithm","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-14T21:22:21Z","receivedAt":"2018-12-14T21:22:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen enumerating objects to place in a pack-file during 'git\npack-objects --revs', we discover the \"frontier\" of commits\nthat we care about and the boundary with commit we find\nuninteresting. From that point, we walk trees to discover which\ntrees and blobs are uninteresting. Finally, we walk trees from the\ninteresting commits to find the interesting objects that are\nplaced in the pack.\n\nThis commit introduces a new, \"sparse\" way to discover the\nuninteresting trees. We use the perspective of a single user trying\nto push their topic to a large repository. That user likely changed\na very small fraction of the paths in their working directory, but\nwe spend a lot of time walking all reachable trees.\n\nThe way to switch the logic to work in this sparse way is to start\ncaring about which paths introduce new trees. While it is not\npossible to generate a diff between the frontier boundary and all\nof the interesting commits, we can simulate that behavior by\ninspecting all of the root trees as a whole, then recursing down\nto the set of trees at each path.\n\nWe already had taken the first step by passing an oidset to\nmark_trees_uninteresting_sparse(). We now create a dictionary\nwhose keys are paths and values are oidsets. We consider the set\nof trees that appear at each path. While we inspect a tree, we\nadd its subtrees to the oidsets corresponding to the tree entry's\npath. We also mark trees as UNINTERESTING if the tree we are\nparsing is UNINTERESTING.\n\nTo actually improve the peformance, we need to terminate our\nrecursion. If the oidset contains only UNINTERESTING trees, then\nwe do not continue the recursion. This avoids walking trees that\nare likely to not be reachable from interesting trees. If the\noidset contains only interesting trees, then we will walk these\ntrees in the final stage that collects the intersting objects to\nplace in the pack. Thus, we only recurse if the oidset contains\nboth interesting and UNINITERESTING trees.\n\nThere are a few ways that this is not a universally better option.\n\nFirst, we can pack extra objects. If someone copies a subtree\nfrom one tree to another, the first tree will appear UNINTERESTING\nand we will not recurse to see that the subtree should also be\nUNINTERESTING. We will walk the new tree and see the subtree as\na \"new\" object and add it to the pack. We add a test case that\ndemonstrates this as a way to prove that the --sparse option is\nactually working.\n\nSecond, we can have extra memory pressure. If instead of being a\nsingle user pushing a small topic we are a server sending new\nobjects from across the entire working directory, then we will\ngain very little (the recursion will rarely terminate early) but\nwill spend extra time maintaining the path-oidset dictionaries.\n\nDespite these potential drawbacks, the benefits of the algorithm\nare clear. By adding a counter to 'add_children_by_path' and\n'mark_tree_contents_uninteresting', I measured the number of\nparsed trees for the two algorithms in a variety of repos.\n\nFor git.git, I used the following input:\n\n\tv2.19.0\n\t^v2.19.0~10\n\n Objects to pack: 550\nWalked (old alg): 282\nWalked (new alg): 130\n\nFor the Linux repo, I used the following input:\n\n\tv4.18\n\t^v4.18~10\n\n Objects to pack:   518\nWalked (old alg): 4,836\nWalked (new alg):   188\n\nThe two repos above are rather \"wide and flat\" compared to\nother repos that I have used in the past. As a comparison,\nI tested an old topic branch in the Azure DevOps repo, which\nhas a much deeper folder structure than the Linux repo.\n\n Objects to pack:    220\nWalked (old alg): 22,804\nWalked (new alg):    129\n\nI used the number of walked trees the main metric above because\nit is consistent across multiple runs. When I ran my tests, the\nperformance of the pack-objects command with the same options\ncould change the end-to-end time by 10x depending on the file\nsystem being warm. However, by repeating the same test on repeat\nI could get more consistent timing results. The git.git and\nLinux tests were too fast overall (less than 0.5s) to measure\nan end-to-end difference. The Azure DevOps case was slow enough\nto see the time improve from 15s to 1s in the warm case. The\ncold case was 90s to 9s in my testing.\n\nThese improvements will have even larger benefits in the super-\nlarge Windows repository. In our experiments, we see the\n\"Enumerate objects\" phase of pack-objects taking 60-80% of the\nend-to-end time of non-trivial pushes, taking longer than the\nnetwork time to send the pack and the server time to verify the\npack.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n revision.c                     | 139 ++++++++++++++++++++++++++++++---\n t/t5322-pack-objects-sparse.sh |  21 +++--\n 2 files changed, 144 insertions(+), 16 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex f9eb6400f1..63bf6230dc 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -27,6 +27,7 @@\n #include \"commit-reach.h\"\n #include \"commit-graph.h\"\n #include \"prio-queue.h\"\n+#include \"hashmap.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -99,29 +100,147 @@ void mark_tree_uninteresting(struct repository *r, struct tree *tree)\n \tmark_tree_contents_uninteresting(r, tree);\n }\n \n+struct path_and_oids_entry {\n+\tstruct hashmap_entry ent;\n+\tchar *path;\n+\tstruct oidset set;\n+};\n+\n+static int path_and_oids_cmp(const void *hashmap_cmp_fn_data,\n+\t\t\t     const struct path_and_oids_entry *e1,\n+\t\t\t     const struct path_and_oids_entry *e2,\n+\t\t\t     const void *keydata)\n+{\n+\treturn strcmp(e1->path, e2->path);\n+}\n+\n+int map_flags = 0;\n+static void paths_and_oids_init(struct hashmap *map)\n+{\n+\thashmap_init(map, (hashmap_cmp_fn) path_and_oids_cmp, &map_flags, 0);\n+}\n+\n+static void paths_and_oids_clear(struct hashmap *map)\n+{\n+\tstruct hashmap_iter iter;\n+\tstruct path_and_oids_entry *entry;\n+\thashmap_iter_init(map, &iter);\n+\n+\twhile ((entry = (struct path_and_oids_entry *)hashmap_iter_next(&iter))) {\n+\t\toidset_clear(&entry->set);\n+\t\tfree(entry->path);\n+\t}\n+\n+\thashmap_free(map, 1);\n+}\n+\n+static void paths_and_oids_insert(struct hashmap *map,\n+\t\t\t\t  const char *path,\n+\t\t\t\t  const struct object_id *oid)\n+{\n+\tint hash = strhash(path);\n+\tstruct path_and_oids_entry key;\n+\tstruct path_and_oids_entry *entry;\n+\n+\thashmap_entry_init(&key, hash);\n+\tkey.path = xstrdup(path);\n+\toidset_init(&key.set, 0);\n+\n+\tif (!(entry = (struct path_and_oids_entry *)hashmap_get(map, &key, NULL))) {\n+\t\tentry = xcalloc(1, sizeof(struct path_and_oids_entry));\n+\t\thashmap_entry_init(entry, hash);\n+\t\tentry->path = key.path;\n+\t\toidset_init(&entry->set, 16);\n+\t\thashmap_put(map, entry);\n+\t} else {\n+\t\tfree(key.path);\n+\t}\n+\n+\toidset_insert(&entry->set, oid);\n+}\n+\n+static void add_children_by_path(struct repository *r,\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct hashmap *map)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\n+\tif (!tree)\n+\t\treturn;\n+\n+\tif (parse_tree_gently(tree, 1) < 0)\n+\t\treturn;\n+\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tswitch (object_type(entry.mode)) {\n+\t\tcase OBJ_TREE:\n+\t\t\tpaths_and_oids_insert(map, entry.path, entry.oid);\n+\n+\t\t\tif (tree->object.flags & UNINTERESTING) {\n+\t\t\t\tstruct tree *child = lookup_tree(r, entry.oid);\n+\t\t\t\tif (child)\n+\t\t\t\t\tchild->object.flags |= UNINTERESTING;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\tcase OBJ_BLOB:\n+\t\t\tif (tree->object.flags & UNINTERESTING) {\n+\t\t\t\tstruct blob *child = lookup_blob(r, entry.oid);\n+\t\t\t\tif (child)\n+\t\t\t\t\tchild->object.flags |= UNINTERESTING;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\t/* Subproject commit - not in this repository */\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfree_tree_buffer(tree);\n+}\n+\n void mark_trees_uninteresting_sparse(struct repository *r,\n \t\t\t\t     struct oidset *set)\n {\n+\tunsigned has_interesting = 0, has_uninteresting = 0;\n+\tstruct hashmap map;\n+\tstruct hashmap_iter map_iter;\n+\tstruct path_and_oids_entry *entry;\n \tstruct object_id *oid;\n \tstruct oidset_iter iter;\n \n \toidset_iter_init(set, &iter);\n-\twhile ((oid = oidset_iter_next(&iter))) {\n+\twhile ((!has_interesting || !has_uninteresting) &&\n+\t       (oid = oidset_iter_next(&iter))) {\n \t\tstruct tree *tree = lookup_tree(r, oid);\n \n \t\tif (!tree)\n \t\t\tcontinue;\n \n-\t\tif (tree->object.flags & UNINTERESTING) {\n-\t\t\t/*\n-\t\t\t * Remove the flag so the next call\n-\t\t\t * is not a no-op. The flag is added\n-\t\t\t * in mark_tree_unintersting().\n-\t\t\t */\n-\t\t\ttree->object.flags ^= UNINTERESTING;\n-\t\t\tmark_tree_uninteresting(r, tree);\n-\t\t}\n+\t\tif (tree->object.flags & UNINTERESTING)\n+\t\t\thas_uninteresting = 1;\n+\t\telse\n+\t\t\thas_interesting = 1;\n+\t}\n+\n+\t/* Do not walk unless we have both types of trees. */\n+\tif (!has_uninteresting || !has_interesting)\n+\t\treturn;\n+\n+\tpaths_and_oids_init(&map);\n+\n+\toidset_iter_init(set, &iter);\n+\twhile ((oid = oidset_iter_next(&iter))) {\n+\t\tstruct tree *tree = lookup_tree(r, oid);\n+\t\tadd_children_by_path(r, tree, &map);\n \t}\n+\n+\thashmap_iter_init(&map, &map_iter);\n+\twhile ((entry = hashmap_iter_next(&map_iter)))\n+\t\tmark_trees_uninteresting_sparse(r, &entry->set);\n+\n+\tpaths_and_oids_clear(&map);\n }\n \n struct commit_stack {\ndiff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\nindex 81f6805bc3..45dba6e014 100755\n--- a/t/t5322-pack-objects-sparse.sh\n+++ b/t/t5322-pack-objects-sparse.sh\n@@ -83,22 +83,25 @@ test_expect_success 'sparse pack-objects' '\n \ttest_cmp expect_objects.txt sparse_objects.txt\n '\n \n+# Demonstrate that the algorithms differ when we copy a tree wholesale\n+# from one folder to another.\n+\n test_expect_success 'duplicate a folder from f1 into f3' '\n \tmkdir f3/f4 &&\n \tcp -r f1/f1/* f3/f4 &&\n \tgit add f3/f4 &&\n \tgit commit -m \"Copied f1/f1 to f3/f4\" &&\n-\tcat >packinput.txt <<-EOF &&\n+\tcat >packinput.txt <<-EOF\n \ttopic1\n \t^topic1~1\n \tEOF\n-\tgit rev-parse\t\t\\\n-\t\ttopic1\t\t\\\n-\t\ttopic1^{tree}\t\\\n-\t\ttopic1:f3 | sort >expect_objects.txt\n '\n \n test_expect_success 'non-sparse pack-objects' '\n+\tgit rev-parse\t\t\t\\\n+\t\ttopic1\t\t\t\\\n+\t\ttopic1^{tree}\t\t\\\n+\t\ttopic1:f3 | sort >expect_objects.txt &&\n \tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n \tgit index-pack -o nonsparse.idx nonsparse.pack &&\n \tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n@@ -106,10 +109,16 @@ test_expect_success 'non-sparse pack-objects' '\n '\n \n test_expect_success 'sparse pack-objects' '\n+\tgit rev-parse\t\t\t\\\n+\t\ttopic1\t\t\t\\\n+\t\ttopic1^{tree}\t\t\\\n+\t\ttopic1:f3\t\t\\\n+\t\ttopic1:f3/f4\t\t\\\n+\t\ttopic1:f3/f4/data.txt | sort >expect_sparse_objects.txt &&\n \tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n \tgit index-pack -o sparse.idx sparse.pack &&\n \tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n-\ttest_cmp expect_objects.txt sparse_objects.txt\n+\ttest_cmp expect_sparse_objects.txt sparse_objects.txt\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"365377","messageId":"f386f6c3c9dba2ef699a6605e4214aa723ec405f.1544822533.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 5/6] pack-objects: create pack.useSparse setting","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-14T21:22:22Z","receivedAt":"2018-12-14T21:22:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nThe '--sparse' flag in 'git pack-objects' changes the algorithm\nused to enumerate objects to one that is faster for individual\nusers pushing new objects that change only a small cone of the\nworking directory. The sparse algorithm is not recommended for a\nserver, which likely sends new objects that appear across the\nentire working directory.\n\nCreate a 'pack.useSparse' setting that enables this new algorithm.\nThis allows 'git push' to use this algorithm without passing a\n'--sparse' flag all the way through four levels of run_command()\ncalls.\n\nIf the '--no-sparse' flag is set, then this config setting is\noverridden.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n Documentation/config/pack.txt  |  9 +++++++++\n builtin/pack-objects.c         |  4 ++++\n t/t5322-pack-objects-sparse.sh | 15 +++++++++++++++\n 3 files changed, 28 insertions(+)\n\ndiff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt\nindex edac75c83f..425c73aa52 100644\n--- a/Documentation/config/pack.txt\n+++ b/Documentation/config/pack.txt\n@@ -105,6 +105,15 @@ pack.useBitmaps::\n \ttrue. You should not generally need to turn this off unless\n \tyou are debugging pack bitmaps.\n \n+pack.useSparse::\n+\tWhen true, git will default to using the '--sparse' option in\n+\t'git pack-objects' when the '--revs' option is present. This\n+\talgorithm only walks trees that appear in paths that introduce new\n+\tobjects. This can have significant performance benefits when\n+\tcomputing a pack to send a small change. However, it is possible\n+\tthat extra objects are added to the pack-file if the included\n+\tcommits contain certain types of direct renames.\n+\n pack.writeBitmaps (deprecated)::\n \tThis is a deprecated synonym for `repack.writeBitmaps`.\n \ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7d5b0735e3..124b1bafc4 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2711,6 +2711,10 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(k, \"pack.usesparse\")) {\n+\t\tsparse = git_config_bool(k, v);\n+\t\treturn 0;\n+\t}\n \tif (!strcmp(k, \"pack.threads\")) {\n \t\tdelta_search_threads = git_config_int(k, v);\n \t\tif (delta_search_threads < 0)\ndiff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\nindex 45dba6e014..8f5699bd91 100755\n--- a/t/t5322-pack-objects-sparse.sh\n+++ b/t/t5322-pack-objects-sparse.sh\n@@ -121,4 +121,19 @@ test_expect_success 'sparse pack-objects' '\n \ttest_cmp expect_sparse_objects.txt sparse_objects.txt\n '\n \n+test_expect_success 'pack.useSparse enables algorithm' '\n+\tgit config pack.useSparse true &&\n+\tgit pack-objects --stdout --revs <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_sparse_objects.txt sparse_objects.txt\n+'\n+\n+test_expect_success 'pack.useSparse overridden' '\n+\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_objects.txt sparse_objects.txt\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"365378","messageId":"d011a9c1b1c197fa1a77e924a50463a984efc0d1.1544822533.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 6/6] pack-objects: create GIT_TEST_PACK_SPARSE","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-12-14T21:22:23Z","receivedAt":"2018-12-14T21:22:29Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nCreate a test variable GIT_TEST_PACK_SPARSE to enable the sparse\nobject walk algorithm by default during the test suite. Enabling\nthis variable ensures coverage in many interesting cases, such as\nshallow clones, partial clones, and missing objects.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n builtin/pack-objects.c         | 1 +\n t/README                       | 4 ++++\n t/t5322-pack-objects-sparse.sh | 6 +++---\n 3 files changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 124b1bafc4..507d381153 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3331,6 +3331,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \n \tread_replace_refs = 0;\n \n+\tsparse = git_env_bool(\"GIT_TEST_PACK_SPARSE\", 0);\n \treset_pack_idx_option(&pack_idx_opts);\n \tgit_config(git_pack_config, NULL);\n \ndiff --git a/t/README b/t/README\nindex 28711cc508..8b6dfe1864 100644\n--- a/t/README\n+++ b/t/README\n@@ -342,6 +342,10 @@ GIT_TEST_INDEX_VERSION=<n> exercises the index read/write code path\n for the index version specified.  Can be set to any valid version\n (currently 2, 3, or 4).\n \n+GIT_TEST_PACK_SPARSE=<boolean> if enabled will default the pack-objects\n+builtin to use the sparse object walk. This can still be overridden by\n+the --no-sparse command-line argument.\n+\n GIT_TEST_PRELOAD_INDEX=<boolean> exercises the preload-index code path\n by overriding the minimum number of cache entries required per thread.\n \ndiff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\nindex 8f5699bd91..e8cf41d1c6 100755\n--- a/t/t5322-pack-objects-sparse.sh\n+++ b/t/t5322-pack-objects-sparse.sh\n@@ -36,7 +36,7 @@ test_expect_success 'setup repo' '\n '\n \n test_expect_success 'non-sparse pack-objects' '\n-\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >nonsparse.pack &&\n \tgit index-pack -o nonsparse.idx nonsparse.pack &&\n \tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n \ttest_cmp expect_objects.txt nonsparse_objects.txt\n@@ -70,7 +70,7 @@ test_expect_success 'duplicate a folder from f3 and commit to topic1' '\n '\n \n test_expect_success 'non-sparse pack-objects' '\n-\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >nonsparse.pack &&\n \tgit index-pack -o nonsparse.idx nonsparse.pack &&\n \tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n \ttest_cmp expect_objects.txt nonsparse_objects.txt\n@@ -102,7 +102,7 @@ test_expect_success 'non-sparse pack-objects' '\n \t\ttopic1\t\t\t\\\n \t\ttopic1^{tree}\t\t\\\n \t\ttopic1:f3 | sort >expect_objects.txt &&\n-\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >nonsparse.pack &&\n \tgit index-pack -o nonsparse.idx nonsparse.pack &&\n \tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n \ttest_cmp expect_objects.txt nonsparse_objects.txt\n-- \ngitgitgadget\n"},{"id":"365390","messageId":"87efaj1y77.fsf@evledraar.gmail.com","threadId":"49914","inReplyTo":"c44172c35ece7aafec02c7f3c8438ccca4f69023.1544822533.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 4/6] revision: implement sparse algorithm","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-12-14T23:32:44Z","receivedAt":"2018-12-14T23:32:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Dec 14 2018, Derrick Stolee via GitGitGadget wrote:\n\n> Despite these potential drawbacks, the benefits of the algorithm\n> are clear. By adding a counter to 'add_children_by_path' and\n> 'mark_tree_contents_uninteresting', I measured the number of\n> parsed trees for the two algorithms in a variety of repos.\n\nAh, so we take one or the other. I tested this with:\n\n    diff --git a/revision.c b/revision.c\n    index 63bf6230dc..e9c67aa550 100644\n    --- a/revision.c\n    +++ b/revision.c\n    @@ -61,6 +61,8 @@ static void mark_tree_contents_uninteresting(struct repository *r,\n            struct tree_desc desc;\n            struct name_entry entry;\n\n    +       fprintf(stderr, \"MTCU\\n\");\n    +\n            if (parse_tree_gently(tree, 1) < 0)\n                    return;\n\n    @@ -166,6 +168,8 @@ static void add_children_by_path(struct repository *r,\n            struct tree_desc desc;\n            struct name_entry entry;\n\n    +       fprintf(stderr, \"ACBP\\n\");\n    +\n            if (!tree)\n                    return;\n\nAnd tried testing my pushing this branch to my git.git:\n\n    $ for v in true false; do git push --delete avar push/sparse; ./git -c pack.usesparse=$v --exec-path=$PWD push avar HEAD 2>&1 | grep -e MTCU -e ACBP | uniq -c; done\n    To github.com:avar/git.git\n     - [deleted]               push/sparse\n         22 ACBP\n    To github.com:avar/git.git\n     - [deleted]               push/sparse\n        184 MTCU\n\nBut snipping around a bit from the commit message:\n\n>When enumerating objects to place in a pack-file during 'git\n>pack-objects --revs', we discover the \"frontier\" of commits\n>that we care about and the boundary with commit we find\n>uninteresting. From that point, we walk trees to discover which\n>trees and blobs are uninteresting. Finally, we walk trees from the\n>interesting commits to find the interesting objects that are\n>placed in the pack.\n>[...]\n>These improvements will have even larger benefits in the super-\n>large Windows repository. In our experiments, we see the\n>\"Enumerate objects\" phase of pack-objects taking 60-80% of the\n>end-to-end time of non-trivial pushes, taking longer than the\n>network time to send the pack and the server time to verify the\n>pack.\n\nIf I change my monkeypatch to:\n\n    diff --git a/revision.c b/revision.c\n    index 63bf6230dc..9171ca27c5 100644\n    --- a/revision.c\n    +++ b/revision.c\n    @@ -63,0 +64,3 @@ static void mark_tree_contents_uninteresting(struct repository *r,\n    +       fprintf(stderr, \"MTCU\\n\");\n    +       sleep(1);\n    +\n    @@ -168,0 +172,3 @@ static void add_children_by_path(struct repository *r,\n    +       fprintf(stderr, \"ACBP\\n\");\n    +       sleep(1);\n    +\n\nWe spend a long time printing those out before we ever get to\n\"Enumerating objects\".\n\nWhich was where I was trying to test this, i.e. is this a lot of work we\nperform before we print out the progress bar, and regardless of this\noptimization should have other progress output there, so we can see this\ntime we're spending on this?\n"},{"id":"365478","messageId":"867aa5c3-60e0-2467-795a-40aac58f306b@gmail.com","threadId":"49914","inReplyTo":"87efaj1y77.fsf@evledraar.gmail.com","subject":"Re: [PATCH v4 4/6] revision: implement sparse algorithm","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-12-17T14:20:20Z","receivedAt":"2018-12-17T14:20:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/14/2018 6:32 PM, Ævar Arnfjörð Bjarmason wrote:\n> On Fri, Dec 14 2018, Derrick Stolee via GitGitGadget wrote:\n>\n>> Despite these potential drawbacks, the benefits of the algorithm\n>> are clear. By adding a counter to 'add_children_by_path' and\n>> 'mark_tree_contents_uninteresting', I measured the number of\n>> parsed trees for the two algorithms in a variety of repos.\n> We spend a long time printing those out before we ever get to\n> \"Enumerating objects\".\n>\n> Which was where I was trying to test this, i.e. is this a lot of work we\n> perform before we print out the progress bar, and regardless of this\n> optimization should have other progress output there, so we can see this\n> time we're spending on this?\n\nIt is true that part of the problem is that a 'git push' will sit for a \nwhile without presenting any feedback until this part of the algorithm \nis complete. The current series intends to significantly reduce this time.\n\nAs for adding progress to this step, I'm open to it. It can be done as a \nsequel series.\n\nWhat would we use to describe this section? \"Enumerating remote objects\"?\n\nThanks,\n-Stolee\n"},{"id":"365479","messageId":"87a7l41b78.fsf@evledraar.gmail.com","threadId":"49914","inReplyTo":"867aa5c3-60e0-2467-795a-40aac58f306b@gmail.com","subject":"Re: [PATCH v4 4/6] revision: implement sparse algorithm","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-12-17T14:26:19Z","receivedAt":"2018-12-17T14:26:26Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Dec 17 2018, Derrick Stolee wrote:\n\n> On 12/14/2018 6:32 PM, Ævar Arnfjörð Bjarmason wrote:\n>> On Fri, Dec 14 2018, Derrick Stolee via GitGitGadget wrote:\n>>\n>>> Despite these potential drawbacks, the benefits of the algorithm\n>>> are clear. By adding a counter to 'add_children_by_path' and\n>>> 'mark_tree_contents_uninteresting', I measured the number of\n>>> parsed trees for the two algorithms in a variety of repos.\n>> We spend a long time printing those out before we ever get to\n>> \"Enumerating objects\".\n>>\n>> Which was where I was trying to test this, i.e. is this a lot of work we\n>> perform before we print out the progress bar, and regardless of this\n>> optimization should have other progress output there, so we can see this\n>> time we're spending on this?\n>\n> It is true that part of the problem is that a 'git push' will sit for\n> a while without presenting any feedback until this part of the\n> algorithm is complete. The current series intends to significantly\n> reduce this time.\n\n\n> As for adding progress to this step, I'm open to it. It can be done as\n> a sequel series.\n\nOkey. To clarify I wasn't complaining about the lack of progress output,\nwe didn't have it before, just clarifying (as I've found out now) that\nwhen you're talking about \"enumerating objects\" in your commit message\nit's *not* what we're doing when we show the \"Enumerating objects\"\nprogress bar, but an unrelated step prior to that.\n\nI thought I might have been holding this wrong.\n\n> What would we use to describe this section? \"Enumerating remote\n> objects\"?\n\nIsn't this \"Discovering objects to push to remote\" i.e. \"Selecting\nobjects\", but then what's \"Enumerating objects\" really doing? \"Looping\nover objects we selected before and creating a pack\"?\n"},{"id":"365481","messageId":"161dd748-2ba6-cac7-433e-bd493d429759@gmail.com","threadId":"49914","inReplyTo":"87a7l41b78.fsf@evledraar.gmail.com","subject":"Re: [PATCH v4 4/6] revision: implement sparse algorithm","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-12-17T14:50:51Z","receivedAt":"2018-12-17T14:50:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/17/2018 9:26 AM, Ævar Arnfjörð Bjarmason wrote:\n> On Mon, Dec 17 2018, Derrick Stolee wrote:\n>\n>\n>> As for adding progress to this step, I'm open to it. It can be done as\n>> a sequel series.\n> Okey. To clarify I wasn't complaining about the lack of progress output,\n> we didn't have it before, just clarifying (as I've found out now) that\n> when you're talking about \"enumerating objects\" in your commit message\n> it's *not* what we're doing when we show the \"Enumerating objects\"\n> progress bar, but an unrelated step prior to that.\nPart of the problem is that in builtin/pack-objects.c, we have the \nfollowing:\n\n         if (progress)\n                 progress_state = start_progress(_(\"Enumerating \nobjects\"), 0);\n         if (!use_internal_rev_list)\n                 read_object_list_from_stdin();\n         else {\n                 get_object_list(rp.argc, rp.argv);\n                 argv_array_clear(&rp);\n         }\n         cleanup_preferred_base();\n         if (include_tag && nr_result)\n                 for_each_ref(add_ref_tag, NULL);\n         stop_progress(&progress_state);\n\nand the logic for walking uninteresting objects is the \nmark_edges_uninteresting() inside get_object_list() (both entirely \ncontained in this progress state). Perhaps the best thing to do is to \nuntangle the progress for the two modes based on \n'use_internal_rev_list'. Could we then have the progress for \nget_object_list() be \"Walking objects\" instead?\n\nThanks,\n-Stolee\n"},{"id":"366572","messageId":"xmqqzhs7x9nm.fsf@gitster-ct.c.googlers.com","threadId":"49914","inReplyTo":"817e30a287e12ce8e94ce41fcb969dd8ae53b9ce.1544822533.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 1/6] revision: add mark_tree_uninteresting_sparse","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-11T19:43:25Z","receivedAt":"2019-01-11T19:43:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> From: Derrick Stolee <dstolee@microsoft.com>\n>\n> In preparation for a new algorithm that walks fewer trees when\n> creating a pack from a set of revisions, create a method that\n> takes an oidset of tree oids and marks reachable objects as\n> UNINTERESTING.\n>\n> The current implementation uses the existing\n> mark_tree_uninteresting to recursively walk the trees and blobs.\n> This will walk the same number of trees as the old mechanism.\n>\n> There is one new assumption in this approach: we are also given\n> the oids of the interesting trees. This implementation does not\n> use those trees at the moment, but we will use them in a later\n> rewrite of this method.\n>\n> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>\n> ---\n>  revision.c | 25 +++++++++++++++++++++++++\n>  revision.h |  2 ++\n>  2 files changed, 27 insertions(+)\n>\n> diff --git a/revision.c b/revision.c\n> index 13e0519c02..f9eb6400f1 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -99,6 +99,31 @@ void mark_tree_uninteresting(struct repository *r, struct tree *tree)\n>  \tmark_tree_contents_uninteresting(r, tree);\n>  }\n>  \n> +void mark_trees_uninteresting_sparse(struct repository *r,\n> +\t\t\t\t     struct oidset *set)\n> +{\n> +\tstruct object_id *oid;\n> +\tstruct oidset_iter iter;\n> +\n> +\toidset_iter_init(set, &iter);\n> +\twhile ((oid = oidset_iter_next(&iter))) {\n> +\t\tstruct tree *tree = lookup_tree(r, oid);\n> +\n> +\t\tif (!tree)\n> +\t\t\tcontinue;\n> +\n> +\t\tif (tree->object.flags & UNINTERESTING) {\n> +\t\t\t/*\n> +\t\t\t * Remove the flag so the next call\n> +\t\t\t * is not a no-op. The flag is added\n> +\t\t\t * in mark_tree_unintersting().\n> +\t\t\t */\n> +\t\t\ttree->object.flags ^= UNINTERESTING;\n> +\t\t\tmark_tree_uninteresting(r, tree);\n> +\t\t}\n\nThe proposed log message claims that the method takes an oidset and\nmarks reachable objects as uninteresting, but the implementation\nonly marks those that are reachable from already uninteresting\ntrees.  Either one of them must be wrong.\n\nDid you mean to have this instead?\n\n\t\tif (!tree)\n\t\t\tcontinue;\n\t\t/*\n\t\t * Force traversal of the tree, even if it has been\n\t\t * already marked as UNINTERESTING.\n\t\t */\n\t\ttree->object.flags &= ~UNINTERESTING;\n\t\tmark_tree_uninteresting(r, tree);\n\nBy the way, one of the bigger reasons why I have to ask, instead of\nmaking an educated guess, is because \"struct oidset *set\" parameter\ndoes not give any useful information with the variable name to the\nreaders.  We know it is a set because its type is oidset; readers\nneed to know what meaning the 'set' has, what it is used for, why\nthe caller wants to (or decides not to) place a tree object in the\nset when it calls it.  None of that can be read from its name.\n"},{"id":"366573","messageId":"xmqqva2vx7p2.fsf@gitster-ct.c.googlers.com","threadId":"49914","inReplyTo":"xmqqzhs7x9nm.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 1/6] revision: add mark_tree_uninteresting_sparse","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-11T20:25:45Z","receivedAt":"2019-01-11T20:25:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n>> In preparation for a new algorithm that walks fewer trees when\n>> creating a pack from a set of revisions, create a method that\n>> takes an oidset of tree oids and marks reachable objects as\n>> UNINTERESTING.\n>> ...\n>> There is one new assumption in this approach: we are also given\n>> the oids of the interesting trees. This implementation does not\n>> use those trees at the moment, but we will use them in a later\n>> rewrite of this method.\n\nAhh....\n\n\n> The proposed log message claims that the method takes an oidset and\n> marks reachable objects as uninteresting, but the implementation\n> only marks those that are reachable from already uninteresting\n> trees.  Either one of them must be wrong.\n>\n> Did you mean to have this instead?\n>\n> \t\tif (!tree)\n> \t\t\tcontinue;\n> \t\t/*\n> \t\t * Force traversal of the tree, even if it has been\n> \t\t * already marked as UNINTERESTING.\n> \t\t */\n> \t\ttree->object.flags &= ~UNINTERESTING;\n> \t\tmark_tree_uninteresting(r, tree);\n\nSo, I assumed that the implementation was wrong, but it is the other\nway around.  You do mean to pick only already uninteresting trees\nout of \"set\" and mark its reachables.\n\nOne thing that would make me worried is what help the callers of\nthis function will get (or they may have to devise the way\nthemselves) to avoid having to traverse the same tree number of\ntimes.  A tree can be made uninteresting after a traversal of\nanother tree that contains it, but the logic in this function\n\n> +\t\tif (tree->object.flags & UNINTERESTING) {\n> +\t\t\t/*\n> +\t\t\t * Remove the flag so the next call\n> +\t\t\t * is not a no-op. The flag is added\n> +\t\t\t * in mark_tree_unintersting().\n> +\t\t\t */\n> +\t\t\ttree->object.flags ^= UNINTERESTING;\n> +\t\t\tmark_tree_uninteresting(r, tree);\n> +\t\t}\n\nignores the fact that it is already UNINTERESTING (in fact, in a\nsense it is even worse---it cannot be used to make a not-yet\nUNINTERESTING one into UNINTERESTING), drops the UNINTERESING bit\nand forces the traversal of that tree.  The only thing I see that\nwould work as a saving grace is that mark_tree_uninteresting()\nitself would honor existing UNINTERESING bit and refuses to recurse\ninto its subtrees, but that means blobs at the top-level of such a\ntree would be marked UNINTERESING while those in its subtrees can be\nleft not-UNINTERESING, which sounds like inviting a mess.\n\nIt does *not* immediately mean this function is misdesigned.  It\njust means that the caller needs to carefully follow whatever\ncalling convention this series will establish in the later patches\n(which we haven't seen yet at this point).\n\n> By the way, one of the bigger reasons why I have to ask, instead of\n> making an educated guess, is because \"struct oidset *set\" parameter\n> does not give any useful information with the variable name to the\n> readers.  We know it is a set because its type is oidset; readers\n> need to know what meaning the 'set' has, what it is used for, why\n> the caller wants to (or decides not to) place a tree object in the\n> set when it calls it.  None of that can be read from its name.\n"},{"id":"366576","messageId":"5a25813e-32f6-34a4-9f24-b65415b75dda@gmail.com","threadId":"49914","inReplyTo":"xmqqva2vx7p2.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 1/6] revision: add mark_tree_uninteresting_sparse","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-01-11T22:05:50Z","receivedAt":"2019-01-11T22:05:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/11/2019 3:25 PM, Junio C Hamano wrote:\n> Junio C Hamano <gitster@pobox.com> writes:\n>\n> So, I assumed that the implementation was wrong, but it is the other\n> way around.  You do mean to pick only already uninteresting trees\n> out of \"set\" and mark its reachables.\n>\n> One thing that would make me worried is what help the callers of\n> this function will get (or they may have to devise the way\n> themselves) to avoid having to traverse the same tree number of\n> times.  A tree can be made uninteresting after a traversal of\n> another tree that contains it, but the logic in this function\n>\n>> +\t\tif (tree->object.flags & UNINTERESTING) {\n>> +\t\t\t/*\n>> +\t\t\t * Remove the flag so the next call\n>> +\t\t\t * is not a no-op. The flag is added\n>> +\t\t\t * in mark_tree_unintersting().\n>> +\t\t\t */\n>> +\t\t\ttree->object.flags ^= UNINTERESTING;\n>> +\t\t\tmark_tree_uninteresting(r, tree);\n>> +\t\t}\n> ignores the fact that it is already UNINTERESTING (in fact, in a\n> sense it is even worse---it cannot be used to make a not-yet\n> UNINTERESTING one into UNINTERESTING), drops the UNINTERESING bit\n> and forces the traversal of that tree.  The only thing I see that\n> would work as a saving grace is that mark_tree_uninteresting()\n> itself would honor existing UNINTERESING bit and refuses to recurse\n> into its subtrees, but that means blobs at the top-level of such a\n> tree would be marked UNINTERESING while those in its subtrees can be\n> left not-UNINTERESING, which sounds like inviting a mess.\n>\n> It does *not* immediately mean this function is misdesigned.  It\n> just means that the caller needs to carefully follow whatever\n> calling convention this series will establish in the later patches\n> (which we haven't seen yet at this point).\n>\nI'm sorry that this implementation is particularly confusing. It is \ncreated only as a placeholder so we can wire up the --sparse option to \nuse this method without being \"wrong\" but is then removed entirely in \nPATCH 4/6:\n\n...\n\nvoid mark_trees_uninteresting_sparse(struct repository *r,\n  \t\t\t\t     struct oidset *set)\n  {\n+\tunsigned has_interesting = 0, has_uninteresting = 0;\n+\tstruct hashmap map;\n+\tstruct hashmap_iter map_iter;\n+\tstruct path_and_oids_entry *entry;\n  \tstruct object_id *oid;\n  \tstruct oidset_iter iter;\n  \n  \toidset_iter_init(set, &iter);\n-\twhile ((oid = oidset_iter_next(&iter))) {\n+\twhile ((!has_interesting || !has_uninteresting) &&\n+\t       (oid = oidset_iter_next(&iter))) {\n  \t\tstruct tree *tree = lookup_tree(r, oid);\n  \n  \t\tif (!tree)\n  \t\t\tcontinue;\n  \n-\t\tif (tree->object.flags & UNINTERESTING) {\n-\t\t\t/*\n-\t\t\t * Remove the flag so the next call\n-\t\t\t * is not a no-op. The flag is added\n-\t\t\t * in mark_tree_unintersting().\n-\t\t\t */\n-\t\t\ttree->object.flags ^= UNINTERESTING;\n-\t\t\tmark_tree_uninteresting(r, tree);\n-\t\t}\n+\t\tif (tree->object.flags & UNINTERESTING)\n+\t\t\thas_uninteresting = 1;\n+\t\telse\n+\t\t\thas_interesting = 1;\n+\t}\n...\n\nYou are definitely correct that \"set\" is not a valuable variable name. It could instead be \"tree_oids\" to be slightly more informative.\n\nThanks,\n-Stolee\n\n"},{"id":"366593","messageId":"xmqqmuo6yghg.fsf@gitster-ct.c.googlers.com","threadId":"49914","inReplyTo":"ab733daff5398fd07ff051c323f51b70efbc2e57.1544822533.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 3/6] pack-objects: add --sparse option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-11T22:30:35Z","receivedAt":"2019-01-11T22:30:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> From: Derrick Stolee <dstolee@microsoft.com>\n>\n> Add a '--sparse' option flag to the pack-objects builtin. This\n> allows the user to specify that they want to use the new logic\n> for walking trees. This logic currently does not differ from the\n> existing output, but will in a later change.\n>\n> Create a new test script, t5322-pack-objects-sparse.sh, to ensure\n> the object list that is selected matches what we expect. When we\n> update the logic to walk in a sparse fashion, the final test will\n> be updated to show the extra objects that are added.\n\nSomehow checking the \"these are exactly what we expect\" feels\nbrittle.  In a history with three relevant commits A---B---C,\npacking B..C could omit trees and blobs in C that appear in A but\nnot in B, but traditionally, because we stop traversal at B and do\nnot even look at A, we do not notice that such objects that need to\ncomplete C's tree are already available in a repository that has B.\nThe approach of test in this patch feels similar to saying \"we must\nsend these duplicates because we must stay dumb\", even though with a\nperfect knowledge about the history, e.g. bitmap, we would be able\nto omit these objects in A that appear in C but not in B.\n\nI think we want to test test both \"are we sending enough, even\nthough we might be wasting some bandwidth by not noticing the other\nside already have some?\" and \"are we still avoiding from becoming\noverly stupid, even though we may be cheating to save traversal cost\nand risking to redundantly send some objects?\"\n\nIOW, a set of tests to make sure that truly new objects are all sent\nby seeing what is sent is a strict superset of what we expect.  And\nanother set of tests to make sure that objects that are so obviously\n(this criterion may be highly subjective) be present in the\nreceiving repository (e.g. the tree object of commit B and what it\ncontains when seinding B..C) are never sent, even when using an\nalgorithm that are tuned for traversal cost over bandwidth\nconsumption.\n\nThe code change in this step looks all trivially good, and it may\nwant to be squashed into the previous step to become a single patch.\nOtherwise, [2/6] would be adding a dead code that nobody exercises\nuntil this step.\n\nThanks.\n\n\n> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>\n> ---\n>  Documentation/git-pack-objects.txt |  11 ++-\n>  builtin/pack-objects.c             |   5 +-\n>  t/t5322-pack-objects-sparse.sh     | 115 +++++++++++++++++++++++++++++\n>  3 files changed, 129 insertions(+), 2 deletions(-)\n>  create mode 100755 t/t5322-pack-objects-sparse.sh\n>\n> diff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\n> index 40c825c381..e45f3e680d 100644\n> --- a/Documentation/git-pack-objects.txt\n> +++ b/Documentation/git-pack-objects.txt\n> @@ -14,7 +14,7 @@ SYNOPSIS\n>  \t[--local] [--incremental] [--window=<n>] [--depth=<n>]\n>  \t[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\n>  \t[--stdout [--filter=<filter-spec>] | base-name]\n> -\t[--shallow] [--keep-true-parents] < object-list\n> +\t[--shallow] [--keep-true-parents] [--sparse] < object-list\n>  \n>  \n>  DESCRIPTION\n> @@ -196,6 +196,15 @@ depth is 4095.\n>  \tAdd --no-reuse-object if you want to force a uniform compression\n>  \tlevel on all data no matter the source.\n>  \n> +--sparse::\n> +\tUse the \"sparse\" algorithm to determine which objects to include in\n> +\tthe pack, when combined with the \"--revs\" option. This algorithm\n> +\tonly walks trees that appear in paths that introduce new objects.\n> +\tThis can have significant performance benefits when computing\n> +\ta pack to send a small change. However, it is possible that extra\n> +\tobjects are added to the pack-file if the included commits contain\n> +\tcertain types of direct renames.\n> +\n>  --thin::\n>  \tCreate a \"thin\" pack by omitting the common objects between a\n>  \tsender and a receiver in order to reduce network transfer. This\n> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\n> index 5f70d840a7..7d5b0735e3 100644\n> --- a/builtin/pack-objects.c\n> +++ b/builtin/pack-objects.c\n> @@ -84,6 +84,7 @@ static unsigned long pack_size_limit;\n>  static int depth = 50;\n>  static int delta_search_threads;\n>  static int pack_to_stdout;\n> +static int sparse;\n>  static int thin;\n>  static int num_preferred_base;\n>  static struct progress *progress_state;\n> @@ -3135,7 +3136,7 @@ static void get_object_list(int ac, const char **av)\n>  \n>  \tif (prepare_revision_walk(&revs))\n>  \t\tdie(_(\"revision walk setup failed\"));\n> -\tmark_edges_uninteresting(&revs, show_edge, 0);\n> +\tmark_edges_uninteresting(&revs, show_edge, sparse);\n>  \n>  \tif (!fn_show_object)\n>  \t\tfn_show_object = show_object;\n> @@ -3292,6 +3293,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n>  \t\t{ OPTION_CALLBACK, 0, \"unpack-unreachable\", NULL, N_(\"time\"),\n>  \t\t  N_(\"unpack unreachable objects newer than <time>\"),\n>  \t\t  PARSE_OPT_OPTARG, option_parse_unpack_unreachable },\n> +\t\tOPT_BOOL(0, \"sparse\", &sparse,\n> +\t\t\t N_(\"use the sparse reachability algorithm\")),\n>  \t\tOPT_BOOL(0, \"thin\", &thin,\n>  \t\t\t N_(\"create thin packs\")),\n>  \t\tOPT_BOOL(0, \"shallow\", &shallow,\n> diff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\n> new file mode 100755\n> index 0000000000..81f6805bc3\n> --- /dev/null\n> +++ b/t/t5322-pack-objects-sparse.sh\n> @@ -0,0 +1,115 @@\n> +#!/bin/sh\n> +\n> +test_description='pack-objects object selection using sparse algorithm'\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'setup repo' '\n> +\ttest_commit initial &&\n> +\tfor i in $(test_seq 1 3)\n> +\tdo\n> +\t\tmkdir f$i &&\n> +\t\tfor j in $(test_seq 1 3)\n> +\t\tdo\n> +\t\t\tmkdir f$i/f$j &&\n> +\t\t\techo $j >f$i/f$j/data.txt\n> +\t\tdone\n> +\tdone &&\n> +\tgit add . &&\n> +\tgit commit -m \"Initialized trees\" &&\n> +\tfor i in $(test_seq 1 3)\n> +\tdo\n> +\t\tgit checkout -b topic$i master &&\n> +\t\techo change-$i >f$i/f$i/data.txt &&\n> +\t\tgit commit -a -m \"Changed f$i/f$i/data.txt\"\n> +\tdone &&\n> +\tcat >packinput.txt <<-EOF &&\n> +\ttopic1\n> +\t^topic2\n> +\t^topic3\n> +\tEOF\n> +\tgit rev-parse\t\t\t\\\n> +\t\ttopic1\t\t\t\\\n> +\t\ttopic1^{tree}\t\t\\\n> +\t\ttopic1:f1\t\t\\\n> +\t\ttopic1:f1/f1\t\t\\\n> +\t\ttopic1:f1/f1/data.txt | sort >expect_objects.txt\n> +'\n> +\n> +test_expect_success 'non-sparse pack-objects' '\n> +\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n> +\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n> +\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n> +\ttest_cmp expect_objects.txt nonsparse_objects.txt\n> +'\n> +\n> +test_expect_success 'sparse pack-objects' '\n> +\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n> +\tgit index-pack -o sparse.idx sparse.pack &&\n> +\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n> +\ttest_cmp expect_objects.txt sparse_objects.txt\n> +'\n> +\n> +# Demonstrate that both algorithms send \"extra\" objects because\n> +# they are not in the frontier.\n> +\n> +test_expect_success 'duplicate a folder from f3 and commit to topic1' '\n> +\tgit checkout topic1 &&\n> +\techo change-3 >f3/f3/data.txt &&\n> +\tgit commit -a -m \"Changed f3/f3/data.txt\" &&\n> +\tgit rev-parse\t\t\t\\\n> +\t\ttopic1~1\t\t\\\n> +\t\ttopic1~1^{tree}\t\t\\\n> +\t\ttopic1^{tree}\t\t\\\n> +\t\ttopic1\t\t\t\\\n> +\t\ttopic1:f1\t\t\\\n> +\t\ttopic1:f1/f1\t\t\\\n> +\t\ttopic1:f1/f1/data.txt\t\\\n> +\t\ttopic1:f3\t\t\\\n> +\t\ttopic1:f3/f3\t\t\\\n> +\t\ttopic1:f3/f3/data.txt | sort >expect_objects.txt\n> +'\n> +\n> +test_expect_success 'non-sparse pack-objects' '\n> +\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n> +\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n> +\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n> +\ttest_cmp expect_objects.txt nonsparse_objects.txt\n> +'\n> +\n> +test_expect_success 'sparse pack-objects' '\n> +\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n> +\tgit index-pack -o sparse.idx sparse.pack &&\n> +\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n> +\ttest_cmp expect_objects.txt sparse_objects.txt\n> +'\n> +\n> +test_expect_success 'duplicate a folder from f1 into f3' '\n> +\tmkdir f3/f4 &&\n> +\tcp -r f1/f1/* f3/f4 &&\n> +\tgit add f3/f4 &&\n> +\tgit commit -m \"Copied f1/f1 to f3/f4\" &&\n> +\tcat >packinput.txt <<-EOF &&\n> +\ttopic1\n> +\t^topic1~1\n> +\tEOF\n> +\tgit rev-parse\t\t\\\n> +\t\ttopic1\t\t\\\n> +\t\ttopic1^{tree}\t\\\n> +\t\ttopic1:f3 | sort >expect_objects.txt\n> +'\n> +\n> +test_expect_success 'non-sparse pack-objects' '\n> +\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n> +\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n> +\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n> +\ttest_cmp expect_objects.txt nonsparse_objects.txt\n> +'\n> +\n> +test_expect_success 'sparse pack-objects' '\n> +\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n> +\tgit index-pack -o sparse.idx sparse.pack &&\n> +\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n> +\ttest_cmp expect_objects.txt sparse_objects.txt\n> +'\n> +\n> +test_done\n"},{"id":"366594","messageId":"xmqqh8eeye6x.fsf@gitster-ct.c.googlers.com","threadId":"49914","inReplyTo":"c44172c35ece7aafec02c7f3c8438ccca4f69023.1544822533.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 4/6] revision: implement sparse algorithm","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-11T23:20:06Z","receivedAt":"2019-01-11T23:20:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> From: Derrick Stolee <dstolee@microsoft.com>\n>\n> When enumerating objects to place in a pack-file during 'git\n> pack-objects --revs', we discover the \"frontier\" of commits\n> that we care about and the boundary with commit we find\n> uninteresting. From that point, we walk trees to discover which\n> trees and blobs are uninteresting. Finally, we walk trees from the\n> interesting commits to find the interesting objects that are\n> placed in the pack.\n>\n> This commit introduces a new, \"sparse\" way to discover the\n> uninteresting trees. We use the perspective of a single user trying\n> to push their topic to a large repository. That user likely changed\n> a very small fraction of the paths in their working directory, but\n> we spend a lot of time walking all reachable trees.\n>\n> The way to switch the logic to work in this sparse way is to start\n> caring about which paths introduce new trees. While it is not\n> possible to generate a diff between the frontier boundary and all\n> of the interesting commits, we can simulate that behavior by\n> inspecting all of the root trees as a whole, then recursing down\n> to the set of trees at each path.\n>\n> We already had taken the first step by passing an oidset to\n> mark_trees_uninteresting_sparse(). We now create a dictionary\n> whose keys are paths and values are oidsets. We consider the set\n> of trees that appear at each path. While we inspect a tree, we\n> add its subtrees to the oidsets corresponding to the tree entry's\n> path. We also mark trees as UNINTERESTING if the tree we are\n> parsing is UNINTERESTING.\n>\n> To actually improve the peformance, we need to terminate our\n\nperformance?\n\n> recursion. If the oidset contains only UNINTERESTING trees, then\n\nSo, at this point, it is not quite clear what \"the oidset\" refers to\nto me.  When you are looking at the root tree of one commit, you'd\nadd the object ID of the tree to the dictionary keyed by the path\n(i.e. root of the tree).  You traverse to the parent and keep adding\nthese root trees, and after a while you'd hit a negative commit and\nadd its tree with UNINTERESTING bit.  Before that point, the oidset\nfor the root of the tree has only interesting trees, so you won't\nrecurse into its contents, but now at the final point you'd walk to\npropagate UNINTERESTING bit down?\n\n> we do not continue the recursion. This avoids walking trees that\n> are likely to not be reachable from interesting trees. If the\n> oidset contains only interesting trees, then we will walk these\n> trees in the final stage that collects the intersting objects to\n> place in the pack. Thus, we only recurse if the oidset contains\n> both interesting and UNINITERESTING trees.\n>\n> There are a few ways that this is not a universally better option.\n>\n> First, we can pack extra objects. If someone copies a subtree\n> from one tree to another, the first tree will appear UNINTERESTING\n> and we will not recurse to see that the subtree should also be\n> UNINTERESTING. We will walk the new tree and see the subtree as\n> a \"new\" object and add it to the pack. We add a test case that\n> demonstrates this as a way to prove that the --sparse option is\n> actually working.\n\nThat's an interesting use of the word \"working\" ;-)\n\n> Second, we can have extra memory pressure. If instead of being a\n> single user pushing a small topic we are a server sending new\n> objects from across the entire working directory, then we will\n> gain very little (the recursion will rarely terminate early) but\n> will spend extra time maintaining the path-oidset dictionaries.\n> ...\n> I used the number of walked trees the main metric above because\n> it is consistent across multiple runs. When I ran my tests, the\n> performance of the pack-objects command with the same options\n> could change the end-to-end time by 10x depending on the file\n> system being warm. However, by repeating the same test on repeat\n> I could get more consistent timing results. The git.git and\n> Linux tests were too fast overall (less than 0.5s) to measure\n> an end-to-end difference. The Azure DevOps case was slow enough\n> to see the time improve from 15s to 1s in the warm case. The\n> cold case was 90s to 9s in my testing.\n\nUnderstandable.  Projects that tend to be deep (e.g. Java?) would\nlikely to benefit if they are structured and partitioned very well.\n\n> +struct path_and_oids_entry {\n> +\tstruct hashmap_entry ent;\n> +\tchar *path;\n> +\tstruct oidset set;\n\nAgain \"set\"?  If I understand the description of the logic explained\nin the proposed commit log message correctly, then this is a set of\n\"tree objects at this path\", right?  At least calling this \"trees\"\nmay give us slightly better indication.\n\n> +static int path_and_oids_cmp(const void *hashmap_cmp_fn_data,\n> +\t\t\t     const struct path_and_oids_entry *e1,\n> +\t\t\t     const struct path_and_oids_entry *e2,\n> +\t\t\t     const void *keydata)\n> +{\n> +\treturn strcmp(e1->path, e2->path);\n> +}\n> +\n> +int map_flags = 0;\n\nThat's too bland a name for a system-wide global variable.  If it is\na file-scope \"static\", that might be more acceptable.\n\nIn any case, let BSS take care of the initialization and do not\ninitialize an int explicitly to 0.\n\nIs it even used?  At least at this step the comparison callback\nfunction does not seem to make any use of the fn-data, so a better\norganization may be to delay the introduction of this variable to a\nlater step where it actually gets used, at which time, the code may\nhave a better context to give the variable a more suitable name.\n\n> +static void paths_and_oids_init(struct hashmap *map)\n> +{\n> +\thashmap_init(map, (hashmap_cmp_fn) path_and_oids_cmp, &map_flags, 0);\n> +}\n> +\n> +static void paths_and_oids_clear(struct hashmap *map)\n> +{\n> +\tstruct hashmap_iter iter;\n> +\tstruct path_and_oids_entry *entry;\n> +\thashmap_iter_init(map, &iter);\n> +\n> +\twhile ((entry = (struct path_and_oids_entry *)hashmap_iter_next(&iter))) {\n> +\t\toidset_clear(&entry->set);\n> +\t\tfree(entry->path);\n> +\t}\n> +\n> +\thashmap_free(map, 1);\n> +}\n> +\n> +static void paths_and_oids_insert(struct hashmap *map,\n> +\t\t\t\t  const char *path,\n> +\t\t\t\t  const struct object_id *oid)\n> +{\n> +\tint hash = strhash(path);\n> +\tstruct path_and_oids_entry key;\n> +\tstruct path_and_oids_entry *entry;\n> +\n> +\thashmap_entry_init(&key, hash);\n> +\tkey.path = xstrdup(path);\n> +\toidset_init(&key.set, 0);\n> +\n> +\tif (!(entry = (struct path_and_oids_entry *)hashmap_get(map, &key, NULL))) {\n> +\t\tentry = xcalloc(1, sizeof(struct path_and_oids_entry));\n> +\t\thashmap_entry_init(entry, hash);\n> +\t\tentry->path = key.path;\n> +\t\toidset_init(&entry->set, 16);\n> +\t\thashmap_put(map, entry);\n> +\t} else {\n> +\t\tfree(key.path);\n> +\t}\n\nHmph, having to always allocate only to free sounds wasteful.  \n\nCan't we instead do the look-up with \"path\" that are given read-only\nto us, and allocate only when we are creating the entry in the\ndictionary for the first time?  After all, we'd make many look-ups\nof the same path to add trees for the same path and only one of them\n(i.e. the first one) is used to create an entry in the dictionary.\n\nOr does it not matter?\n\n> +\n> +\toidset_insert(&entry->set, oid);\n> +}\n> +\n> +static void add_children_by_path(struct repository *r,\n> +\t\t\t\t struct tree *tree,\n> +\t\t\t\t struct hashmap *map)\n> +{\n> +\tstruct tree_desc desc;\n> +\tstruct name_entry entry;\n> +\n> +\tif (!tree)\n> +\t\treturn;\n> +\n> +\tif (parse_tree_gently(tree, 1) < 0)\n> +\t\treturn;\n> +\n> +\tinit_tree_desc(&desc, tree->buffer, tree->size);\n> +\twhile (tree_entry(&desc, &entry)) {\n> +\t\tswitch (object_type(entry.mode)) {\n> +\t\tcase OBJ_TREE:\n> +\t\t\tpaths_and_oids_insert(map, entry.path, entry.oid);\n> +\n> +\t\t\tif (tree->object.flags & UNINTERESTING) {\n> +\t\t\t\tstruct tree *child = lookup_tree(r, entry.oid);\n> +\t\t\t\tif (child)\n> +\t\t\t\t\tchild->object.flags |= UNINTERESTING;\n> +\t\t\t}\n> +\t\t\tbreak;\n> +\t\tcase OBJ_BLOB:\n> +\t\t\tif (tree->object.flags & UNINTERESTING) {\n> +\t\t\t\tstruct blob *child = lookup_blob(r, entry.oid);\n> +\t\t\t\tif (child)\n> +\t\t\t\t\tchild->object.flags |= UNINTERESTING;\n> +\t\t\t}\n> +\t\t\tbreak;\n> +\t\tdefault:\n> +\t\t\t/* Subproject commit - not in this repository */\n> +\t\t\tbreak;\n> +\t\t}\n> +\t}\n> +\n> +\tfree_tree_buffer(tree);\n> +}\n\nOK.\n\n>  void mark_trees_uninteresting_sparse(struct repository *r,\n>  \t\t\t\t     struct oidset *set)\n>  {\n> +\tunsigned has_interesting = 0, has_uninteresting = 0;\n> +\tstruct hashmap map;\n> +\tstruct hashmap_iter map_iter;\n> +\tstruct path_and_oids_entry *entry;\n>  \tstruct object_id *oid;\n>  \tstruct oidset_iter iter;\n>  \n>  \toidset_iter_init(set, &iter);\n> -\twhile ((oid = oidset_iter_next(&iter))) {\n> +\twhile ((!has_interesting || !has_uninteresting) &&\n> +\t       (oid = oidset_iter_next(&iter))) {\n>  \t\tstruct tree *tree = lookup_tree(r, oid);\n>  \n>  \t\tif (!tree)\n>  \t\t\tcontinue;\n>  \n> +\t\tif (tree->object.flags & UNINTERESTING)\n> +\t\t\thas_uninteresting = 1;\n> +\t\telse\n> +\t\t\thas_interesting = 1;\n> +\t}\n> +\n> +\t/* Do not walk unless we have both types of trees. */\n> +\tif (!has_uninteresting || !has_interesting)\n> +\t\treturn;\n\nOK.\n\n> +\tpaths_and_oids_init(&map);\n> +\n> +\toidset_iter_init(set, &iter);\n> +\twhile ((oid = oidset_iter_next(&iter))) {\n> +\t\tstruct tree *tree = lookup_tree(r, oid);\n> +\t\tadd_children_by_path(r, tree, &map);\n>  \t}\n> +\n> +\thashmap_iter_init(&map, &map_iter);\n> +\twhile ((entry = hashmap_iter_next(&map_iter)))\n> +\t\tmark_trees_uninteresting_sparse(r, &entry->set);\n> +\n> +\tpaths_and_oids_clear(&map);\n>  }\n>  \n>  struct commit_stack {\n> diff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\n> index 81f6805bc3..45dba6e014 100755\n> --- a/t/t5322-pack-objects-sparse.sh\n> +++ b/t/t5322-pack-objects-sparse.sh\n> @@ -83,22 +83,25 @@ test_expect_success 'sparse pack-objects' '\n>  \ttest_cmp expect_objects.txt sparse_objects.txt\n>  '\n>  \n> +# Demonstrate that the algorithms differ when we copy a tree wholesale\n> +# from one folder to another.\n> +\n>  test_expect_success 'duplicate a folder from f1 into f3' '\n>  \tmkdir f3/f4 &&\n>  \tcp -r f1/f1/* f3/f4 &&\n>  \tgit add f3/f4 &&\n>  \tgit commit -m \"Copied f1/f1 to f3/f4\" &&\n> -\tcat >packinput.txt <<-EOF &&\n> +\tcat >packinput.txt <<-EOF\n>  \ttopic1\n>  \t^topic1~1\n>  \tEOF\n> -\tgit rev-parse\t\t\\\n> -\t\ttopic1\t\t\\\n> -\t\ttopic1^{tree}\t\\\n> -\t\ttopic1:f3 | sort >expect_objects.txt\n>  '\n>  \n>  test_expect_success 'non-sparse pack-objects' '\n> +\tgit rev-parse\t\t\t\\\n> +\t\ttopic1\t\t\t\\\n> +\t\ttopic1^{tree}\t\t\\\n> +\t\ttopic1:f3 | sort >expect_objects.txt &&\n\nOK, the change above sort of makes sense ;-)\n\n>  \tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n>  \tgit index-pack -o nonsparse.idx nonsparse.pack &&\n>  \tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n> @@ -106,10 +109,16 @@ test_expect_success 'non-sparse pack-objects' '\n>  '\n>  \n>  test_expect_success 'sparse pack-objects' '\n> +\tgit rev-parse\t\t\t\\\n> +\t\ttopic1\t\t\t\\\n> +\t\ttopic1^{tree}\t\t\\\n> +\t\ttopic1:f3\t\t\\\n> +\t\ttopic1:f3/f4\t\t\\\n> +\t\ttopic1:f3/f4/data.txt | sort >expect_sparse_objects.txt &&\n\nAnd the reason why f4 and its contents are wastefully sent is\nbecause the whole tree is duplicated, which is understandable.\n\n>  \tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n>  \tgit index-pack -o sparse.idx sparse.pack &&\n>  \tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n> -\ttest_cmp expect_objects.txt sparse_objects.txt\n> +\ttest_cmp expect_sparse_objects.txt sparse_objects.txt\n>  '\n>  \n>  test_done\n"},{"id":"366595","messageId":"xmqqa7k6ye6c.fsf@gitster-ct.c.googlers.com","threadId":"49914","inReplyTo":"39dc89beb91ac12c94d13f7931a4d9ebc602681f.1544822533.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 2/6] list-objects: consume sparse tree walk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-11T23:20:27Z","receivedAt":"2019-01-11T23:20:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n\n> From: Derrick Stolee <dstolee@microsoft.com>\n>\n> When creating a pack-file using 'git pack-objects --revs' we provide\n> a list of interesting and uninteresting commits. For example, a push\n> operation would make the local topic branch be interesting and the\n> known remote refs as uninteresting. We want to discover the set of\n> new objects to send to the server as a thin pack.\n>\n> We walk these commits until we discover a frontier of commits such\n> that every commit walk starting at interesting commits ends in a root\n> commit or unintersting commit. We then need to discover which\n> non-commit objects are reachable from  uninteresting commits. This\n> commit walk is not changing during this series.\n>\n> The mark_edges_uninteresting() method in list-objects.c iterates on\n> the commit list and does the following:\n>\n> * If the commit is UNINTERSTING, then mark its root tree and every\n>   object it can reach as UNINTERESTING.\n>\n> * If the commit is interesting, then mark the root tree of every\n>   UNINTERSTING parent (and all objects that tree can reach) as\n>   UNINTERSTING.\n>\n> At the very end, we repeat the process on every commit directly\n> given to the revision walk from stdin. This helps ensure we properly\n> cover shallow commits that otherwise were not included in the\n> frontier.\n>\n> The logic to recursively follow trees is in the\n> mark_tree_uninteresting() method in revision.c. The algorithm avoids\n> duplicate work by not recursing into trees that are already marked\n> UNINTERSTING.\n>\n> Add a new 'sparse' option to the mark_edges_uninteresting() method\n> that performs this logic in a slightly new way. As we iterate over\n> the commits, we add all of the root trees to an oidset. Then, call\n> mark_trees_uninteresting_sparse() on that oidset. Note that we\n> include interesting trees in this process. The current implementation\n> of mark_trees_unintersting_sparse() will walk the same trees as\n> the old logic, but this will be replaced in a later change.\n\nIt is unclear what \"a slightly new way\" refers to.  The updated code\nadds the UNINTERSTING edge commits and UNINTERSTING parents of\ninteresting edge commits (the latter is done using the new\nadd-edge-parents() helper function) to an oidset and calls the new\nhelper introduced in [1/6] to ensure all the objects reachable from\nthese UNINTERSTING trees become UNINTERSTING.\n\nWhich seems to be doing exactly the same thing as the original.\n\nPuzzled.\n"},{"id":"366714","messageId":"8c50e066-b132-cf6e-ed0f-ce843cf3a634@gmail.com","threadId":"49914","inReplyTo":"xmqqmuo6yghg.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 3/6] pack-objects: add --sparse option","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2019-01-15T15:06:53Z","receivedAt":"2019-01-15T15:06:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/11/2019 5:30 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" <gitgitgadget@gmail.com> writes:\n>\n>> From: Derrick Stolee <dstolee@microsoft.com>\n>>\n>> Add a '--sparse' option flag to the pack-objects builtin. This\n>> allows the user to specify that they want to use the new logic\n>> for walking trees. This logic currently does not differ from the\n>> existing output, but will in a later change.\n>>\n>> Create a new test script, t5322-pack-objects-sparse.sh, to ensure\n>> the object list that is selected matches what we expect. When we\n>> update the logic to walk in a sparse fashion, the final test will\n>> be updated to show the extra objects that are added.\n> Somehow checking the \"these are exactly what we expect\" feels\n> brittle.  In a history with three relevant commits A---B---C,\n> packing B..C could omit trees and blobs in C that appear in A but\n> not in B, but traditionally, because we stop traversal at B and do\n> not even look at A, we do not notice that such objects that need to\n> complete C's tree are already available in a repository that has B.\n> The approach of test in this patch feels similar to saying \"we must\n> send these duplicates because we must stay dumb\", even though with a\n> perfect knowledge about the history, e.g. bitmap, we would be able\n> to omit these objects in A that appear in C but not in B.\nMy intention with this test was to show that the existing algorithm also \nsends \"extra\" objects in certain situations, which is later contrasted \nby a test where the new algorithm sends objects the old algorithm did not.\n\nInstead of \"we must stay dumb\" I wanted to say \"we are not dumber (in \nthis case)\".\n\nI understand your brittle feeling. If the logic changed in either to be \nsmarter, then we would need to change the test. In some sense, that does \nmean we have extra coverage of \"this is how it works now\" so we would \nunderstand the behavior change of that hypothetical future change.\n\n> I think we want to test test both \"are we sending enough, even\n> though we might be wasting some bandwidth by not noticing the other\n> side already have some?\" and \"are we still avoiding from becoming\n> overly stupid, even though we may be cheating to save traversal cost\n> and risking to redundantly send some objects?\"\n>\n> IOW, a set of tests to make sure that truly new objects are all sent\n> by seeing what is sent is a strict superset of what we expect.  And\n> another set of tests to make sure that objects that are so obviously\n> (this criterion may be highly subjective) be present in the\n> receiving repository (e.g. the tree object of commit B and what it\n> contains when seinding B..C) are never sent, even when using an\n> algorithm that are tuned for traversal cost over bandwidth\n> consumption.\n\nTo properly test \"these objects are included,\" do we have an established \npattern for saying \"this file is a subset of that file\"? Or, should I \nuse something like `comm expected actual >common && test_cmp expected \ncommon`?\n\n> The code change in this step looks all trivially good, and it may\n> want to be squashed into the previous step to become a single patch.\n> Otherwise, [2/6] would be adding a dead code that nobody exercises\n> until this step.\n\nWill squash.\n\nThanks,\n-Stolee\n"},{"id":"366727","messageId":"xmqq4la9u6do.fsf@gitster-ct.c.googlers.com","threadId":"49914","inReplyTo":"8c50e066-b132-cf6e-ed0f-ce843cf3a634@gmail.com","subject":"Re: [PATCH v4 3/6] pack-objects: add --sparse option","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-15T18:23:47Z","receivedAt":"2019-01-15T18:23:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee <stolee@gmail.com> writes:\n\n> Instead of \"we must stay dumb\" I wanted to say \"we are not dumber (in\n> this case)\".\n\nThanks, then what we aim for are compatible ;-)  I do agree that we\nshould make sure we will not ever become overly stupid.\n\n> To properly test \"these objects are included,\" do we have an\n> established pattern for saying \"this file is a subset of that file\"?\n> Or, should I use something like `comm expected actual >common &&\n> test_cmp expected common`?\n\nYeah, \"comm -23 must-exist actual\" would be a natural thing to use\nwhen we have two sorted files and want to see what is missing from\nthe actual output among those that must exist.  t6500 and t9350\nalready seem to use the tool, so it should be OK from portability's\npoint of view to non-UNIX systems.\n\n\n"},{"id":"366860","messageId":"0747f7494ef2be5e61bad610d2fe0b7db617bb49.1547663156.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 2/5] list-objects: consume sparse tree walk","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T18:25:58Z","receivedAt":"2019-01-16T18:26:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen creating a pack-file using 'git pack-objects --revs' we provide\na list of interesting and uninteresting commits. For example, a push\noperation would make the local topic branch be interesting and the\nknown remote refs as uninteresting. We want to discover the set of\nnew objects to send to the server as a thin pack.\n\nWe walk these commits until we discover a frontier of commits such\nthat every commit walk starting at interesting commits ends in a root\ncommit or unintersting commit. We then need to discover which\nnon-commit objects are reachable from  uninteresting commits. This\ncommit walk is not changing during this series.\n\nThe mark_edges_uninteresting() method in list-objects.c iterates on\nthe commit list and does the following:\n\n* If the commit is UNINTERSTING, then mark its root tree and every\n  object it can reach as UNINTERESTING.\n\n* If the commit is interesting, then mark the root tree of every\n  UNINTERSTING parent (and all objects that tree can reach) as\n  UNINTERSTING.\n\nAt the very end, we repeat the process on every commit directly\ngiven to the revision walk from stdin. This helps ensure we properly\ncover shallow commits that otherwise were not included in the\nfrontier.\n\nThe logic to recursively follow trees is in the\nmark_tree_uninteresting() method in revision.c. The algorithm avoids\nduplicate work by not recursing into trees that are already marked\nUNINTERSTING.\n\nAdd a new 'sparse' option to the mark_edges_uninteresting() method\nthat performs this logic in a slightly different way. As we iterate\nover the commits, we add all of the root trees to an oidset. Then,\ncall mark_trees_uninteresting_sparse() on that oidset. Note that we\ninclude interesting trees in this process. The current implementation\nof mark_trees_unintersting_sparse() will walk the same trees as\nthe old logic, but this will be replaced in a later change.\n\nAdd a '--sparse' flag in 'git pack-objects' to call this new logic.\nAdd a new test script t/t5322-pack-objects-sparse.sh that tests this\noption. The tests currently demonstrate that the resulting object\nlist is the same as the old algorithm. This includes a case where\nboth algorithms pack an object that is not needed by a remote due to\nlimits on the explored set of trees. When the sparse algorithm is\nchanged in a later commit, we will add a test that demonstrates a\nchange of behavior in some cases.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n Documentation/git-pack-objects.txt |  11 ++-\n bisect.c                           |   2 +-\n builtin/pack-objects.c             |   5 +-\n builtin/rev-list.c                 |   2 +-\n http-push.c                        |   2 +-\n list-objects.c                     |  70 +++++++++++++++---\n list-objects.h                     |   4 +-\n t/t5322-pack-objects-sparse.sh     | 113 +++++++++++++++++++++++++++++\n 8 files changed, 192 insertions(+), 17 deletions(-)\n create mode 100755 t/t5322-pack-objects-sparse.sh\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 40c825c381..e45f3e680d 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -14,7 +14,7 @@ SYNOPSIS\n \t[--local] [--incremental] [--window=<n>] [--depth=<n>]\n \t[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\n \t[--stdout [--filter=<filter-spec>] | base-name]\n-\t[--shallow] [--keep-true-parents] < object-list\n+\t[--shallow] [--keep-true-parents] [--sparse] < object-list\n \n \n DESCRIPTION\n@@ -196,6 +196,15 @@ depth is 4095.\n \tAdd --no-reuse-object if you want to force a uniform compression\n \tlevel on all data no matter the source.\n \n+--sparse::\n+\tUse the \"sparse\" algorithm to determine which objects to include in\n+\tthe pack, when combined with the \"--revs\" option. This algorithm\n+\tonly walks trees that appear in paths that introduce new objects.\n+\tThis can have significant performance benefits when computing\n+\ta pack to send a small change. However, it is possible that extra\n+\tobjects are added to the pack-file if the included commits contain\n+\tcertain types of direct renames.\n+\n --thin::\n \tCreate a \"thin\" pack by omitting the common objects between a\n \tsender and a receiver in order to reduce network transfer. This\ndiff --git a/bisect.c b/bisect.c\nindex 487675c672..842f8b4b8f 100644\n--- a/bisect.c\n+++ b/bisect.c\n@@ -656,7 +656,7 @@ static void bisect_common(struct rev_info *revs)\n \tif (prepare_revision_walk(revs))\n \t\tdie(\"revision walk setup failed\");\n \tif (revs->tree_objects)\n-\t\tmark_edges_uninteresting(revs, NULL);\n+\t\tmark_edges_uninteresting(revs, NULL, 0);\n }\n \n static void exit_if_skipped_commits(struct commit_list *tried,\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 411aefd687..7d5b0735e3 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -84,6 +84,7 @@ static unsigned long pack_size_limit;\n static int depth = 50;\n static int delta_search_threads;\n static int pack_to_stdout;\n+static int sparse;\n static int thin;\n static int num_preferred_base;\n static struct progress *progress_state;\n@@ -3135,7 +3136,7 @@ static void get_object_list(int ac, const char **av)\n \n \tif (prepare_revision_walk(&revs))\n \t\tdie(_(\"revision walk setup failed\"));\n-\tmark_edges_uninteresting(&revs, show_edge);\n+\tmark_edges_uninteresting(&revs, show_edge, sparse);\n \n \tif (!fn_show_object)\n \t\tfn_show_object = show_object;\n@@ -3292,6 +3293,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t{ OPTION_CALLBACK, 0, \"unpack-unreachable\", NULL, N_(\"time\"),\n \t\t  N_(\"unpack unreachable objects newer than <time>\"),\n \t\t  PARSE_OPT_OPTARG, option_parse_unpack_unreachable },\n+\t\tOPT_BOOL(0, \"sparse\", &sparse,\n+\t\t\t N_(\"use the sparse reachability algorithm\")),\n \t\tOPT_BOOL(0, \"thin\", &thin,\n \t\t\t N_(\"create thin packs\")),\n \t\tOPT_BOOL(0, \"shallow\", &shallow,\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 2880ed37e3..9663cbfae0 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -543,7 +543,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \tif (revs.tree_objects)\n-\t\tmark_edges_uninteresting(&revs, show_edge);\n+\t\tmark_edges_uninteresting(&revs, show_edge, 0);\n \n \tif (bisect_list) {\n \t\tint reaches, all;\ndiff --git a/http-push.c b/http-push.c\nindex cd48590912..ea52d6f9f6 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -1933,7 +1933,7 @@ int cmd_main(int argc, const char **argv)\n \t\tpushing = 0;\n \t\tif (prepare_revision_walk(&revs))\n \t\t\tdie(\"revision walk setup failed\");\n-\t\tmark_edges_uninteresting(&revs, NULL);\n+\t\tmark_edges_uninteresting(&revs, NULL, 0);\n \t\tobjects_to_send = get_delta(&revs, ref_lock);\n \t\tfinish_all_active_slots();\n \ndiff --git a/list-objects.c b/list-objects.c\nindex c41cc80db5..fb728f7842 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -222,25 +222,73 @@ static void mark_edge_parents_uninteresting(struct commit *commit,\n \t}\n }\n \n-void mark_edges_uninteresting(struct rev_info *revs, show_edge_fn show_edge)\n+static void add_edge_parents(struct commit *commit,\n+\t\t\t     struct rev_info *revs,\n+\t\t\t     show_edge_fn show_edge,\n+\t\t\t     struct oidset *set)\n+{\n+\tstruct commit_list *parents;\n+\n+\tfor (parents = commit->parents; parents; parents = parents->next) {\n+\t\tstruct commit *parent = parents->item;\n+\t\tstruct tree *tree = get_commit_tree(parent);\n+\n+\t\tif (!tree)\n+\t\t\tcontinue;\n+\n+\t\toidset_insert(set, &tree->object.oid);\n+\n+\t\tif (!(parent->object.flags & UNINTERESTING))\n+\t\t\tcontinue;\n+\t\ttree->object.flags |= UNINTERESTING;\n+\n+\t\tif (revs->edge_hint && !(parent->object.flags & SHOWN)) {\n+\t\t\tparent->object.flags |= SHOWN;\n+\t\t\tshow_edge(parent);\n+\t\t}\n+\t}\n+}\n+\n+void mark_edges_uninteresting(struct rev_info *revs,\n+\t\t\t      show_edge_fn show_edge,\n+\t\t\t      int sparse)\n {\n \tstruct commit_list *list;\n \tint i;\n \n-\tfor (list = revs->commits; list; list = list->next) {\n-\t\tstruct commit *commit = list->item;\n+\tif (sparse) {\n+\t\tstruct oidset set;\n+\t\toidset_init(&set, 16);\n \n-\t\tif (commit->object.flags & UNINTERESTING) {\n-\t\t\tmark_tree_uninteresting(revs->repo,\n-\t\t\t\t\t\tget_commit_tree(commit));\n-\t\t\tif (revs->edge_hint_aggressive && !(commit->object.flags & SHOWN)) {\n-\t\t\t\tcommit->object.flags |= SHOWN;\n-\t\t\t\tshow_edge(commit);\n+\t\tfor (list = revs->commits; list; list = list->next) {\n+\t\t\tstruct commit *commit = list->item;\n+\t\t\tstruct tree *tree = get_commit_tree(commit);\n+\n+\t\t\tif (commit->object.flags & UNINTERESTING)\n+\t\t\t\ttree->object.flags |= UNINTERESTING;\n+\n+\t\t\toidset_insert(&set, &tree->object.oid);\n+\t\t\tadd_edge_parents(commit, revs, show_edge, &set);\n+\t\t}\n+\n+\t\tmark_trees_uninteresting_sparse(revs->repo, &set);\n+\t\toidset_clear(&set);\n+\t} else {\n+\t\tfor (list = revs->commits; list; list = list->next) {\n+\t\t\tstruct commit *commit = list->item;\n+\t\t\tif (commit->object.flags & UNINTERESTING) {\n+\t\t\t\tmark_tree_uninteresting(revs->repo,\n+\t\t\t\t\t\t\tget_commit_tree(commit));\n+\t\t\t\tif (revs->edge_hint_aggressive && !(commit->object.flags & SHOWN)) {\n+\t\t\t\t\tcommit->object.flags |= SHOWN;\n+\t\t\t\t\tshow_edge(commit);\n+\t\t\t\t}\n+\t\t\t\tcontinue;\n \t\t\t}\n-\t\t\tcontinue;\n+\t\t\tmark_edge_parents_uninteresting(commit, revs, show_edge);\n \t\t}\n-\t\tmark_edge_parents_uninteresting(commit, revs, show_edge);\n \t}\n+\n \tif (revs->edge_hint_aggressive) {\n \t\tfor (i = 0; i < revs->cmdline.nr; i++) {\n \t\t\tstruct object *obj = revs->cmdline.rev[i].item;\ndiff --git a/list-objects.h b/list-objects.h\nindex ad40762926..a952680e46 100644\n--- a/list-objects.h\n+++ b/list-objects.h\n@@ -10,7 +10,9 @@ typedef void (*show_object_fn)(struct object *, const char *, void *);\n void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);\n \n typedef void (*show_edge_fn)(struct commit *);\n-void mark_edges_uninteresting(struct rev_info *, show_edge_fn);\n+void mark_edges_uninteresting(struct rev_info *revs,\n+\t\t\t      show_edge_fn show_edge,\n+\t\t\t      int sparse);\n \n struct oidset;\n struct list_objects_filter_options;\ndiff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\nnew file mode 100755\nindex 0000000000..30aef6498a\n--- /dev/null\n+++ b/t/t5322-pack-objects-sparse.sh\n@@ -0,0 +1,113 @@\n+#!/bin/sh\n+\n+test_description='pack-objects object selection using sparse algorithm'\n+. ./test-lib.sh\n+\n+test_expect_success 'setup repo' '\n+\ttest_commit initial &&\n+\tfor i in $(test_seq 1 3)\n+\tdo\n+\t\tmkdir f$i &&\n+\t\tfor j in $(test_seq 1 3)\n+\t\tdo\n+\t\t\tmkdir f$i/f$j &&\n+\t\t\techo $j >f$i/f$j/data.txt\n+\t\tdone\n+\tdone &&\n+\tgit add . &&\n+\tgit commit -m \"Initialized trees\" &&\n+\tfor i in $(test_seq 1 3)\n+\tdo\n+\t\tgit checkout -b topic$i master &&\n+\t\techo change-$i >f$i/f$i/data.txt &&\n+\t\tgit commit -a -m \"Changed f$i/f$i/data.txt\"\n+\tdone &&\n+\tcat >packinput.txt <<-EOF &&\n+\ttopic1\n+\t^topic2\n+\t^topic3\n+\tEOF\n+\tgit rev-parse\t\t\t\\\n+\t\ttopic1\t\t\t\\\n+\t\ttopic1^{tree}\t\t\\\n+\t\ttopic1:f1\t\t\\\n+\t\ttopic1:f1/f1\t\t\\\n+\t\ttopic1:f1/f1/data.txt | sort >expect_objects.txt\n+'\n+\n+test_expect_success 'non-sparse pack-objects' '\n+\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n+\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n+\ttest_cmp expect_objects.txt nonsparse_objects.txt\n+'\n+\n+test_expect_success 'sparse pack-objects' '\n+\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_objects.txt sparse_objects.txt\n+'\n+\n+test_expect_success 'duplicate a folder from f3 and commit to topic1' '\n+\tgit checkout topic1 &&\n+\techo change-3 >f3/f3/data.txt &&\n+\tgit commit -a -m \"Changed f3/f3/data.txt\" &&\n+\tgit rev-parse\t\t\t\\\n+\t\ttopic1~1\t\t\\\n+\t\ttopic1~1^{tree}\t\t\\\n+\t\ttopic1^{tree}\t\t\\\n+\t\ttopic1\t\t\t\\\n+\t\ttopic1:f1\t\t\\\n+\t\ttopic1:f1/f1\t\t\\\n+\t\ttopic1:f1/f1/data.txt | sort >required_objects.txt\n+'\n+\n+test_expect_success 'non-sparse pack-objects' '\n+\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n+\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n+\tcomm -1 -2 required_objects.txt nonsparse_objects.txt >nonsparse_required_objects.txt &&\n+\ttest_cmp required_objects.txt nonsparse_required_objects.txt\n+'\n+\n+test_expect_success 'sparse pack-objects' '\n+\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\tcomm -1 -2 required_objects.txt sparse_objects.txt >sparse_required_objects.txt &&\n+\ttest_cmp required_objects.txt sparse_required_objects.txt\n+'\n+\n+test_expect_success 'duplicate a folder from f1 into f3' '\n+\tmkdir f3/f4 &&\n+\tcp -r f1/f1/* f3/f4 &&\n+\tgit add f3/f4 &&\n+\tgit commit -m \"Copied f1/f1 to f3/f4\" &&\n+\tcat >packinput.txt <<-EOF &&\n+\ttopic1\n+\t^topic1~1\n+\tEOF\n+\tgit rev-parse\t\t\\\n+\t\ttopic1\t\t\\\n+\t\ttopic1^{tree}   \\\n+\t\ttopic1:f3 | sort >required_objects.txt\n+'\n+\n+test_expect_success 'non-sparse pack-objects' '\n+\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n+\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n+\tcomm -1 -2 required_objects.txt nonsparse_objects.txt >nonsparse_required_objects.txt &&\n+\ttest_cmp required_objects.txt nonsparse_required_objects.txt\n+'\n+\n+test_expect_success 'sparse pack-objects' '\n+\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\tcomm -1 -2 required_objects.txt sparse_objects.txt >sparse_required_objects.txt &&\n+\ttest_cmp required_objects.txt sparse_required_objects.txt\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"366862","messageId":"327c10247748c33edd9c18d0388f22a0289b12d9.1547663156.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 3/5] revision: implement sparse algorithm","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T18:25:59Z","receivedAt":"2019-01-16T18:26:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nWhen enumerating objects to place in a pack-file during 'git\npack-objects --revs', we discover the \"frontier\" of commits\nthat we care about and the boundary with commit we find\nuninteresting. From that point, we walk trees to discover which\ntrees and blobs are uninteresting. Finally, we walk trees from the\ninteresting commits to find the interesting objects that are\nplaced in the pack.\n\nThis commit introduces a new, \"sparse\" way to discover the\nuninteresting trees. We use the perspective of a single user trying\nto push their topic to a large repository. That user likely changed\na very small fraction of the paths in their working directory, but\nwe spend a lot of time walking all reachable trees.\n\nThe way to switch the logic to work in this sparse way is to start\ncaring about which paths introduce new trees. While it is not\npossible to generate a diff between the frontier boundary and all\nof the interesting commits, we can simulate that behavior by\ninspecting all of the root trees as a whole, then recursing down\nto the set of trees at each path.\n\nWe already had taken the first step by passing an oidset to\nmark_trees_uninteresting_sparse(). We now create a dictionary\nwhose keys are paths and values are oidsets. We consider the set\nof trees that appear at each path. While we inspect a tree, we\nadd its subtrees to the oidsets corresponding to the tree entry's\npath. We also mark trees as UNINTERESTING if the tree we are\nparsing is UNINTERESTING.\n\nTo actually improve the performance, we need to terminate our\nrecursion. If the oidset contains only UNINTERESTING trees, then\nwe do not continue the recursion. This avoids walking trees that\nare likely to not be reachable from interesting trees. If the\noidset contains only interesting trees, then we will walk these\ntrees in the final stage that collects the intersting objects to\nplace in the pack. Thus, we only recurse if the oidset contains\nboth interesting and UNINITERESTING trees.\n\nThere are a few ways that this is not a universally better option.\n\nFirst, we can pack extra objects. If someone copies a subtree\nfrom one tree to another, the first tree will appear UNINTERESTING\nand we will not recurse to see that the subtree should also be\nUNINTERESTING. We will walk the new tree and see the subtree as\na \"new\" object and add it to the pack. A test is modified to\ndemonstrate this behavior and to verify that the new logic is\nbeing exercised.\n\nSecond, we can have extra memory pressure. If instead of being a\nsingle user pushing a small topic we are a server sending new\nobjects from across the entire working directory, then we will\ngain very little (the recursion will rarely terminate early) but\nwill spend extra time maintaining the path-oidset dictionaries.\n\nDespite these potential drawbacks, the benefits of the algorithm\nare clear. By adding a counter to 'add_children_by_path' and\n'mark_tree_contents_uninteresting', I measured the number of\nparsed trees for the two algorithms in a variety of repos.\n\nFor git.git, I used the following input:\n\n\tv2.19.0\n\t^v2.19.0~10\n\n Objects to pack: 550\nWalked (old alg): 282\nWalked (new alg): 130\n\nFor the Linux repo, I used the following input:\n\n\tv4.18\n\t^v4.18~10\n\n Objects to pack:   518\nWalked (old alg): 4,836\nWalked (new alg):   188\n\nThe two repos above are rather \"wide and flat\" compared to\nother repos that I have used in the past. As a comparison,\nI tested an old topic branch in the Azure DevOps repo, which\nhas a much deeper folder structure than the Linux repo.\n\n Objects to pack:    220\nWalked (old alg): 22,804\nWalked (new alg):    129\n\nI used the number of walked trees the main metric above because\nit is consistent across multiple runs. When I ran my tests, the\nperformance of the pack-objects command with the same options\ncould change the end-to-end time by 10x depending on the file\nsystem being warm. However, by repeating the same test on repeat\nI could get more consistent timing results. The git.git and\nLinux tests were too fast overall (less than 0.5s) to measure\nan end-to-end difference. The Azure DevOps case was slow enough\nto see the time improve from 15s to 1s in the warm case. The\ncold case was 90s to 9s in my testing.\n\nThese improvements will have even larger benefits in the super-\nlarge Windows repository. In our experiments, we see the\n\"Enumerate objects\" phase of pack-objects taking 60-80% of the\nend-to-end time of non-trivial pushes, taking longer than the\nnetwork time to send the pack and the server time to verify the\npack.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n revision.c                     | 138 ++++++++++++++++++++++++++++++---\n t/t5322-pack-objects-sparse.sh |  14 +++-\n 2 files changed, 139 insertions(+), 13 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex 60421f3a10..5de739384a 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -27,6 +27,7 @@\n #include \"commit-reach.h\"\n #include \"commit-graph.h\"\n #include \"prio-queue.h\"\n+#include \"hashmap.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -99,29 +100,146 @@ void mark_tree_uninteresting(struct repository *r, struct tree *tree)\n \tmark_tree_contents_uninteresting(r, tree);\n }\n \n+struct path_and_oids_entry {\n+\tstruct hashmap_entry ent;\n+\tchar *path;\n+\tstruct oidset trees;\n+};\n+\n+static int path_and_oids_cmp(const void *hashmap_cmp_fn_data,\n+\t\t\t     const struct path_and_oids_entry *e1,\n+\t\t\t     const struct path_and_oids_entry *e2,\n+\t\t\t     const void *keydata)\n+{\n+\treturn strcmp(e1->path, e2->path);\n+}\n+\n+static void paths_and_oids_init(struct hashmap *map)\n+{\n+\thashmap_init(map, (hashmap_cmp_fn) path_and_oids_cmp, NULL, 0);\n+}\n+\n+static void paths_and_oids_clear(struct hashmap *map)\n+{\n+\tstruct hashmap_iter iter;\n+\tstruct path_and_oids_entry *entry;\n+\thashmap_iter_init(map, &iter);\n+\n+\twhile ((entry = (struct path_and_oids_entry *)hashmap_iter_next(&iter))) {\n+\t\toidset_clear(&entry->trees);\n+\t\tfree(entry->path);\n+\t}\n+\n+\thashmap_free(map, 1);\n+}\n+\n+static void paths_and_oids_insert(struct hashmap *map,\n+\t\t\t\t  const char *path,\n+\t\t\t\t  const struct object_id *oid)\n+{\n+\tint hash = strhash(path);\n+\tstruct path_and_oids_entry key;\n+\tstruct path_and_oids_entry *entry;\n+\n+\thashmap_entry_init(&key, hash);\n+\n+\t/* use a shallow copy for the lookup */\n+\tkey.path = (char *)path;\n+\toidset_init(&key.trees, 0);\n+\n+\tif (!(entry = (struct path_and_oids_entry *)hashmap_get(map, &key, NULL))) {\n+\t\tentry = xcalloc(1, sizeof(struct path_and_oids_entry));\n+\t\thashmap_entry_init(entry, hash);\n+\t\tentry->path = xstrdup(key.path);\n+\t\toidset_init(&entry->trees, 16);\n+\t\thashmap_put(map, entry);\n+\t}\n+\n+\toidset_insert(&entry->trees, oid);\n+}\n+\n+static void add_children_by_path(struct repository *r,\n+\t\t\t\t struct tree *tree,\n+\t\t\t\t struct hashmap *map)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\n+\tif (!tree)\n+\t\treturn;\n+\n+\tif (parse_tree_gently(tree, 1) < 0)\n+\t\treturn;\n+\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tswitch (object_type(entry.mode)) {\n+\t\tcase OBJ_TREE:\n+\t\t\tpaths_and_oids_insert(map, entry.path, entry.oid);\n+\n+\t\t\tif (tree->object.flags & UNINTERESTING) {\n+\t\t\t\tstruct tree *child = lookup_tree(r, entry.oid);\n+\t\t\t\tif (child)\n+\t\t\t\t\tchild->object.flags |= UNINTERESTING;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\tcase OBJ_BLOB:\n+\t\t\tif (tree->object.flags & UNINTERESTING) {\n+\t\t\t\tstruct blob *child = lookup_blob(r, entry.oid);\n+\t\t\t\tif (child)\n+\t\t\t\t\tchild->object.flags |= UNINTERESTING;\n+\t\t\t}\n+\t\t\tbreak;\n+\t\tdefault:\n+\t\t\t/* Subproject commit - not in this repository */\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n+\tfree_tree_buffer(tree);\n+}\n+\n void mark_trees_uninteresting_sparse(struct repository *r,\n \t\t\t\t     struct oidset *trees)\n {\n+\tunsigned has_interesting = 0, has_uninteresting = 0;\n+\tstruct hashmap map;\n+\tstruct hashmap_iter map_iter;\n+\tstruct path_and_oids_entry *entry;\n \tstruct object_id *oid;\n \tstruct oidset_iter iter;\n \n \toidset_iter_init(trees, &iter);\n-\twhile ((oid = oidset_iter_next(&iter))) {\n+\twhile ((!has_interesting || !has_uninteresting) &&\n+\t       (oid = oidset_iter_next(&iter))) {\n \t\tstruct tree *tree = lookup_tree(r, oid);\n \n \t\tif (!tree)\n \t\t\tcontinue;\n \n-\t\tif (tree->object.flags & UNINTERESTING) {\n-\t\t\t/*\n-\t\t\t * Remove the flag so the next call\n-\t\t\t * is not a no-op. The flag is added\n-\t\t\t * in mark_tree_unintersting().\n-\t\t\t */\n-\t\t\ttree->object.flags ^= UNINTERESTING;\n-\t\t\tmark_tree_uninteresting(r, tree);\n-\t\t}\n+\t\tif (tree->object.flags & UNINTERESTING)\n+\t\t\thas_uninteresting = 1;\n+\t\telse\n+\t\t\thas_interesting = 1;\n+\t}\n+\n+\t/* Do not walk unless we have both types of trees. */\n+\tif (!has_uninteresting || !has_interesting)\n+\t\treturn;\n+\n+\tpaths_and_oids_init(&map);\n+\n+\toidset_iter_init(trees, &iter);\n+\twhile ((oid = oidset_iter_next(&iter))) {\n+\t\tstruct tree *tree = lookup_tree(r, oid);\n+\t\tadd_children_by_path(r, tree, &map);\n \t}\n+\n+\thashmap_iter_init(&map, &map_iter);\n+\twhile ((entry = hashmap_iter_next(&map_iter)))\n+\t\tmark_trees_uninteresting_sparse(r, &entry->trees);\n+\n+\tpaths_and_oids_clear(&map);\n }\n \n struct commit_stack {\ndiff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\nindex 30aef6498a..9f2a6e5d31 100755\n--- a/t/t5322-pack-objects-sparse.sh\n+++ b/t/t5322-pack-objects-sparse.sh\n@@ -79,6 +79,9 @@ test_expect_success 'sparse pack-objects' '\n \ttest_cmp required_objects.txt sparse_required_objects.txt\n '\n \n+# Demonstrate that the algorithms differ when we copy a tree wholesale\n+# from one folder to another.\n+\n test_expect_success 'duplicate a folder from f1 into f3' '\n \tmkdir f3/f4 &&\n \tcp -r f1/f1/* f3/f4 &&\n@@ -95,7 +98,7 @@ test_expect_success 'duplicate a folder from f1 into f3' '\n '\n \n test_expect_success 'non-sparse pack-objects' '\n-\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >nonsparse.pack &&\n \tgit index-pack -o nonsparse.idx nonsparse.pack &&\n \tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n \tcomm -1 -2 required_objects.txt nonsparse_objects.txt >nonsparse_required_objects.txt &&\n@@ -103,11 +106,16 @@ test_expect_success 'non-sparse pack-objects' '\n '\n \n test_expect_success 'sparse pack-objects' '\n+\tgit rev-parse\t\t\t\\\n+\t\ttopic1\t\t\t\\\n+\t\ttopic1^{tree}\t\t\\\n+\t\ttopic1:f3\t\t\\\n+\t\ttopic1:f3/f4\t\t\\\n+\t\ttopic1:f3/f4/data.txt | sort >expect_sparse_objects.txt &&\n \tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n \tgit index-pack -o sparse.idx sparse.pack &&\n \tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n-\tcomm -1 -2 required_objects.txt sparse_objects.txt >sparse_required_objects.txt &&\n-\ttest_cmp required_objects.txt sparse_required_objects.txt\n+\ttest_cmp expect_sparse_objects.txt sparse_objects.txt\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"366861","messageId":"28111d70d448a8220ad080de3932691a8be35505.1547663156.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 4/5] pack-objects: create pack.useSparse setting","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T18:26:00Z","receivedAt":"2019-01-16T18:26:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nThe '--sparse' flag in 'git pack-objects' changes the algorithm\nused to enumerate objects to one that is faster for individual\nusers pushing new objects that change only a small cone of the\nworking directory. The sparse algorithm is not recommended for a\nserver, which likely sends new objects that appear across the\nentire working directory.\n\nCreate a 'pack.useSparse' setting that enables this new algorithm.\nThis allows 'git push' to use this algorithm without passing a\n'--sparse' flag all the way through four levels of run_command()\ncalls.\n\nIf the '--no-sparse' flag is set, then this config setting is\noverridden.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n Documentation/config/pack.txt  |  9 +++++++++\n builtin/pack-objects.c         |  4 ++++\n t/t5322-pack-objects-sparse.sh | 15 +++++++++++++++\n 3 files changed, 28 insertions(+)\n\ndiff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt\nindex edac75c83f..425c73aa52 100644\n--- a/Documentation/config/pack.txt\n+++ b/Documentation/config/pack.txt\n@@ -105,6 +105,15 @@ pack.useBitmaps::\n \ttrue. You should not generally need to turn this off unless\n \tyou are debugging pack bitmaps.\n \n+pack.useSparse::\n+\tWhen true, git will default to using the '--sparse' option in\n+\t'git pack-objects' when the '--revs' option is present. This\n+\talgorithm only walks trees that appear in paths that introduce new\n+\tobjects. This can have significant performance benefits when\n+\tcomputing a pack to send a small change. However, it is possible\n+\tthat extra objects are added to the pack-file if the included\n+\tcommits contain certain types of direct renames.\n+\n pack.writeBitmaps (deprecated)::\n \tThis is a deprecated synonym for `repack.writeBitmaps`.\n \ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 7d5b0735e3..124b1bafc4 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2711,6 +2711,10 @@ static int git_pack_config(const char *k, const char *v, void *cb)\n \t\tuse_bitmap_index_default = git_config_bool(k, v);\n \t\treturn 0;\n \t}\n+\tif (!strcmp(k, \"pack.usesparse\")) {\n+\t\tsparse = git_config_bool(k, v);\n+\t\treturn 0;\n+\t}\n \tif (!strcmp(k, \"pack.threads\")) {\n \t\tdelta_search_threads = git_config_int(k, v);\n \t\tif (delta_search_threads < 0)\ndiff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\nindex 9f2a6e5d31..3233fafc90 100755\n--- a/t/t5322-pack-objects-sparse.sh\n+++ b/t/t5322-pack-objects-sparse.sh\n@@ -118,4 +118,19 @@ test_expect_success 'sparse pack-objects' '\n \ttest_cmp expect_sparse_objects.txt sparse_objects.txt\n '\n \n+test_expect_success 'pack.useSparse enables algorithm' '\n+\tgit config pack.useSparse true &&\n+\tgit pack-objects --stdout --revs <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp expect_sparse_objects.txt sparse_objects.txt\n+'\n+\n+test_expect_success 'pack.useSparse overridden' '\n+\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >sparse.pack &&\n+\tgit index-pack -o sparse.idx sparse.pack &&\n+\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n+\ttest_cmp required_objects.txt sparse_objects.txt\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"366863","messageId":"ae046e85254d2a23cefca42b111695e6db28791c.1547663156.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 5/5] pack-objects: create GIT_TEST_PACK_SPARSE","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T18:26:01Z","receivedAt":"2019-01-16T18:26:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nCreate a test variable GIT_TEST_PACK_SPARSE to enable the sparse\nobject walk algorithm by default during the test suite. Enabling\nthis variable ensures coverage in many interesting cases, such as\nshallow clones, partial clones, and missing objects.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n builtin/pack-objects.c         | 1 +\n t/README                       | 4 ++++\n t/t5322-pack-objects-sparse.sh | 4 ++--\n 3 files changed, 7 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 124b1bafc4..507d381153 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -3331,6 +3331,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \n \tread_replace_refs = 0;\n \n+\tsparse = git_env_bool(\"GIT_TEST_PACK_SPARSE\", 0);\n \treset_pack_idx_option(&pack_idx_opts);\n \tgit_config(git_pack_config, NULL);\n \ndiff --git a/t/README b/t/README\nindex 28711cc508..8b6dfe1864 100644\n--- a/t/README\n+++ b/t/README\n@@ -342,6 +342,10 @@ GIT_TEST_INDEX_VERSION=<n> exercises the index read/write code path\n for the index version specified.  Can be set to any valid version\n (currently 2, 3, or 4).\n \n+GIT_TEST_PACK_SPARSE=<boolean> if enabled will default the pack-objects\n+builtin to use the sparse object walk. This can still be overridden by\n+the --no-sparse command-line argument.\n+\n GIT_TEST_PRELOAD_INDEX=<boolean> exercises the preload-index code path\n by overriding the minimum number of cache entries required per thread.\n \ndiff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\nindex 3233fafc90..7124b5581a 100755\n--- a/t/t5322-pack-objects-sparse.sh\n+++ b/t/t5322-pack-objects-sparse.sh\n@@ -36,7 +36,7 @@ test_expect_success 'setup repo' '\n '\n \n test_expect_success 'non-sparse pack-objects' '\n-\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >nonsparse.pack &&\n \tgit index-pack -o nonsparse.idx nonsparse.pack &&\n \tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n \ttest_cmp expect_objects.txt nonsparse_objects.txt\n@@ -64,7 +64,7 @@ test_expect_success 'duplicate a folder from f3 and commit to topic1' '\n '\n \n test_expect_success 'non-sparse pack-objects' '\n-\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n+\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >nonsparse.pack &&\n \tgit index-pack -o nonsparse.idx nonsparse.pack &&\n \tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n \tcomm -1 -2 required_objects.txt nonsparse_objects.txt >nonsparse_required_objects.txt &&\n-- \ngitgitgadget\n"},{"id":"366864","messageId":"pull.89.v5.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v5 0/5] Add a new \"sparse\" tree walk algorithm","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T18:25:56Z","receivedAt":"2019-01-16T18:26:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"One of the biggest remaining pain points for users of very large\nrepositories is the time it takes to run 'git push'. We inspected some slow\npushes by our developers and found that the \"Enumerating Objects\" phase of a\npush was very slow. This is unsurprising, because this is why reachability\nbitmaps exist. However, reachability bitmaps are not available to us because\nof the single pack-file requirement. The bitmap approach is intended for\nservers anyway, and clients have a much different behavior pattern.\n\nSpecifically, clients are normally pushing a very small number of objects\ncompared to the entire working directory. A typical user changes only a\nsmall cone of the working directory, so let's use that to our benefit.\n\nCreate a new \"sparse\" mode for 'git pack-objects' that uses the paths that\nintroduce new objects to direct our search into the reachable trees. By\ncollecting trees at each path, we can then recurse into a path only when\nthere are uninteresting and interesting trees at that path. This gains a\nsignificant performance boost for small topics while presenting a\npossibility of packing extra objects.\n\nThe main algorithm change is in patch 4, but is set up a little bit in\npatches 1 and 2.\n\nAs demonstrated in the included test script, we see that the existing\nalgorithm can send extra objects due to the way we specify the \"frontier\".\nBut we can send even more objects if a user copies objects from one folder\nto another. I say \"copy\" because a rename would (usually) change the\noriginal folder and trigger a walk into that path, discovering the objects.\n\nIn order to benefit from this approach, the user can opt-in using the\npack.useSparse config setting. This setting can be overridden using the\n'--no-sparse' option.\n\nUpdate in V2: \n\n * Added GIT_TEST_PACK_SPARSE test option.\n * Fixed test breakages when GIT_TEST_PACK_SPARSE is enabled by adding null\n   checks.\n\nUpdate in V3:\n\n * Change documentation around 'pack.useSparse' config setting to better\n   advertise to users.\n * Mostly a ping now that v2.20.0 is out.\n\nUpdates in V4:\n\n * Switched to using hashmap instead of string_list for the path/oidset\n   dictionary. (This is due to some fear that the string_list performance\n   would degrade for a very wide tree, but I am unable to measure a\n   performance difference.)\n * Some cleanup of code snippets across commits.\n * Some grammar cleanup in the commit messages.\n\nUpdates in V5:\n\n * Renamed the generic \"set\" to \"trees\"\n * Used a less-restrictive test condition when testing the old algorithm,\n   except for the case where we want to verify that the new algorithm is\n   being run.\n * Removed a global variable that was not used. Thanks, Ramsay and Junio!\n\nDerrick Stolee (5):\n  revision: add mark_tree_uninteresting_sparse\n  list-objects: consume sparse tree walk\n  revision: implement sparse algorithm\n  pack-objects: create pack.useSparse setting\n  pack-objects: create GIT_TEST_PACK_SPARSE\n\n Documentation/config/pack.txt      |   9 ++\n Documentation/git-pack-objects.txt |  11 ++-\n bisect.c                           |   2 +-\n builtin/pack-objects.c             |  10 +-\n builtin/rev-list.c                 |   2 +-\n http-push.c                        |   2 +-\n list-objects.c                     |  70 +++++++++++---\n list-objects.h                     |   4 +-\n revision.c                         | 143 +++++++++++++++++++++++++++++\n revision.h                         |   2 +\n t/README                           |   4 +\n t/t5322-pack-objects-sparse.sh     | 136 +++++++++++++++++++++++++++\n 12 files changed, 378 insertions(+), 17 deletions(-)\n create mode 100755 t/t5322-pack-objects-sparse.sh\n\n\nbase-commit: a1598010f775d82b5adf12c29d0f5bc9b41434c6\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-89%2Fderrickstolee%2Fpush%2Fsparse-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-89/derrickstolee/push/sparse-v5\nPull-Request: https://github.com/gitgitgadget/git/pull/89\n\nRange-diff vs v4:\n\n 1:  817e30a287 ! 1:  02ef702884 revision: add mark_tree_uninteresting_sparse\n     @@ -9,7 +9,10 @@\n      \n          The current implementation uses the existing\n          mark_tree_uninteresting to recursively walk the trees and blobs.\n     -    This will walk the same number of trees as the old mechanism.\n     +    This will walk the same number of trees as the old mechanism. To\n     +    ensure that mark_tree_uninteresting walks the tree, we need to\n     +    remove the UNINTERESTING flag before calling the method. This\n     +    implementation will be replaced entirely in a later commit.\n      \n          There is one new assumption in this approach: we are also given\n          the oids of the interesting trees. This implementation does not\n     @@ -26,12 +29,12 @@\n       }\n       \n      +void mark_trees_uninteresting_sparse(struct repository *r,\n     -+\t\t\t\t     struct oidset *set)\n     ++\t\t\t\t     struct oidset *trees)\n      +{\n      +\tstruct object_id *oid;\n      +\tstruct oidset_iter iter;\n      +\n     -+\toidset_iter_init(set, &iter);\n     ++\toidset_iter_init(trees, &iter);\n      +\twhile ((oid = oidset_iter_next(&iter))) {\n      +\t\tstruct tree *tree = lookup_tree(r, oid);\n      +\n     @@ -69,7 +72,7 @@\n       \n       void mark_parents_uninteresting(struct commit *commit);\n       void mark_tree_uninteresting(struct repository *r, struct tree *tree);\n     -+void mark_trees_uninteresting_sparse(struct repository *r, struct oidset *set);\n     ++void mark_trees_uninteresting_sparse(struct repository *r, struct oidset *trees);\n       \n       void show_object_with_name(FILE *, struct object *, const char *);\n       \n 2:  39dc89beb9 ! 2:  0747f7494e list-objects: consume sparse tree walk\n     @@ -35,18 +35,53 @@\n          UNINTERSTING.\n      \n          Add a new 'sparse' option to the mark_edges_uninteresting() method\n     -    that performs this logic in a slightly new way. As we iterate over\n     -    the commits, we add all of the root trees to an oidset. Then, call\n     -    mark_trees_uninteresting_sparse() on that oidset. Note that we\n     +    that performs this logic in a slightly different way. As we iterate\n     +    over the commits, we add all of the root trees to an oidset. Then,\n     +    call mark_trees_uninteresting_sparse() on that oidset. Note that we\n          include interesting trees in this process. The current implementation\n          of mark_trees_unintersting_sparse() will walk the same trees as\n          the old logic, but this will be replaced in a later change.\n      \n     -    The sparse option is not used by any callers at the moment, but\n     -    will be wired to 'git pack-objects' in the next change.\n     +    Add a '--sparse' flag in 'git pack-objects' to call this new logic.\n     +    Add a new test script t/t5322-pack-objects-sparse.sh that tests this\n     +    option. The tests currently demonstrate that the resulting object\n     +    list is the same as the old algorithm. This includes a case where\n     +    both algorithms pack an object that is not needed by a remote due to\n     +    limits on the explored set of trees. When the sparse algorithm is\n     +    changed in a later commit, we will add a test that demonstrates a\n     +    change of behavior in some cases.\n      \n          Signed-off-by: Derrick Stolee <dstolee@microsoft.com>\n      \n     + diff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\n     + --- a/Documentation/git-pack-objects.txt\n     + +++ b/Documentation/git-pack-objects.txt\n     +@@\n     + \t[--local] [--incremental] [--window=<n>] [--depth=<n>]\n     + \t[--revs [--unpacked | --all]] [--keep-pack=<pack-name>]\n     + \t[--stdout [--filter=<filter-spec>] | base-name]\n     +-\t[--shallow] [--keep-true-parents] < object-list\n     ++\t[--shallow] [--keep-true-parents] [--sparse] < object-list\n     + \n     + \n     + DESCRIPTION\n     +@@\n     + \tAdd --no-reuse-object if you want to force a uniform compression\n     + \tlevel on all data no matter the source.\n     + \n     ++--sparse::\n     ++\tUse the \"sparse\" algorithm to determine which objects to include in\n     ++\tthe pack, when combined with the \"--revs\" option. This algorithm\n     ++\tonly walks trees that appear in paths that introduce new objects.\n     ++\tThis can have significant performance benefits when computing\n     ++\ta pack to send a small change. However, it is possible that extra\n     ++\tobjects are added to the pack-file if the included commits contain\n     ++\tcertain types of direct renames.\n     ++\n     + --thin::\n     + \tCreate a \"thin\" pack by omitting the common objects between a\n     + \tsender and a receiver in order to reduce network transfer. This\n     +\n       diff --git a/bisect.c b/bisect.c\n       --- a/bisect.c\n       +++ b/bisect.c\n     @@ -64,14 +99,31 @@\n       --- a/builtin/pack-objects.c\n       +++ b/builtin/pack-objects.c\n      @@\n     + static int depth = 50;\n     + static int delta_search_threads;\n     + static int pack_to_stdout;\n     ++static int sparse;\n     + static int thin;\n     + static int num_preferred_base;\n     + static struct progress *progress_state;\n     +@@\n       \n       \tif (prepare_revision_walk(&revs))\n       \t\tdie(_(\"revision walk setup failed\"));\n      -\tmark_edges_uninteresting(&revs, show_edge);\n     -+\tmark_edges_uninteresting(&revs, show_edge, 0);\n     ++\tmark_edges_uninteresting(&revs, show_edge, sparse);\n       \n       \tif (!fn_show_object)\n       \t\tfn_show_object = show_object;\n     +@@\n     + \t\t{ OPTION_CALLBACK, 0, \"unpack-unreachable\", NULL, N_(\"time\"),\n     + \t\t  N_(\"unpack unreachable objects newer than <time>\"),\n     + \t\t  PARSE_OPT_OPTARG, option_parse_unpack_unreachable },\n     ++\t\tOPT_BOOL(0, \"sparse\", &sparse,\n     ++\t\t\t N_(\"use the sparse reachability algorithm\")),\n     + \t\tOPT_BOOL(0, \"thin\", &thin,\n     + \t\t\t N_(\"create thin packs\")),\n     + \t\tOPT_BOOL(0, \"shallow\", &shallow,\n      \n       diff --git a/builtin/rev-list.c b/builtin/rev-list.c\n       --- a/builtin/rev-list.c\n     @@ -202,3 +254,122 @@\n       \n       struct oidset;\n       struct list_objects_filter_options;\n     +\n     + diff --git a/t/t5322-pack-objects-sparse.sh b/t/t5322-pack-objects-sparse.sh\n     + new file mode 100755\n     + --- /dev/null\n     + +++ b/t/t5322-pack-objects-sparse.sh\n     +@@\n     ++#!/bin/sh\n     ++\n     ++test_description='pack-objects object selection using sparse algorithm'\n     ++. ./test-lib.sh\n     ++\n     ++test_expect_success 'setup repo' '\n     ++\ttest_commit initial &&\n     ++\tfor i in $(test_seq 1 3)\n     ++\tdo\n     ++\t\tmkdir f$i &&\n     ++\t\tfor j in $(test_seq 1 3)\n     ++\t\tdo\n     ++\t\t\tmkdir f$i/f$j &&\n     ++\t\t\techo $j >f$i/f$j/data.txt\n     ++\t\tdone\n     ++\tdone &&\n     ++\tgit add . &&\n     ++\tgit commit -m \"Initialized trees\" &&\n     ++\tfor i in $(test_seq 1 3)\n     ++\tdo\n     ++\t\tgit checkout -b topic$i master &&\n     ++\t\techo change-$i >f$i/f$i/data.txt &&\n     ++\t\tgit commit -a -m \"Changed f$i/f$i/data.txt\"\n     ++\tdone &&\n     ++\tcat >packinput.txt <<-EOF &&\n     ++\ttopic1\n     ++\t^topic2\n     ++\t^topic3\n     ++\tEOF\n     ++\tgit rev-parse\t\t\t\\\n     ++\t\ttopic1\t\t\t\\\n     ++\t\ttopic1^{tree}\t\t\\\n     ++\t\ttopic1:f1\t\t\\\n     ++\t\ttopic1:f1/f1\t\t\\\n     ++\t\ttopic1:f1/f1/data.txt | sort >expect_objects.txt\n     ++'\n     ++\n     ++test_expect_success 'non-sparse pack-objects' '\n     ++\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n     ++\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n     ++\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n     ++\ttest_cmp expect_objects.txt nonsparse_objects.txt\n     ++'\n     ++\n     ++test_expect_success 'sparse pack-objects' '\n     ++\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n     ++\tgit index-pack -o sparse.idx sparse.pack &&\n     ++\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n     ++\ttest_cmp expect_objects.txt sparse_objects.txt\n     ++'\n     ++\n     ++test_expect_success 'duplicate a folder from f3 and commit to topic1' '\n     ++\tgit checkout topic1 &&\n     ++\techo change-3 >f3/f3/data.txt &&\n     ++\tgit commit -a -m \"Changed f3/f3/data.txt\" &&\n     ++\tgit rev-parse\t\t\t\\\n     ++\t\ttopic1~1\t\t\\\n     ++\t\ttopic1~1^{tree}\t\t\\\n     ++\t\ttopic1^{tree}\t\t\\\n     ++\t\ttopic1\t\t\t\\\n     ++\t\ttopic1:f1\t\t\\\n     ++\t\ttopic1:f1/f1\t\t\\\n     ++\t\ttopic1:f1/f1/data.txt | sort >required_objects.txt\n     ++'\n     ++\n     ++test_expect_success 'non-sparse pack-objects' '\n     ++\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n     ++\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n     ++\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n     ++\tcomm -1 -2 required_objects.txt nonsparse_objects.txt >nonsparse_required_objects.txt &&\n     ++\ttest_cmp required_objects.txt nonsparse_required_objects.txt\n     ++'\n     ++\n     ++test_expect_success 'sparse pack-objects' '\n     ++\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n     ++\tgit index-pack -o sparse.idx sparse.pack &&\n     ++\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n     ++\tcomm -1 -2 required_objects.txt sparse_objects.txt >sparse_required_objects.txt &&\n     ++\ttest_cmp required_objects.txt sparse_required_objects.txt\n     ++'\n     ++\n     ++test_expect_success 'duplicate a folder from f1 into f3' '\n     ++\tmkdir f3/f4 &&\n     ++\tcp -r f1/f1/* f3/f4 &&\n     ++\tgit add f3/f4 &&\n     ++\tgit commit -m \"Copied f1/f1 to f3/f4\" &&\n     ++\tcat >packinput.txt <<-EOF &&\n     ++\ttopic1\n     ++\t^topic1~1\n     ++\tEOF\n     ++\tgit rev-parse\t\t\\\n     ++\t\ttopic1\t\t\\\n     ++\t\ttopic1^{tree}   \\\n     ++\t\ttopic1:f3 | sort >required_objects.txt\n     ++'\n     ++\n     ++test_expect_success 'non-sparse pack-objects' '\n     ++\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n     ++\tgit index-pack -o nonsparse.idx nonsparse.pack &&\n     ++\tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n     ++\tcomm -1 -2 required_objects.txt nonsparse_objects.txt >nonsparse_required_objects.txt &&\n     ++\ttest_cmp required_objects.txt nonsparse_required_objects.txt\n     ++'\n     ++\n     ++test_expect_success 'sparse pack-objects' '\n     ++\tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n     ++\tgit index-pack -o sparse.idx sparse.pack &&\n     ++\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n     ++\tcomm -1 -2 required_objects.txt sparse_objects.txt >sparse_required_objects.txt &&\n     ++\ttest_cmp required_objects.txt sparse_required_objects.txt\n     ++'\n     ++\n     ++test_done\n 3:  ab733daff5 < -:  ---------- pack-objects: add --sparse option\n 4:  c44172c35e ! 3:  327c102477 revision: implement sparse algorithm\n     @@ -31,7 +31,7 @@\n          path. We also mark trees as UNINTERESTING if the tree we are\n          parsing is UNINTERESTING.\n      \n     -    To actually improve the peformance, we need to terminate our\n     +    To actually improve the performance, we need to terminate our\n          recursion. If the oidset contains only UNINTERESTING trees, then\n          we do not continue the recursion. This avoids walking trees that\n          are likely to not be reachable from interesting trees. If the\n     @@ -46,9 +46,9 @@\n          from one tree to another, the first tree will appear UNINTERESTING\n          and we will not recurse to see that the subtree should also be\n          UNINTERESTING. We will walk the new tree and see the subtree as\n     -    a \"new\" object and add it to the pack. We add a test case that\n     -    demonstrates this as a way to prove that the --sparse option is\n     -    actually working.\n     +    a \"new\" object and add it to the pack. A test is modified to\n     +    demonstrate this behavior and to verify that the new logic is\n     +    being exercised.\n      \n          Second, we can have extra memory pressure. If instead of being a\n          single user pushing a small topic we are a server sending new\n     @@ -126,7 +126,7 @@\n      +struct path_and_oids_entry {\n      +\tstruct hashmap_entry ent;\n      +\tchar *path;\n     -+\tstruct oidset set;\n     ++\tstruct oidset trees;\n      +};\n      +\n      +static int path_and_oids_cmp(const void *hashmap_cmp_fn_data,\n     @@ -137,10 +137,9 @@\n      +\treturn strcmp(e1->path, e2->path);\n      +}\n      +\n     -+int map_flags = 0;\n      +static void paths_and_oids_init(struct hashmap *map)\n      +{\n     -+\thashmap_init(map, (hashmap_cmp_fn) path_and_oids_cmp, &map_flags, 0);\n     ++\thashmap_init(map, (hashmap_cmp_fn) path_and_oids_cmp, NULL, 0);\n      +}\n      +\n      +static void paths_and_oids_clear(struct hashmap *map)\n     @@ -150,7 +149,7 @@\n      +\thashmap_iter_init(map, &iter);\n      +\n      +\twhile ((entry = (struct path_and_oids_entry *)hashmap_iter_next(&iter))) {\n     -+\t\toidset_clear(&entry->set);\n     ++\t\toidset_clear(&entry->trees);\n      +\t\tfree(entry->path);\n      +\t}\n      +\n     @@ -166,20 +165,20 @@\n      +\tstruct path_and_oids_entry *entry;\n      +\n      +\thashmap_entry_init(&key, hash);\n     -+\tkey.path = xstrdup(path);\n     -+\toidset_init(&key.set, 0);\n     ++\n     ++\t/* use a shallow copy for the lookup */\n     ++\tkey.path = (char *)path;\n     ++\toidset_init(&key.trees, 0);\n      +\n      +\tif (!(entry = (struct path_and_oids_entry *)hashmap_get(map, &key, NULL))) {\n      +\t\tentry = xcalloc(1, sizeof(struct path_and_oids_entry));\n      +\t\thashmap_entry_init(entry, hash);\n     -+\t\tentry->path = key.path;\n     -+\t\toidset_init(&entry->set, 16);\n     ++\t\tentry->path = xstrdup(key.path);\n     ++\t\toidset_init(&entry->trees, 16);\n      +\t\thashmap_put(map, entry);\n     -+\t} else {\n     -+\t\tfree(key.path);\n      +\t}\n      +\n     -+\toidset_insert(&entry->set, oid);\n     ++\toidset_insert(&entry->trees, oid);\n      +}\n      +\n      +static void add_children_by_path(struct repository *r,\n     @@ -224,7 +223,7 @@\n      +}\n      +\n       void mark_trees_uninteresting_sparse(struct repository *r,\n     - \t\t\t\t     struct oidset *set)\n     + \t\t\t\t     struct oidset *trees)\n       {\n      +\tunsigned has_interesting = 0, has_uninteresting = 0;\n      +\tstruct hashmap map;\n     @@ -233,7 +232,7 @@\n       \tstruct object_id *oid;\n       \tstruct oidset_iter iter;\n       \n     - \toidset_iter_init(set, &iter);\n     + \toidset_iter_init(trees, &iter);\n      -\twhile ((oid = oidset_iter_next(&iter))) {\n      +\twhile ((!has_interesting || !has_uninteresting) &&\n      +\t       (oid = oidset_iter_next(&iter))) {\n     @@ -263,7 +262,7 @@\n      +\n      +\tpaths_and_oids_init(&map);\n      +\n     -+\toidset_iter_init(set, &iter);\n     ++\toidset_iter_init(trees, &iter);\n      +\twhile ((oid = oidset_iter_next(&iter))) {\n      +\t\tstruct tree *tree = lookup_tree(r, oid);\n      +\t\tadd_children_by_path(r, tree, &map);\n     @@ -271,7 +270,7 @@\n      +\n      +\thashmap_iter_init(&map, &map_iter);\n      +\twhile ((entry = hashmap_iter_next(&map_iter)))\n     -+\t\tmark_trees_uninteresting_sparse(r, &entry->set);\n     ++\t\tmark_trees_uninteresting_sparse(r, &entry->trees);\n      +\n      +\tpaths_and_oids_clear(&map);\n       }\n     @@ -282,7 +281,7 @@\n       --- a/t/t5322-pack-objects-sparse.sh\n       +++ b/t/t5322-pack-objects-sparse.sh\n      @@\n     - \ttest_cmp expect_objects.txt sparse_objects.txt\n     + \ttest_cmp required_objects.txt sparse_required_objects.txt\n       '\n       \n      +# Demonstrate that the algorithms differ when we copy a tree wholesale\n     @@ -291,27 +290,15 @@\n       test_expect_success 'duplicate a folder from f1 into f3' '\n       \tmkdir f3/f4 &&\n       \tcp -r f1/f1/* f3/f4 &&\n     - \tgit add f3/f4 &&\n     - \tgit commit -m \"Copied f1/f1 to f3/f4\" &&\n     --\tcat >packinput.txt <<-EOF &&\n     -+\tcat >packinput.txt <<-EOF\n     - \ttopic1\n     - \t^topic1~1\n     - \tEOF\n     --\tgit rev-parse\t\t\\\n     --\t\ttopic1\t\t\\\n     --\t\ttopic1^{tree}\t\\\n     --\t\ttopic1:f3 | sort >expect_objects.txt\n     +@@\n       '\n       \n       test_expect_success 'non-sparse pack-objects' '\n     -+\tgit rev-parse\t\t\t\\\n     -+\t\ttopic1\t\t\t\\\n     -+\t\ttopic1^{tree}\t\t\\\n     -+\t\ttopic1:f3 | sort >expect_objects.txt &&\n     - \tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n     +-\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n     ++\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >nonsparse.pack &&\n       \tgit index-pack -o nonsparse.idx nonsparse.pack &&\n       \tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n     + \tcomm -1 -2 required_objects.txt nonsparse_objects.txt >nonsparse_required_objects.txt &&\n      @@\n       '\n       \n     @@ -325,7 +312,8 @@\n       \tgit pack-objects --stdout --revs --sparse <packinput.txt >sparse.pack &&\n       \tgit index-pack -o sparse.idx sparse.pack &&\n       \tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n     --\ttest_cmp expect_objects.txt sparse_objects.txt\n     +-\tcomm -1 -2 required_objects.txt sparse_objects.txt >sparse_required_objects.txt &&\n     +-\ttest_cmp required_objects.txt sparse_required_objects.txt\n      +\ttest_cmp expect_sparse_objects.txt sparse_objects.txt\n       '\n       \n 5:  f386f6c3c9 ! 4:  28111d70d4 pack-objects: create pack.useSparse setting\n     @@ -73,7 +73,7 @@\n      +\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >sparse.pack &&\n      +\tgit index-pack -o sparse.idx sparse.pack &&\n      +\tgit show-index <sparse.idx | awk \"{print \\$2}\" >sparse_objects.txt &&\n     -+\ttest_cmp expect_objects.txt sparse_objects.txt\n     ++\ttest_cmp required_objects.txt sparse_objects.txt\n      +'\n      +\n       test_done\n 6:  d011a9c1b1 ! 5:  ae046e8525 pack-objects: create GIT_TEST_PACK_SPARSE\n     @@ -56,13 +56,4 @@\n      +\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >nonsparse.pack &&\n       \tgit index-pack -o nonsparse.idx nonsparse.pack &&\n       \tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n     - \ttest_cmp expect_objects.txt nonsparse_objects.txt\n     -@@\n     - \t\ttopic1\t\t\t\\\n     - \t\ttopic1^{tree}\t\t\\\n     - \t\ttopic1:f3 | sort >expect_objects.txt &&\n     --\tgit pack-objects --stdout --revs <packinput.txt >nonsparse.pack &&\n     -+\tgit pack-objects --stdout --revs --no-sparse <packinput.txt >nonsparse.pack &&\n     - \tgit index-pack -o nonsparse.idx nonsparse.pack &&\n     - \tgit show-index <nonsparse.idx | awk \"{print \\$2}\" >nonsparse_objects.txt &&\n     - \ttest_cmp expect_objects.txt nonsparse_objects.txt\n     + \tcomm -1 -2 required_objects.txt nonsparse_objects.txt >nonsparse_required_objects.txt &&\n\n-- \ngitgitgadget\n"},{"id":"366865","messageId":"02ef702884df2ad96be25486837b4009fb997c97.1547663156.git.gitgitgadget@gmail.com","threadId":"49914","inReplyTo":"pull.89.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 1/5] revision: add mark_tree_uninteresting_sparse","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T18:25:58Z","receivedAt":"2019-01-16T18:26:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee <dstolee@microsoft.com>\n\nIn preparation for a new algorithm that walks fewer trees when\ncreating a pack from a set of revisions, create a method that\ntakes an oidset of tree oids and marks reachable objects as\nUNINTERESTING.\n\nThe current implementation uses the existing\nmark_tree_uninteresting to recursively walk the trees and blobs.\nThis will walk the same number of trees as the old mechanism. To\nensure that mark_tree_uninteresting walks the tree, we need to\nremove the UNINTERESTING flag before calling the method. This\nimplementation will be replaced entirely in a later commit.\n\nThere is one new assumption in this approach: we are also given\nthe oids of the interesting trees. This implementation does not\nuse those trees at the moment, but we will use them in a later\nrewrite of this method.\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n revision.c | 25 +++++++++++++++++++++++++\n revision.h |  2 ++\n 2 files changed, 27 insertions(+)\n\ndiff --git a/revision.c b/revision.c\nindex 13e0519c02..60421f3a10 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -99,6 +99,31 @@ void mark_tree_uninteresting(struct repository *r, struct tree *tree)\n \tmark_tree_contents_uninteresting(r, tree);\n }\n \n+void mark_trees_uninteresting_sparse(struct repository *r,\n+\t\t\t\t     struct oidset *trees)\n+{\n+\tstruct object_id *oid;\n+\tstruct oidset_iter iter;\n+\n+\toidset_iter_init(trees, &iter);\n+\twhile ((oid = oidset_iter_next(&iter))) {\n+\t\tstruct tree *tree = lookup_tree(r, oid);\n+\n+\t\tif (!tree)\n+\t\t\tcontinue;\n+\n+\t\tif (tree->object.flags & UNINTERESTING) {\n+\t\t\t/*\n+\t\t\t * Remove the flag so the next call\n+\t\t\t * is not a no-op. The flag is added\n+\t\t\t * in mark_tree_unintersting().\n+\t\t\t */\n+\t\t\ttree->object.flags ^= UNINTERESTING;\n+\t\t\tmark_tree_uninteresting(r, tree);\n+\t\t}\n+\t}\n+}\n+\n struct commit_stack {\n \tstruct commit **items;\n \tsize_t nr, alloc;\ndiff --git a/revision.h b/revision.h\nindex 7987bfcd2e..df684701b9 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -67,6 +67,7 @@ struct rev_cmdline_info {\n #define REVISION_WALK_NO_WALK_SORTED 1\n #define REVISION_WALK_NO_WALK_UNSORTED 2\n \n+struct oidset;\n struct topo_walk_info;\n \n struct rev_info {\n@@ -327,6 +328,7 @@ void put_revision_mark(const struct rev_info *revs,\n \n void mark_parents_uninteresting(struct commit *commit);\n void mark_tree_uninteresting(struct repository *r, struct tree *tree);\n+void mark_trees_uninteresting_sparse(struct repository *r, struct oidset *trees);\n \n void show_object_with_name(FILE *, struct object *, const char *);\n \n-- \ngitgitgadget\n\n"}]}