{"thread":{"id":"38214","subject":"[PATCH v2 1/3] Documentation: add missing article in rev-list-options.txt","startedAt":"2014-12-20T22:51:10Z","lastAt":"2014-12-20T22:51:13Z","messageCount":4,"participants":["brian m. carlson"],"isPatch":true,"patchVersion":2,"patchTotal":3},"messages":[{"id":"253884","messageId":"1419115873-166686-1-git-send-email-sandals@crustytoothpaste.net","threadId":"38214","inReplyTo":null,"subject":"[PATCH v2 0/3] Improve push performance with lots of refs","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2014-12-20T22:51:10Z","receivedAt":"2014-12-20T22:51:10Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"This series contains patches to address a significant push performance\nregression in repositories with large amounts of refs.  It avoids\nperforming expensive edge marking unless the repository is shallow.\n\nThe first patch in the series is a fix for a minor typo I discovered\nwhen editing the documentation.  The second patch implements git\nrev-list --objects-edge-aggressive, and the final patch ensures it's\nused for shallow repos only.  As the final patch was suggested by Junio,\nit will need his sign-off.\n\nI considered Junio's suggestion for a --thin-aggressive, but felt that\nit was better to have the fix localized to a single location to improve\nmaintainability and that --thin-aggressive would be uncommonly used\noutside of automatic invocations.\n\nAlso, as I understand it, the goal of thin packs is to improve\nperformance by sending fewer objects.  In most cases, the benefit of the\ndecreased time spent marking edges push times will dwarf the increase in\ntime it takes for a slightly larger pack to go over the wire, so it\ndoesn't make sense to make this a tunable.\n\nThe original fix was suggested by Duy Nguyen.\n\nbrian m. carlson (3):\n  Documentation: add missing article in rev-list-options.txt\n  rev-list: add an option to mark fewer edges as uninteresting\n  pack-objects: use --objects-edge-aggressive only for shallow repos\n\n Documentation/git-rev-list.txt     | 3 ++-\n Documentation/rev-list-options.txt | 7 ++++++-\n builtin/pack-objects.c             | 4 +++-\n list-objects.c                     | 4 ++--\n revision.c                         | 6 ++++++\n revision.h                         | 1 +\n 6 files changed, 20 insertions(+), 5 deletions(-)\n\n-- \n2.2.1.209.g41e5f3a\n"},{"id":"253883","messageId":"1419115873-166686-2-git-send-email-sandals@crustytoothpaste.net","threadId":"38214","inReplyTo":"1419115873-166686-1-git-send-email-sandals@crustytoothpaste.net","subject":"[PATCH v2 1/3] Documentation: add missing article in rev-list-options.txt","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2014-12-20T22:51:11Z","receivedAt":"2014-12-20T22:51:11Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Add the missing article \"a\".\n\nSigned-off-by: brian m. carlson <sandals@crustytoothpaste.net>\n---\n Documentation/rev-list-options.txt | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex afccfdc..2277fcb 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -653,7 +653,7 @@ These options are mostly targeted for packing of Git repositories.\n --objects-edge::\n \tSimilar to `--objects`, but also print the IDs of excluded\n \tcommits prefixed with a ``-'' character.  This is used by\n-\tlinkgit:git-pack-objects[1] to build ``thin'' pack, which records\n+\tlinkgit:git-pack-objects[1] to build a ``thin'' pack, which records\n \tobjects in deltified form based on objects contained in these\n \texcluded commits to reduce network traffic.\n \n-- \n2.2.1.209.g41e5f3a\n"},{"id":"253886","messageId":"1419115873-166686-3-git-send-email-sandals@crustytoothpaste.net","threadId":"38214","inReplyTo":"1419115873-166686-1-git-send-email-sandals@crustytoothpaste.net","subject":"[PATCH v2 2/3] rev-list: add an option to mark fewer edges as uninteresting","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2014-12-20T22:51:12Z","receivedAt":"2014-12-20T22:51:12Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"In commit fbd4a70 (list-objects: mark more commits as edges in\nmark_edges_uninteresting - 2013-08-16), we marked an increasing number\nof edges uninteresting.  This change, and the subsequent change to make\nthis conditional on --objects-edge, are used by --thin to make much\nsmaller packs for shallow clones.\n\nUnfortunately, they cause a significant performance regression when\npushing non-shallow clones with lots of refs (23.322 seconds vs.\n4.785 seconds with 22400 refs).  Add an option to git rev-list,\n--objects-edge-aggressive, that preserves this more aggressive behavior,\nwhile leaving --objects-edge to provide more performant behavior.\n\nSigned-off-by: brian m. carlson <sandals@crustytoothpaste.net>\n---\n Documentation/git-rev-list.txt     | 3 ++-\n Documentation/rev-list-options.txt | 4 ++++\n list-objects.c                     | 4 ++--\n revision.c                         | 6 ++++++\n revision.h                         | 1 +\n 5 files changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/git-rev-list.txt b/Documentation/git-rev-list.txt\nindex fd7f8b5..5b11922 100644\n--- a/Documentation/git-rev-list.txt\n+++ b/Documentation/git-rev-list.txt\n@@ -46,7 +46,8 @@ SYNOPSIS\n \t     [ \\--extended-regexp | -E ]\n \t     [ \\--fixed-strings | -F ]\n \t     [ \\--date=(local|relative|default|iso|iso-strict|rfc|short) ]\n-\t     [ [\\--objects | \\--objects-edge] [ \\--unpacked ] ]\n+\t     [ [ \\--objects | \\--objects-edge | \\--objects-edge-aggressive ]\n+\t       [ \\--unpacked ] ]\n \t     [ \\--pretty | \\--header ]\n \t     [ \\--bisect ]\n \t     [ \\--bisect-vars ]\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 2277fcb..8cb6f92 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -657,6 +657,10 @@ These options are mostly targeted for packing of Git repositories.\n \tobjects in deltified form based on objects contained in these\n \texcluded commits to reduce network traffic.\n \n+--objects-edge-aggressive::\n+\tSimilar to `--objects-edge`, but it tries harder to find excluded\n+\tcommits at the cost of increased time.\n+\n --unpacked::\n \tOnly useful with `--objects`; print the object IDs that are not\n \tin packs.\ndiff --git a/list-objects.c b/list-objects.c\nindex 2910bec..2a139b6 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -157,7 +157,7 @@ void mark_edges_uninteresting(struct rev_info *revs, show_edge_fn show_edge)\n \n \t\tif (commit->object.flags & UNINTERESTING) {\n \t\t\tmark_tree_uninteresting(commit->tree);\n-\t\t\tif (revs->edge_hint && !(commit->object.flags & SHOWN)) {\n+\t\t\tif (revs->edge_hint_aggressive && !(commit->object.flags & SHOWN)) {\n \t\t\t\tcommit->object.flags |= SHOWN;\n \t\t\t\tshow_edge(commit);\n \t\t\t}\n@@ -165,7 +165,7 @@ void mark_edges_uninteresting(struct rev_info *revs, show_edge_fn show_edge)\n \t\t}\n \t\tmark_edge_parents_uninteresting(commit, revs, show_edge);\n \t}\n-\tif (revs->edge_hint) {\n+\tif (revs->edge_hint_aggressive) {\n \t\tfor (i = 0; i < revs->cmdline.nr; i++) {\n \t\t\tstruct object *obj = revs->cmdline.rev[i].item;\n \t\t\tstruct commit *commit = (struct commit *)obj;\ndiff --git a/revision.c b/revision.c\nindex 75dda92..753dd2f 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1853,6 +1853,12 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg\n \t\trevs->tree_objects = 1;\n \t\trevs->blob_objects = 1;\n \t\trevs->edge_hint = 1;\n+\t} else if (!strcmp(arg, \"--objects-edge-aggressive\")) {\n+\t\trevs->tag_objects = 1;\n+\t\trevs->tree_objects = 1;\n+\t\trevs->blob_objects = 1;\n+\t\trevs->edge_hint = 1;\n+\t\trevs->edge_hint_aggressive = 1;\n \t} else if (!strcmp(arg, \"--verify-objects\")) {\n \t\trevs->tag_objects = 1;\n \t\trevs->tree_objects = 1;\ndiff --git a/revision.h b/revision.h\nindex 9cb5adc..033a244 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -93,6 +93,7 @@ struct rev_info {\n \t\t\tblob_objects:1,\n \t\t\tverify_objects:1,\n \t\t\tedge_hint:1,\n+\t\t\tedge_hint_aggressive:1,\n \t\t\tlimited:1,\n \t\t\tunpacked:1,\n \t\t\tboundary:2,\n-- \n2.2.1.209.g41e5f3a\n"},{"id":"253885","messageId":"1419115873-166686-4-git-send-email-sandals@crustytoothpaste.net","threadId":"38214","inReplyTo":"1419115873-166686-1-git-send-email-sandals@crustytoothpaste.net","subject":"[PATCH v2 3/3] pack-objects: use --objects-edge-aggressive only for shallow repos","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2014-12-20T22:51:13Z","receivedAt":"2014-12-20T22:51:13Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Using --objects-edge-aggressive is important for shallow repos, as it\ncan result in a much smaller pack (in some cases, 10% of the size).\nHowever, it performs poorly on large non-shallow repositories with many\nrefs.  Since shallow repositories are less likely to have many refs (due\nto having less history), the smaller pack size is advantageous there.\nAdjust pack-objects to use --objects-edge-aggressive only for shallow\nrepositories.\n\nSigned-off-by: brian m. carlson <sandals@crustytoothpaste.net>\n---\n Documentation/rev-list-options.txt | 3 ++-\n builtin/pack-objects.c             | 4 +++-\n 2 files changed, 5 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 8cb6f92..2984f40 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -659,7 +659,8 @@ These options are mostly targeted for packing of Git repositories.\n \n --objects-edge-aggressive::\n \tSimilar to `--objects-edge`, but it tries harder to find excluded\n-\tcommits at the cost of increased time.\n+\tcommits at the cost of increased time.  This is used instead of\n+\t`--objects-edge` to build ``thin'' packs for shallow repositories.\n \n --unpacked::\n \tOnly useful with `--objects`; print the object IDs that are not\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 3f9f5c7..f3ba861 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2711,7 +2711,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \targv_array_push(&rp, \"pack-objects\");\n \tif (thin) {\n \t\tuse_internal_rev_list = 1;\n-\t\targv_array_push(&rp, \"--objects-edge\");\n+\t\targv_array_push(&rp, is_repository_shallow()\n+\t\t\t\t? \"--objects-edge-aggressive\"\n+\t\t\t\t: \"--objects-edge\");\n \t} else\n \t\targv_array_push(&rp, \"--objects\");\n \n-- \n2.2.1.209.g41e5f3a\n"}]}