{"thread":{"id":"38249","subject":"[PATCH v4 3/3] pack-objects: use --objects-edge-aggressive for shallow repos","startedAt":"2014-12-24T23:05:37Z","lastAt":"2014-12-24T23:05:40Z","messageCount":4,"participants":["brian m. carlson"],"isPatch":true,"patchVersion":4,"patchTotal":3},"messages":[{"id":"254091","messageId":"1419462340-769147-1-git-send-email-sandals@crustytoothpaste.net","threadId":"38249","inReplyTo":null,"subject":"[PATCH v4 0/3] Improve push performance with lots of refs","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2014-12-24T23:05:37Z","receivedAt":"2014-12-24T23:05:37Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"This series contains patches to address a significant push performance\nregression in repositories with large amounts of refs. It avoids\nperforming expensive edge marking unless the repository is shallow.\n\nThe first patch in the series is a fix for a minor typo I discovered\nwhen editing the documentation. The second patch implements git\nrev-list --objects-edge-aggressive, and the third patch ensures it's\nused for pushing to and fetching from shallow repos only.\n\nThe changes from v3 are to use --objects-edge-aggressive from the point\nit's introduced (this preserves bisectability) and to make higher-level\ncommands pass --shallow for any shallow pushing and fetching instead of\ntrying to have pack-objects determine it.\n\nThe original fix was suggested by Duy Nguyen.\n\nbrian m. carlson (3):\n Documentation: add missing article in rev-list-options.txt\n rev-list: add an option to mark fewer edges as uninteresting\n pack-objects: use --objects-edge-aggressive for shallow repos\n\n Documentation/git-pack-objects.txt | 7 ++++++-\n Documentation/git-rev-list.txt | 3 ++-\n Documentation/rev-list-options.txt | 7 ++++++-\n builtin/pack-objects.c | 7 ++++++-\n list-objects.c | 4 ++--\n revision.c | 6 ++++++\n revision.h | 1 +\n send-pack.c | 3 +++\n upload-pack.c | 4 +++-\n 9 files changed, 35 insertions(+), 7 deletions(-)\n\n-- \n2.2.1.209.g41e5f3a\n"},{"id":"254092","messageId":"1419462340-769147-2-git-send-email-sandals@crustytoothpaste.net","threadId":"38249","inReplyTo":"1419462340-769147-1-git-send-email-sandals@crustytoothpaste.net","subject":"[PATCH v4 1/3] Documentation: add missing article in rev-list-options.txt","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2014-12-24T23:05:38Z","receivedAt":"2014-12-24T23:05:38Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Add the missing article \"a\".\n\nSigned-off-by: brian m. carlson \n---\n Documentation/rev-list-options.txt | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex afccfdc..2277fcb 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -653,7 +653,7 @@ These options are mostly targeted for packing of Git repositories.\n --objects-edge::\n \tSimilar to `--objects`, but also print the IDs of excluded\n \tcommits prefixed with a ``-'' character. This is used by\n-\tlinkgit:git-pack-objects[1] to build ``thin'' pack, which records\n+\tlinkgit:git-pack-objects[1] to build a ``thin'' pack, which records\n \tobjects in deltified form based on objects contained in these\n \texcluded commits to reduce network traffic.\n \n-- \n2.2.1.209.g41e5f3a\n"},{"id":"254090","messageId":"1419462340-769147-3-git-send-email-sandals@crustytoothpaste.net","threadId":"38249","inReplyTo":"1419462340-769147-1-git-send-email-sandals@crustytoothpaste.net","subject":"[PATCH v4 2/3] rev-list: add an option to mark fewer edges as uninteresting","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2014-12-24T23:05:39Z","receivedAt":"2014-12-24T23:05:39Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"In commit fbd4a70 (list-objects: mark more commits as edges in\nmark_edges_uninteresting - 2013-08-16), we marked an increasing number\nof edges uninteresting. This change, and the subsequent change to make\nthis conditional on --objects-edge, are used by --thin to make much\nsmaller packs for shallow clones.\n\nUnfortunately, they cause a significant performance regression when\npushing non-shallow clones with lots of refs (23.322 seconds vs.\n4.785 seconds with 22400 refs). Add an option to git rev-list,\n--objects-edge-aggressive, that preserves this more aggressive behavior,\nwhile leaving --objects-edge to provide more performant behavior.\nPreserve the current behavior for the moment by using the aggressive\noption.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/git-rev-list.txt | 3 ++-\n Documentation/rev-list-options.txt | 4 ++++\n builtin/pack-objects.c | 2 +-\n list-objects.c | 4 ++--\n revision.c | 6 ++++++\n revision.h | 1 +\n 6 files changed, 16 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-rev-list.txt b/Documentation/git-rev-list.txt\nindex fd7f8b5..5b11922 100644\n--- a/Documentation/git-rev-list.txt\n+++ b/Documentation/git-rev-list.txt\n@@ -46,7 +46,8 @@ SYNOPSIS\n \t [ \\--extended-regexp | -E ]\n \t [ \\--fixed-strings | -F ]\n \t [ \\--date=(local|relative|default|iso|iso-strict|rfc|short) ]\n-\t [ [\\--objects | \\--objects-edge] [ \\--unpacked ] ]\n+\t [ [ \\--objects | \\--objects-edge | \\--objects-edge-aggressive ]\n+\t [ \\--unpacked ] ]\n \t [ \\--pretty | \\--header ]\n \t [ \\--bisect ]\n \t [ \\--bisect-vars ]\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 2277fcb..8cb6f92 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -657,6 +657,10 @@ These options are mostly targeted for packing of Git repositories.\n \tobjects in deltified form based on objects contained in these\n \texcluded commits to reduce network traffic.\n \n+--objects-edge-aggressive::\n+\tSimilar to `--objects-edge`, but it tries harder to find excluded\n+\tcommits at the cost of increased time.\n+\n --unpacked::\n \tOnly useful with `--objects`; print the object IDs that are not\n \tin packs.\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 3f9f5c7..f93a17c 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2711,7 +2711,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \targv_array_push(&rp, \"pack-objects\");\n \tif (thin) {\n \t\tuse_internal_rev_list = 1;\n-\t\targv_array_push(&rp, \"--objects-edge\");\n+\t\targv_array_push(&rp, \"--objects-edge-aggressive\");\n \t} else\n \t\targv_array_push(&rp, \"--objects\");\n \ndiff --git a/list-objects.c b/list-objects.c\nindex 2910bec..2a139b6 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -157,7 +157,7 @@ void mark_edges_uninteresting(struct rev_info *revs, show_edge_fn show_edge)\n \n \t\tif (commit->object.flags & UNINTERESTING) {\n \t\t\tmark_tree_uninteresting(commit->tree);\n-\t\t\tif (revs->edge_hint && !(commit->object.flags & SHOWN)) {\n+\t\t\tif (revs->edge_hint_aggressive && !(commit->object.flags & SHOWN)) {\n \t\t\t\tcommit->object.flags |= SHOWN;\n \t\t\t\tshow_edge(commit);\n \t\t\t}\n@@ -165,7 +165,7 @@ void mark_edges_uninteresting(struct rev_info *revs, show_edge_fn show_edge)\n \t\t}\n \t\tmark_edge_parents_uninteresting(commit, revs, show_edge);\n \t}\n-\tif (revs->edge_hint) {\n+\tif (revs->edge_hint_aggressive) {\n \t\tfor (i = 0; i < revs->cmdline.nr; i++) {\n \t\t\tstruct object *obj = revs->cmdline.rev[i].item;\n \t\t\tstruct commit *commit = (struct commit *)obj;\ndiff --git a/revision.c b/revision.c\nindex 75dda92..753dd2f 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1853,6 +1853,12 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg\n \t\trevs->tree_objects = 1;\n \t\trevs->blob_objects = 1;\n \t\trevs->edge_hint = 1;\n+\t} else if (!strcmp(arg, \"--objects-edge-aggressive\")) {\n+\t\trevs->tag_objects = 1;\n+\t\trevs->tree_objects = 1;\n+\t\trevs->blob_objects = 1;\n+\t\trevs->edge_hint = 1;\n+\t\trevs->edge_hint_aggressive = 1;\n \t} else if (!strcmp(arg, \"--verify-objects\")) {\n \t\trevs->tag_objects = 1;\n \t\trevs->tree_objects = 1;\ndiff --git a/revision.h b/revision.h\nindex 9cb5adc..033a244 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -93,6 +93,7 @@ struct rev_info {\n \t\t\tblob_objects:1,\n \t\t\tverify_objects:1,\n \t\t\tedge_hint:1,\n+\t\t\tedge_hint_aggressive:1,\n \t\t\tlimited:1,\n \t\t\tunpacked:1,\n \t\t\tboundary:2,\n-- \n2.2.1.209.g41e5f3a\n"},{"id":"254089","messageId":"1419462340-769147-4-git-send-email-sandals@crustytoothpaste.net","threadId":"38249","inReplyTo":"1419462340-769147-1-git-send-email-sandals@crustytoothpaste.net","subject":"[PATCH v4 3/3] pack-objects: use --objects-edge-aggressive for shallow repos","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2014-12-24T23:05:40Z","receivedAt":"2014-12-24T23:05:40Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"When fetching into or pushing from a shallow repository, we want to\naggressively mark edges as uninteresting, since this decreases the pack\nsize. However, aggressively marking edges can negatively affect\nperformance on large non-shallow repositories with lots of refs.\n\nTeach pack-objects a --shallow option to indicate that we're pushing\nfrom or fetching into a shallow repository. Use\n--objects-edge-aggressive only for shallow repositories and otherwise\nuse --objects-edge, which performs better in the general case. Update\nthe callers to pass the --shallow option when they are dealing with a\nshallow repository.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/git-pack-objects.txt | 7 ++++++-\n Documentation/rev-list-options.txt | 3 ++-\n builtin/pack-objects.c | 7 ++++++-\n send-pack.c | 3 +++\n upload-pack.c | 4 +++-\n 5 files changed, 20 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex d2d8f47..c2f76fb 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -13,7 +13,7 @@ SYNOPSIS\n \t[--no-reuse-delta] [--delta-base-offset] [--non-empty]\n \t[--local] [--incremental] [--window=] [--depth=]\n \t[--revs [--unpacked | --all]] [--stdout | base-name]\n-\t[--keep-true-parents] < object-list\n+\t[--shallow] [--keep-true-parents] < object-list\n \n \n DESCRIPTION\n@@ -190,6 +190,11 @@ required objects and is thus unusable by Git without making it\n self-contained. Use `git index-pack --fix-thin`\n (see linkgit:git-index-pack[1]) to restore the self-contained property.\n \n+--shallow::\n+\tOptimize a pack that will be provided to a client with a shallow\n+\trepository. This option, combined with \\--thin, can result in a\n+\tsmaller pack at the cost of speed.\n+\n --delta-base-offset::\n \tA packed archive can express the base object of a delta as\n \teither a 20-byte object name or as an offset in the\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 8cb6f92..2984f40 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -659,7 +659,8 @@ These options are mostly targeted for packing of Git repositories.\n \n --objects-edge-aggressive::\n \tSimilar to `--objects-edge`, but it tries harder to find excluded\n-\tcommits at the cost of increased time.\n+\tcommits at the cost of increased time. This is used instead of\n+\t`--objects-edge` to build ``thin'' packs for shallow repositories.\n \n --unpacked::\n \tOnly useful with `--objects`; print the object IDs that are not\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex f93a17c..d816587 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -2613,6 +2613,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n {\n \tint use_internal_rev_list = 0;\n \tint thin = 0;\n+\tint shallow = 0;\n \tint all_progress_implied = 0;\n \tstruct argv_array rp = ARGV_ARRAY_INIT;\n \tint rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0;\n@@ -2677,6 +2678,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t PARSE_OPT_OPTARG, option_parse_unpack_unreachable },\n \t\tOPT_BOOL(0, \"thin\", &thin,\n \t\t\t N_(\"create thin packs\")),\n+\t\tOPT_BOOL(0, \"shallow\", &shallow,\n+\t\t\t N_(\"create packs suitable for shallow fetches\")),\n \t\tOPT_BOOL(0, \"honor-pack-keep\", &ignore_packed_keep,\n \t\t\t N_(\"ignore packs that have companion .keep file\")),\n \t\tOPT_INTEGER(0, \"compression\", &pack_compression_level,\n@@ -2711,7 +2714,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \targv_array_push(&rp, \"pack-objects\");\n \tif (thin) {\n \t\tuse_internal_rev_list = 1;\n-\t\targv_array_push(&rp, \"--objects-edge-aggressive\");\n+\t\targv_array_push(&rp, shallow\n+\t\t\t\t? \"--objects-edge-aggressive\"\n+\t\t\t\t: \"--objects-edge\");\n \t} else\n \t\targv_array_push(&rp, \"--objects\");\n \ndiff --git a/send-pack.c b/send-pack.c\nindex 949cb61..25947d7 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -47,6 +47,7 @@ static int pack_objects(int fd, struct ref *refs, struct sha1_array *extra, stru\n \t\tNULL,\n \t\tNULL,\n \t\tNULL,\n+\t\tNULL,\n \t};\n \tstruct child_process po = CHILD_PROCESS_INIT;\n \tint i;\n@@ -60,6 +61,8 @@ static int pack_objects(int fd, struct ref *refs, struct sha1_array *extra, stru\n \t\targv[i++] = \"-q\";\n \tif (args->progress)\n \t\targv[i++] = \"--progress\";\n+\tif (is_repository_shallow())\n+\t\targv[i++] = \"--shallow\";\n \tpo.argv = argv;\n \tpo.in = -1;\n \tpo.out = args->stateless_rpc ? -1 : fd;\ndiff --git a/upload-pack.c b/upload-pack.c\nindex ac9ac15..b531a32 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -86,7 +86,7 @@ static void create_pack_file(void)\n \t\t\"corruption on the remote side.\";\n \tint buffered = -1;\n \tssize_t sz;\n-\tconst char *argv[12];\n+\tconst char *argv[13];\n \tint i, arg = 0;\n \tFILE *pipe_fd;\n \n@@ -100,6 +100,8 @@ static void create_pack_file(void)\n \t\targv[arg++] = \"--thin\";\n \n \targv[arg++] = \"--stdout\";\n+\tif (shallow_nr)\n+\t\targv[arg++] = \"--shallow\";\n \tif (!no_progress)\n \t\targv[arg++] = \"--progress\";\n \tif (use_ofs_delta)\n-- \n2.2.1.209.g41e5f3a\n"}]}