{"thread":{"id":"45654","subject":"[PATCH v8 1/3] read-cache: add strcmp_offset function","startedAt":"2017-04-10T21:14:18Z","lastAt":"2017-04-10T21:21:50Z","messageCount":5,"participants":["git@jeffhostetler.com","Jeff King"],"isPatch":true,"patchVersion":8,"patchTotal":3},"messages":[{"id":"316519","messageId":"20170410211403.25126-2-git@jeffhostetler.com","threadId":"45654","inReplyTo":"20170410211403.25126-1-git@jeffhostetler.com","subject":"[PATCH v8 1/3] read-cache: add strcmp_offset function","fromName":"","fromEmail":"git@jeffhostetler.com","sentAt":"2017-04-10T21:14:01Z","receivedAt":"2017-04-10T21:14:18Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nAdd strcmp_offset() function to also return the offset of the\nfirst change.\n\nAdd unit test and helper to verify.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Makefile                      |  1 +\n cache.h                       |  1 +\n read-cache.c                  | 20 ++++++++++++++++++++\n t/helper/.gitignore           |  1 +\n t/helper/test-strcmp-offset.c | 22 ++++++++++++++++++++++\n t/t0065-strcmp-offset.sh      | 21 +++++++++++++++++++++\n 6 files changed, 66 insertions(+)\n create mode 100644 t/helper/test-strcmp-offset.c\n create mode 100755 t/t0065-strcmp-offset.sh\n\ndiff --git a/Makefile b/Makefile\nindex 9ec6065..4c4c246 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -631,6 +631,7 @@ TEST_PROGRAMS_NEED_X += test-scrap-cache-tree\n TEST_PROGRAMS_NEED_X += test-sha1\n TEST_PROGRAMS_NEED_X += test-sha1-array\n TEST_PROGRAMS_NEED_X += test-sigchain\n+TEST_PROGRAMS_NEED_X += test-strcmp-offset\n TEST_PROGRAMS_NEED_X += test-string-list\n TEST_PROGRAMS_NEED_X += test-submodule-config\n TEST_PROGRAMS_NEED_X += test-subprocess\ndiff --git a/cache.h b/cache.h\nindex 80b6372..3c55047 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -574,6 +574,7 @@ extern int write_locked_index(struct index_state *, struct lock_file *lock, unsi\n extern int discard_index(struct index_state *);\n extern int unmerged_index(const struct index_state *);\n extern int verify_path(const char *path);\n+extern int strcmp_offset(const char *s1, const char *s2, size_t *first_change);\n extern int index_dir_exists(struct index_state *istate, const char *name, int namelen);\n extern void adjust_dirname_case(struct index_state *istate, char *name);\n extern struct cache_entry *index_file_exists(struct index_state *istate, const char *name, int namelen, int igncase);\ndiff --git a/read-cache.c b/read-cache.c\nindex 9054369..97f13a1 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -887,6 +887,26 @@ static int has_file_name(struct index_state *istate,\n \treturn retval;\n }\n \n+\n+/*\n+ * Like strcmp(), but also return the offset of the first change.\n+ * If strings are equal, return the length.\n+ */\n+int strcmp_offset(const char *s1, const char *s2, size_t *first_change)\n+{\n+\tsize_t k;\n+\n+\tif (!first_change)\n+\t\treturn strcmp(s1, s2);\n+\n+\tfor (k = 0; s1[k] == s2[k]; k++)\n+\t\tif (s1[k] == '\\0')\n+\t\t\tbreak;\n+\n+\t*first_change = k;\n+\treturn (unsigned char)s1[k] - (unsigned char)s2[k];\n+}\n+\n /*\n  * Do we have another file with a pathname that is a proper\n  * subset of the name we're trying to add?\ndiff --git a/t/helper/.gitignore b/t/helper/.gitignore\nindex d6e8b36..0a89531 100644\n--- a/t/helper/.gitignore\n+++ b/t/helper/.gitignore\n@@ -25,6 +25,7 @@\n /test-sha1\n /test-sha1-array\n /test-sigchain\n+/test-strcmp-offset\n /test-string-list\n /test-submodule-config\n /test-subprocess\ndiff --git a/t/helper/test-strcmp-offset.c b/t/helper/test-strcmp-offset.c\nnew file mode 100644\nindex 0000000..4a45a54\n--- /dev/null\n+++ b/t/helper/test-strcmp-offset.c\n@@ -0,0 +1,22 @@\n+#include \"cache.h\"\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tint result;\n+\tsize_t offset;\n+\n+\tif (!argv[1] || !argv[2])\n+\t\tdie(\"usage: %s <string1> <string2>\", argv[0]);\n+\n+\tresult = strcmp_offset(argv[1], argv[2], &offset);\n+\n+\t/*\n+\t * Because differnt CRTs behave differently, only rely on signs\n+\t * of the result values.\n+\t */\n+\tresult = (result < 0 ? -1 :\n+\t\t\t  result > 0 ? 1 :\n+\t\t\t  0);\n+\tprintf(\"%d %\"PRIuMAX\"\\n\", result, (uintmax_t)offset);\n+\treturn 0;\n+}\ndiff --git a/t/t0065-strcmp-offset.sh b/t/t0065-strcmp-offset.sh\nnew file mode 100755\nindex 0000000..7d6d214\n--- /dev/null\n+++ b/t/t0065-strcmp-offset.sh\n@@ -0,0 +1,21 @@\n+#!/bin/sh\n+\n+test_description='Test strcmp_offset functionality'\n+\n+. ./test-lib.sh\n+\n+while read s1 s2 expect\n+do\n+\ttest_expect_success \"strcmp_offset($s1, $s2)\" '\n+\t\techo \"$expect\" >expect &&\n+\t\ttest-strcmp-offset \"$s1\" \"$s2\" >actual &&\n+\t\ttest_cmp expect actual\n+\t'\n+done <<-EOF\n+abc abc 0 3\n+abc def -1 0\n+abc abz -1 2\n+abc abcdef -1 3\n+EOF\n+\n+test_done\n-- \n2.9.3\n\n"},{"id":"316520","messageId":"20170410211403.25126-4-git@jeffhostetler.com","threadId":"45654","inReplyTo":"20170410211403.25126-1-git@jeffhostetler.com","subject":"[PATCH v8 3/3] read-cache: speed up add_index_entry during checkout","fromName":"","fromEmail":"git@jeffhostetler.com","sentAt":"2017-04-10T21:14:03Z","receivedAt":"2017-04-10T21:14:20Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nTeach add_index_entry_with_check() and has_dir_name()\nto see if the path of the new item is greater than the\nlast path in the index array before attempting to search\nfor it.\n\nDuring checkout, merge_working_tree() populates the new\nindex in sorted order, so this change will save at least 2\nbinary lookups per file.  This preserves the original\nbehavior but simply checks the last element before starting\nthe search.\n\nThis helps performance on very large repositories.\n\nThis can be seen using p0006-read-tree-checkout.sh and the\nartificial repository created by t/perf/repos/many-files.sh\nwith parameters (5, 10, 9).   (1M files in index.)\n\n1..7\nTest                                                             HEAD^               HEAD\n------------------------------------------------------------------------------------------------------------\n0006.2: read-tree master ballast (1000001)                       4.01(2.71+1.28)     3.24(1.84+1.38) -19.2%\n0006.3: switch between master ballast (1000001)                  8.23(5.60+2.45)     6.73(4.20+2.35) -18.2%\n0006.4: switch between ballast ballast-1 (1000001)               13.36(8.60+4.35)    11.84(7.28+4.08) -11.4%\n0006.5: switch between aliases ballast ballast-alias (1000001)   13.43(8.68+4.27)    12.09(7.28+4.37) -10.0%\n0006.6: add forward sorted items (1000001) (200000)              2.42(1.76+0.42)     2.40(1.66+0.37) -0.8%\n0006.7: add reverse sorted items (1000001) (200000)              18.59(17.96+0.33)   18.70(18.07+0.42) +0.6%\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n read-cache.c | 46 ++++++++++++++++++++++++++++++++++++++++++++--\n 1 file changed, 44 insertions(+), 2 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 97f13a1..a8ef823 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -918,9 +918,24 @@ static int has_dir_name(struct index_state *istate,\n \tint stage = ce_stage(ce);\n \tconst char *name = ce->name;\n \tconst char *slash = name + ce_namelen(ce);\n+\tsize_t len_eq_last;\n+\tint cmp_last = 0;\n+\n+\tif (istate->cache_nr > 0) {\n+\t\t/*\n+\t\t * Compare the entry's full path with the last path in the index.\n+\t\t * If it sorts AFTER the last entry in the index and they have no\n+\t\t * common prefix, then there cannot be any F/D name conflicts.\n+\t\t */\n+\t\tcmp_last = strcmp_offset(name,\n+\t\t\tistate->cache[istate->cache_nr-1]->name,\n+\t\t\t&len_eq_last);\n+\t\tif (cmp_last > 0 && len_eq_last == 0)\n+\t\t\treturn retval;\n+\t}\n \n \tfor (;;) {\n-\t\tint len;\n+\t\tsize_t len;\n \n \t\tfor (;;) {\n \t\t\tif (*--slash == '/')\n@@ -930,6 +945,24 @@ static int has_dir_name(struct index_state *istate,\n \t\t}\n \t\tlen = slash - name;\n \n+\t\tif (cmp_last > 0) {\n+\t\t\t/*\n+\t\t\t * If this part of the directory prefix (including the trailing\n+\t\t\t * slash) already appears in the path of the last entry in the\n+\t\t\t * index, then we cannot also have a file with this prefix (or\n+\t\t\t * any parent directory prefix).\n+\t\t\t */\n+\t\t\tif (len+1 <= len_eq_last)\n+\t\t\t\treturn retval;\n+\t\t\t/*\n+\t\t\t * If this part of the directory prefix (excluding the trailing\n+\t\t\t * slash) is longer than the known equal portions, then this part\n+\t\t\t * of the prefix cannot collide with a file.  Go on to the parent.\n+\t\t\t */\n+\t\t\tif (len > len_eq_last)\n+\t\t\t\tcontinue;\n+\t\t}\n+\n \t\tpos = index_name_stage_pos(istate, name, len, stage);\n \t\tif (pos >= 0) {\n \t\t\t/*\n@@ -1021,7 +1054,16 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e\n \n \tif (!(option & ADD_CACHE_KEEP_CACHE_TREE))\n \t\tcache_tree_invalidate_path(istate, ce->name);\n-\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n+\n+\t/*\n+\t * If this entry's path sorts after the last entry in the index,\n+\t * we can avoid searching for it.\n+\t */\n+\tif (istate->cache_nr > 0 &&\n+\t\tstrcmp(ce->name, istate->cache[istate->cache_nr - 1]->name) > 0)\n+\t\tpos = -istate->cache_nr - 1;\n+\telse\n+\t\tpos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce));\n \n \t/* existing match? Just replace it. */\n \tif (pos >= 0) {\n-- \n2.9.3\n\n"},{"id":"316521","messageId":"20170410211403.25126-3-git@jeffhostetler.com","threadId":"45654","inReplyTo":"20170410211403.25126-1-git@jeffhostetler.com","subject":"[PATCH v8 2/3] p0006-read-tree-checkout: perf test to time read-tree","fromName":"","fromEmail":"git@jeffhostetler.com","sentAt":"2017-04-10T21:14:02Z","receivedAt":"2017-04-10T21:14:21Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nCreated t/perf/repos/many-files.sh to generate large, but\nartificial repositories.\n\nCreated t/perf/p0006-read-tree-checkout.sh to measure\nperformance on various read-tree, checkout, and update-index\noperations.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n t/perf/p0006-read-tree-checkout.sh |  90 ++++++++++++++++++++++++++++\n t/perf/repos/.gitignore            |   1 +\n t/perf/repos/many-files.sh         | 118 +++++++++++++++++++++++++++++++++++++\n 3 files changed, 209 insertions(+)\n create mode 100755 t/perf/p0006-read-tree-checkout.sh\n create mode 100644 t/perf/repos/.gitignore\n create mode 100755 t/perf/repos/many-files.sh\n\ndiff --git a/t/perf/p0006-read-tree-checkout.sh b/t/perf/p0006-read-tree-checkout.sh\nnew file mode 100755\nindex 0000000..69425ae\n--- /dev/null\n+++ b/t/perf/p0006-read-tree-checkout.sh\n@@ -0,0 +1,90 @@\n+#!/bin/sh\n+##\n+## This test measures the performance of various read-tree\n+## and checkout operations.  It is primarily interested in\n+## the algorithmic costs of index operations and recursive\n+## tree traversal -- and NOT disk I/O on thousands of files.\n+## Therefore, it uses sparse-checkout to avoid populating\n+## the ballast files.\n+##\n+## It expects the test repo to have certain characteristics.\n+## Branches:\n+## () master        := an arbitrary commit.\n+## () ballast       := an arbitrary commit with a large number\n+##                     of changes relative to \"master\".\n+## () ballast-alias := a branch pointing to the same commit\n+##                     as \"ballast\".\n+## () ballast-1     := a commit with a 1 file difference from\n+##                     \"ballast\".\n+##\n+## Ballast files in the repository should not appear in\n+## the sparse-checkout (if present).\n+##\n+## See \"t/perf/repos/many-files.sh\" to generate some\n+## synthetic data.\n+## \n+\n+test_description=\"Tests performance of read-tree\"\n+\n+. ./perf-lib.sh\n+\n+test_perf_default_repo\n+\n+test_expect_success 'setup' '\n+\techo \"sparse/*\" >>.git/info/sparse-checkout &&\n+\tgit config --local core.sparsecheckout 1 &&\n+\tgit checkout ballast\n+'\n+\n+nr_files=$(git ls-files | wc -l)\n+\n+test_perf \"read-tree master ballast ($nr_files)\" '\n+\tgit read-tree -m master ballast -n\n+'\n+\n+## Alternate between a commit with and without the ballast.\n+test_perf \"switch between master ballast ($nr_files)\" '\n+\tgit checkout master &&\n+\tgit checkout ballast\n+'\n+\n+## Alternate between 2 commits with the ballast and 1 change.\n+test_perf \"switch between ballast ballast-1 ($nr_files)\" '\n+\tgit checkout ballast-1 &&\n+\tgit checkout ballast\n+'\n+\n+## Alternate between 2 aliases for the same commit.\n+test_perf \"switch between aliases ballast ballast-alias ($nr_files)\" '\n+\tgit checkout ballast-alias &&\n+\tgit checkout ballast\n+'\n+\n+export nr_random=200000\n+\n+## Add random items in sorted order. These will effectively\n+## get appended because \"zzzz/\" sorts after anything already\n+## present in index.  Then delete them in reverse order, so\n+## that the last is removed each time.\n+test_perf \"add forward sorted items ($nr_files) ($nr_random)\" '\n+\tawk \"BEGIN { for (f = 0; f <= $nr_random; f++) printf \\\"r_%06d\\n\\\", f }\" |\n+\tsed \"s|^|100644 $EMPTY_BLOB\tzzzz/|\" |\n+\tgit update-index --index-info &&\n+\tawk \"BEGIN { for (f = $nr_random; f >= 0; f--) printf \\\"r_%06d\\n\\\", f }\" |\n+\tsed \"s|^|000000 $EMPTY_BLOB\tzzzz/|\" |\n+\tgit update-index --index-info\n+'\n+\n+## Add random items in reverse order. These will be inserted\n+## NEAR the end of the index, but not at the end.  Then delete\n+## them in the opposite order, so that we remove from the middle.\n+test_perf \"add reverse sorted items ($nr_files) ($nr_random)\" '\n+\tawk \"BEGIN { for (f = $nr_random; f >= 0; f--) printf \\\"r_%06d\\n\\\", f }\" |\n+\tsed \"s|^|100644 $EMPTY_BLOB\tzzzz/|\" |\n+\tgit update-index --index-info &&\n+\tawk \"BEGIN { for (f = 0; f <= $nr_random; f++) printf \\\"r_%06d\\n\\\", f }\" |\n+\tsed \"s|^|000000 $EMPTY_BLOB\tzzzz/|\" |\n+\tgit update-index --index-info\n+'\n+\n+test_done\ndiff --git a/t/perf/repos/.gitignore b/t/perf/repos/.gitignore\nnew file mode 100644\nindex 0000000..72e3dc3\n--- /dev/null\n+++ b/t/perf/repos/.gitignore\n@@ -0,0 +1 @@\n+gen-*/\ndiff --git a/t/perf/repos/many-files.sh b/t/perf/repos/many-files.sh\nnew file mode 100755\nindex 0000000..a4c44b3\n--- /dev/null\n+++ b/t/perf/repos/many-files.sh\n@@ -0,0 +1,118 @@\n+#!/bin/sh\n+## Generate test data repository \"many-files.git\" using the given parameters.\n+## Usage: [-r repo] [-d depth] [-w width] [-f files]\n+##\n+## -r repo: path to the new repo to be generated\n+## -d depth: the depth of sub-directories\n+## -w width: the number of sub-directories at each level\n+## -f files: the number of files created in each directory\n+##\n+## Note that all files will have the same SHA-1 and each\n+## directory at a level will have the same SHA-1, so we\n+## will potentially have a large index, but not a large\n+## ODB.\n+##\n+## Ballast will be created under \"ballast/\".  Sparse-checkout\n+## will be enabled so that they will not be populated by\n+## default.\n+\n+EMPTY_BLOB=e69de29bb2d1d6434b8b29ae775ad8c2e48c5391\n+\n+set -e\n+\n+## (5, 10, 9) will create 999,999 ballast files.\n+## (4, 10, 9) will create  99,999 ballast files.\n+depth=5\n+width=10\n+files=9\n+\n+while test \"$#\" -ne 0\n+do\n+    case \"$1\" in\n+\t-r)\n+\t    shift;\n+\t    test \"$#\" -ne 0 || { echo 'error: -r requires an argument' >&2; exit 1; }\n+\t    repo=$1;\n+\t    shift ;;\n+\t-d)\n+\t    shift;\n+\t    test \"$#\" -ne 0 || { echo 'error: -d requires an argument' >&2; exit 1; }\n+\t    depth=$1;\n+\t    shift ;;\n+\t-w)\n+\t    shift;\n+\t    test \"$#\" -ne 0 || { echo 'error: -w requires an argument' >&2; exit 1; }\n+\t    width=$1;\n+\t    shift ;;\n+\t-f)\n+\t    shift;\n+\t    test \"$#\" -ne 0 || { echo 'error: -f requires an argument' >&2; exit 1; }\n+\t    files=$1;\n+\t    shift ;;\n+\t*)\n+\t    echo \"error: unknown option '$1'\" >&2; exit 1 ;;\n+\tesac\n+done\n+\n+## Inflate the index with thousands of empty files.\n+## usage: dir depth width files\n+fill_index() {\n+\tawk -v arg_dir=$1 -v arg_depth=$2 -v arg_width=$3 -v arg_files=$4 '\n+\t\tfunction make_paths(dir, depth, width, files, f, w) {\n+\t\t\tfor (f = 1; f <= files; f++) {\n+\t\t\t\tprint dir \"/file\" f\n+\t\t\t}\n+\t\t\tif (depth > 0) {\n+\t\t\t\tfor (w = 1; w <= width; w++) {\n+\t\t\t\t\tmake_paths(dir \"/dir\" w, depth - 1, width, files)\n+\t\t\t\t}\n+\t\t\t}\n+\t\t}\n+\t\tEND { make_paths(arg_dir, arg_depth, arg_width, arg_files) }\n+\t\t' </dev/null |\n+\tsed \"s/^/100644 $EMPTY_BLOB\t/\" |\n+\tgit update-index --index-info\n+\treturn 0\n+}\n+\n+[ -z \"$repo\" ] && repo=gen-many-files-$depth.$width.$files.git\n+\n+mkdir $repo\n+cd $repo\n+git init .\n+\n+## Create initial commit just to define branch.\n+mkdir sparse\n+touch sparse/file.txt\n+echo \"$depth $width $files\" >sparse/repo.params\n+git add sparse\n+git commit -q -m params\n+\n+## Turn on sparse-checkout so that we don't have to populate\n+## the ballast when switching branches.  Use reset --hard to\n+## quickly checkout the new HEAD with minimum actual files.\n+echo 'sparse/*' >>.git/info/sparse-checkout\n+git config --local core.sparsecheckout 1\n+git reset --hard\n+\n+## Inflate the index with thousands of empty files and commit.\n+git checkout -b ballast\n+fill_index \"ballast\" $depth $width $files\n+git commit -q -m \"ballast\"\n+\n+## Create an alias for that commit.\n+git branch \"ballast-alias\"\n+\n+nr_files=$(git ls-files | wc -l)\n+\n+## Modify 1 file and commit.\n+git checkout -b \"ballast-1\"\n+echo x >sparse/file.txt\n+git add sparse\n+git commit -q -m \"ballast plus 1\"\n+\n+## Checkout master to put repo in canonical state.\n+git checkout master\n+\n+echo \"Repository \"$repo\" ($depth, $width, $files) created.  Ballast $nr_files.\"\n+exit 0\n-- \n2.9.3\n\n"},{"id":"316522","messageId":"20170410211403.25126-1-git@jeffhostetler.com","threadId":"45654","inReplyTo":null,"subject":"[PATCH v8 0/3] read-cache: speed up add_index_entry","fromName":"","fromEmail":"git@jeffhostetler.com","sentAt":"2017-04-10T21:14:00Z","receivedAt":"2017-04-10T21:14:24Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nVersion 8 addresses the following:\n  () Refactors strcmp_offset() test helper to just compare the\n     values and return results to stdout.\n  () Updates t0065 to have the actual test logic.\n  () Splits the synthetic test repo generation out of the perf\n     test into a stand-alone script.\n  () Simplifies the p0006 (was p0004) perf test to use an\n     existing repository (whether artificial or real).\n  () Adds additional tests for add/delete (which should be\n     driven by the arg-list rather than an iteration on the\n     index.\n  () Updated the read-cache commit message with new perf data.\n\nJeff Hostetler (3):\n  read-cache: add strcmp_offset function\n  p0006-read-tree-checkout: perf test to time read-tree\n  read-cache: speed up add_index_entry during checkout\n\n Makefile                           |   1 +\n cache.h                            |   1 +\n read-cache.c                       |  66 ++++++++++++++++++++-\n t/helper/.gitignore                |   1 +\n t/helper/test-strcmp-offset.c      |  22 +++++++\n t/perf/p0006-read-tree-checkout.sh |  90 ++++++++++++++++++++++++++++\n t/perf/repos/.gitignore            |   1 +\n t/perf/repos/many-files.sh         | 118 +++++++++++++++++++++++++++++++++++++\n t/t0065-strcmp-offset.sh           |  21 +++++++\n 9 files changed, 319 insertions(+), 2 deletions(-)\n create mode 100644 t/helper/test-strcmp-offset.c\n create mode 100755 t/perf/p0006-read-tree-checkout.sh\n create mode 100644 t/perf/repos/.gitignore\n create mode 100755 t/perf/repos/many-files.sh\n create mode 100755 t/t0065-strcmp-offset.sh\n\n-- \n2.9.3\n\n"},{"id":"316524","messageId":"20170410212142.zc3tsnirjcuoewmw@sigill.intra.peff.net","threadId":"45654","inReplyTo":"20170410211403.25126-3-git@jeffhostetler.com","subject":"Re: [PATCH v8 2/3] p0006-read-tree-checkout: perf test to time read-tree","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-04-10T21:21:43Z","receivedAt":"2017-04-10T21:21:50Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Apr 10, 2017 at 09:14:02PM +0000, git@jeffhostetler.com wrote:\n\n> From: Jeff Hostetler <jeffhost@microsoft.com>\n> \n> Created t/perf/repos/many-files.sh to generate large, but\n> artificial repositories.\n\nI think this is a good direction. In the long run we might want some\nkind of magic to pull from the \"library\" of repos when running perf\ntests, but it's not a big deal to run the script manually and point\nGIT_PERF_REPO at the result.\n\nAs a bonus, this should be faster when running perf tests, since we can\nreuse the built repo when testing each version of Git.\n\n> +## This test measures the performance of various read-tree\n> +## and checkout operations.  It is primarily interested in\n> +## the algorithmic costs of index operations and recursive\n> +## tree traversal -- and NOT disk I/O on thousands of files.\n> +## Therefore, it uses sparse-checkout to avoid populating\n> +## the ballast files.\n> +##\n> +## It expects the test repo to have certain characteristics.\n> +## Branches:\n> +## () master        := an arbitrary commit.\n> +## () ballast       := an arbitrary commit with a large number\n> +##                     of changes relative to \"master\".\n> +## () ballast-alias := a branch pointing to the same commit\n> +##                     as \"ballast\".\n> +## () ballast-1     := a commit with a 1 file difference from\n> +##                     \"ballast\".\n\nI'm OK with leaving these requirements on the repo in the name of\nsimplicity, though it does make it harder to perf-test against a regular\nrepo.\n\nI wonder if we could make reasonable guesses, like:\n\n  master => HEAD\n  ballast => $(git rev-list HEAD | tail -n 1)\n  ballast-alias => git branch $ballast\n  ballast-1 => HEAD^\n\nThat would approximate your conditions in a real-world repository, and\nit should be easy to make your synthetic one fit the bill exactly.\n\nI don't know if you'd want to turn on sparse checkout manually or not\nwhen testing a real-world repo.\n\n-Peff\n"}]}