{"thread":{"id":"45240","subject":"[PATCH v4 01/22] config: mark an error message up for translation","startedAt":"2017-02-27T18:02:08Z","lastAt":"2017-03-01T21:45:33Z","messageCount":26,"participants":["Christian Couder","Junio C Hamano"],"isPatch":true,"patchVersion":4,"patchTotal":22},"messages":[{"id":"312730","messageId":"20170227180019.18666-2-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 01/22] config: mark an error message up for translation","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T17:59:58Z","receivedAt":"2017-02-27T18:02:08Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n config.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex c6b874a7bf..2ac1aa19b0 100644\n--- a/config.c\n+++ b/config.c\n@@ -1728,8 +1728,8 @@ int git_config_get_untracked_cache(void)\n \t\tif (!strcasecmp(v, \"keep\"))\n \t\t\treturn -1;\n \n-\t\terror(\"unknown core.untrackedCache value '%s'; \"\n-\t\t      \"using 'keep' default value\", v);\n+\t\terror(_(\"unknown core.untrackedCache value '%s'; \"\n+\t\t\t\"using 'keep' default value\"), v);\n \t\treturn -1;\n \t}\n \n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312731","messageId":"20170227180019.18666-5-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 04/22] split-index: add {add,remove}_split_index() functions","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T18:00:01Z","receivedAt":"2017-02-27T18:02:10Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Also use the functions in cmd_update_index() in\nbuiltin/update-index.c.\n\nThese functions will be used in a following commit to tweak\nour use of the split-index feature depending on the setting\nof a configuration variable.\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n builtin/update-index.c | 18 ++++++------------\n split-index.c          | 22 ++++++++++++++++++++++\n split-index.h          |  2 ++\n 3 files changed, 30 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex d530e89368..24fdadfa4b 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1099,18 +1099,12 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (split_index > 0) {\n-\t\tinit_split_index(&the_index);\n-\t\tthe_index.cache_changed |= SPLIT_INDEX_ORDERED;\n-\t} else if (!split_index && the_index.split_index) {\n-\t\t/*\n-\t\t * can't discard_split_index(&the_index); because that\n-\t\t * will destroy split_index->base->cache[], which may\n-\t\t * be shared with the_index.cache[]. So yeah we're\n-\t\t * leaking a bit here.\n-\t\t */\n-\t\tthe_index.split_index = NULL;\n-\t\tthe_index.cache_changed |= SOMETHING_CHANGED;\n-\t}\n+\t\tif (the_index.split_index)\n+\t\t\tthe_index.cache_changed |= SPLIT_INDEX_ORDERED;\n+\t\telse\n+\t\t\tadd_split_index(&the_index);\n+\t} else if (!split_index)\n+\t\tremove_split_index(&the_index);\n \n \tswitch (untracked_cache) {\n \tcase UC_UNSPECIFIED:\ndiff --git a/split-index.c b/split-index.c\nindex 615f4cac05..f519e60f87 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -317,3 +317,25 @@ void replace_index_entry_in_base(struct index_state *istate,\n \t\tistate->split_index->base->cache[new->index - 1] = new;\n \t}\n }\n+\n+void add_split_index(struct index_state *istate)\n+{\n+\tif (!istate->split_index) {\n+\t\tinit_split_index(istate);\n+\t\tistate->cache_changed |= SPLIT_INDEX_ORDERED;\n+\t}\n+}\n+\n+void remove_split_index(struct index_state *istate)\n+{\n+\tif (istate->split_index) {\n+\t\t/*\n+\t\t * can't discard_split_index(&the_index); because that\n+\t\t * will destroy split_index->base->cache[], which may\n+\t\t * be shared with the_index.cache[]. So yeah we're\n+\t\t * leaking a bit here.\n+\t\t */\n+\t\tistate->split_index = NULL;\n+\t\tistate->cache_changed |= SOMETHING_CHANGED;\n+\t}\n+}\ndiff --git a/split-index.h b/split-index.h\nindex c1324f521a..df91c1bda8 100644\n--- a/split-index.h\n+++ b/split-index.h\n@@ -31,5 +31,7 @@ void merge_base_index(struct index_state *istate);\n void prepare_to_write_split_index(struct index_state *istate);\n void finish_writing_split_index(struct index_state *istate);\n void discard_split_index(struct index_state *istate);\n+void add_split_index(struct index_state *istate);\n+void remove_split_index(struct index_state *istate);\n \n #endif\n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312732","messageId":"20170227180019.18666-4-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 03/22] config: add git_config_get_split_index()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T18:00:00Z","receivedAt":"2017-02-27T18:02:12Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This new function will be used in a following commit to know\nif we want to use the split index feature or not.\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n cache.h  |  1 +\n config.c | 10 ++++++++++\n 2 files changed, 11 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex 61fc86e6d7..014aa7ea11 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1882,6 +1882,7 @@ extern int git_config_get_bool_or_int(const char *key, int *is_bool, int *dest);\n extern int git_config_get_maybe_bool(const char *key, int *dest);\n extern int git_config_get_pathname(const char *key, const char **dest);\n extern int git_config_get_untracked_cache(void);\n+extern int git_config_get_split_index(void);\n \n /*\n  * This is a hack for test programs like test-dump-untracked-cache to\ndiff --git a/config.c b/config.c\nindex 2ac1aa19b0..2a97696be7 100644\n--- a/config.c\n+++ b/config.c\n@@ -1736,6 +1736,16 @@ int git_config_get_untracked_cache(void)\n \treturn -1; /* default value */\n }\n \n+int git_config_get_split_index(void)\n+{\n+\tint val;\n+\n+\tif (!git_config_get_maybe_bool(\"core.splitindex\", &val))\n+\t\treturn val;\n+\n+\treturn -1; /* default value */\n+}\n+\n NORETURN\n void git_die_config_linenr(const char *key, const char *filename, int linenr)\n {\n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312733","messageId":"20170227180019.18666-11-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 10/22] config: add git_config_get_max_percent_split_change()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T18:00:07Z","receivedAt":"2017-02-27T18:02:16Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This new function will be used in a following commit to get the\nvalue of the \"splitIndex.maxPercentChange\" config variable.\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n cache.h  |  1 +\n config.c | 15 +++++++++++++++\n 2 files changed, 16 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex 014aa7ea11..955e80913e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1883,6 +1883,7 @@ extern int git_config_get_maybe_bool(const char *key, int *dest);\n extern int git_config_get_pathname(const char *key, const char **dest);\n extern int git_config_get_untracked_cache(void);\n extern int git_config_get_split_index(void);\n+extern int git_config_get_max_percent_split_change(void);\n \n /*\n  * This is a hack for test programs like test-dump-untracked-cache to\ndiff --git a/config.c b/config.c\nindex 2a97696be7..35b6f02960 100644\n--- a/config.c\n+++ b/config.c\n@@ -1746,6 +1746,21 @@ int git_config_get_split_index(void)\n \treturn -1; /* default value */\n }\n \n+int git_config_get_max_percent_split_change(void)\n+{\n+\tint val = -1;\n+\n+\tif (!git_config_get_int(\"splitindex.maxpercentchange\", &val)) {\n+\t\tif (0 <= val && val <= 100)\n+\t\t\treturn val;\n+\n+\t\treturn error(_(\"splitIndex.maxPercentChange value '%d' \"\n+\t\t\t       \"should be between 0 and 100\"), val);\n+\t}\n+\n+\treturn -1; /* default value */\n+}\n+\n NORETURN\n void git_die_config_linenr(const char *key, const char *filename, int linenr)\n {\n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312734","messageId":"20170227180019.18666-12-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 11/22] read-cache: regenerate shared index if necessary","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T18:00:08Z","receivedAt":"2017-02-27T18:02:19Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"When writing a new split-index and there is a big number of cache\nentries in the split-index compared to the shared index, it is a\ngood idea to regenerate the shared index.\n\nBy default when the ratio reaches 20%, we will push back all\nthe entries from the split-index into a new shared index file\ninstead of just creating a new split-index file.\n\nThe threshold can be configured using the\n\"splitIndex.maxPercentChange\" config variable.\n\nWe need to adjust the existing tests in t1700 by setting\n\"splitIndex.maxPercentChange\" to 100 at the beginning of t1700,\nas the existing tests are assuming that the shared index is\nregenerated only when `git update-index --split-index` is used.\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n read-cache.c           | 32 ++++++++++++++++++++++++++++++++\n t/t1700-split-index.sh |  1 +\n 2 files changed, 33 insertions(+)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 99bc274b8d..aeb413a508 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2212,6 +2212,36 @@ static int write_shared_index(struct index_state *istate,\n \treturn ret;\n }\n \n+static const int default_max_percent_split_change = 20;\n+\n+static int too_many_not_shared_entries(struct index_state *istate)\n+{\n+\tint i, not_shared = 0;\n+\tint max_split = git_config_get_max_percent_split_change();\n+\n+\tswitch (max_split) {\n+\tcase -1:\n+\t\t/* not or badly configured: use the default value */\n+\t\tmax_split = default_max_percent_split_change;\n+\t\tbreak;\n+\tcase 0:\n+\t\treturn 1; /* 0% means always write a new shared index */\n+\tcase 100:\n+\t\treturn 0; /* 100% means never write a new shared index */\n+\tdefault:\n+\t\tbreak; /* just use the configured value */\n+\t}\n+\n+\t/* Count not shared entries */\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = istate->cache[i];\n+\t\tif (!ce->index)\n+\t\t\tnot_shared++;\n+\t}\n+\n+\treturn (int64_t)istate->cache_nr * max_split < (int64_t)not_shared * 100;\n+}\n+\n int write_locked_index(struct index_state *istate, struct lock_file *lock,\n \t\t       unsigned flags)\n {\n@@ -2229,6 +2259,8 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,\n \t\tif ((v & 15) < 6)\n \t\t\tistate->cache_changed |= SPLIT_INDEX_ORDERED;\n \t}\n+\tif (too_many_not_shared_entries(istate))\n+\t\tistate->cache_changed |= SPLIT_INDEX_ORDERED;\n \tif (istate->cache_changed & SPLIT_INDEX_ORDERED) {\n \t\tint ret = write_shared_index(istate, lock, flags);\n \t\tif (ret)\ndiff --git a/t/t1700-split-index.sh b/t/t1700-split-index.sh\nindex 1659986d8d..df19b812fd 100755\n--- a/t/t1700-split-index.sh\n+++ b/t/t1700-split-index.sh\n@@ -8,6 +8,7 @@ test_description='split index mode tests'\n sane_unset GIT_TEST_SPLIT_INDEX\n \n test_expect_success 'enable split index' '\n+\tgit config splitIndex.maxPercentChange 100 &&\n \tgit update-index --split-index &&\n \ttest-dump-split-index .git/index >actual &&\n \tindexversion=$(test-index-version <.git/index) &&\n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312735","messageId":"20170227180019.18666-6-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 05/22] read-cache: add and then use tweak_split_index()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T18:00:02Z","receivedAt":"2017-02-27T18:02:22Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This will make us use the split-index feature or not depending\non the value of the \"core.splitIndex\" config variable.\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n read-cache.c | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 9054369dd0..99bc274b8d 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1558,10 +1558,27 @@ static void tweak_untracked_cache(struct index_state *istate)\n \t}\n }\n \n+static void tweak_split_index(struct index_state *istate)\n+{\n+\tswitch (git_config_get_split_index()) {\n+\tcase -1: /* unset: do nothing */\n+\t\tbreak;\n+\tcase 0: /* false */\n+\t\tremove_split_index(istate);\n+\t\tbreak;\n+\tcase 1: /* true */\n+\t\tadd_split_index(istate);\n+\t\tbreak;\n+\tdefault: /* unknown value: do nothing */\n+\t\tbreak;\n+\t}\n+}\n+\n static void post_read_index_from(struct index_state *istate)\n {\n \tcheck_ce_order(istate);\n \ttweak_untracked_cache(istate);\n+\ttweak_split_index(istate);\n }\n \n /* remember to discard_cache() before reading a different cache! */\n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312736","messageId":"20170227180019.18666-7-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 06/22] update-index: warn in case of split-index incoherency","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T18:00:03Z","receivedAt":"2017-02-27T18:02:23Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"When users are using `git update-index --(no-)split-index`, they\nmay expect the split-index feature to be used or not according to\nthe option they just used, but this might not be the case if the\nnew \"core.splitIndex\" config variable has been set. In this case\nlet's warn about what will happen and why.\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n builtin/update-index.c | 11 ++++++++++-\n 1 file changed, 10 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 24fdadfa4b..d74d72cc7f 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1099,12 +1099,21 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (split_index > 0) {\n+\t\tif (git_config_get_split_index() == 0)\n+\t\t\twarning(_(\"core.splitIndex is set to false; \"\n+\t\t\t\t  \"remove or change it, if you really want to \"\n+\t\t\t\t  \"enable split index\"));\n \t\tif (the_index.split_index)\n \t\t\tthe_index.cache_changed |= SPLIT_INDEX_ORDERED;\n \t\telse\n \t\t\tadd_split_index(&the_index);\n-\t} else if (!split_index)\n+\t} else if (!split_index) {\n+\t\tif (git_config_get_split_index() == 1)\n+\t\t\twarning(_(\"core.splitIndex is set to true; \"\n+\t\t\t\t  \"remove or change it, if you really want to \"\n+\t\t\t\t  \"disable split index\"));\n \t\tremove_split_index(&the_index);\n+\t}\n \n \tswitch (untracked_cache) {\n \tcase UC_UNSPECIFIED:\n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312737","messageId":"20170227180019.18666-8-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 07/22] t1700: add tests for core.splitIndex","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T18:00:04Z","receivedAt":"2017-02-27T18:02:25Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n t/t1700-split-index.sh | 37 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 37 insertions(+)\n\ndiff --git a/t/t1700-split-index.sh b/t/t1700-split-index.sh\nindex cb68b8dc1e..1659986d8d 100755\n--- a/t/t1700-split-index.sh\n+++ b/t/t1700-split-index.sh\n@@ -200,4 +200,41 @@ test_expect_success 'unify index, two files remain' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'set core.splitIndex config variable to true' '\n+\tgit config core.splitIndex true &&\n+\t: >three &&\n+\tgit update-index --add three &&\n+\tgit ls-files --stage >ls-files.actual &&\n+\tcat >ls-files.expect <<-EOF &&\n+\t100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0\tone\n+\t100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0\tthree\n+\t100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0\ttwo\n+\tEOF\n+\ttest_cmp ls-files.expect ls-files.actual &&\n+\tBASE=$(test-dump-split-index .git/index | grep \"^base\") &&\n+\ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n+\tcat >expect <<-EOF &&\n+\t$BASE\n+\treplacements:\n+\tdeletions:\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'set core.splitIndex config variable to false' '\n+\tgit config core.splitIndex false &&\n+\tgit update-index --force-remove three &&\n+\tgit ls-files --stage >ls-files.actual &&\n+\tcat >ls-files.expect <<-EOF &&\n+\t100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0\tone\n+\t100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0\ttwo\n+\tEOF\n+\ttest_cmp ls-files.expect ls-files.actual &&\n+\ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n+\tcat >expect <<-EOF &&\n+\tnot a split index\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312738","messageId":"20170227180019.18666-9-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 08/22] Documentation/config: add information for core.splitIndex","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T18:00:05Z","receivedAt":"2017-02-27T18:02:28Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n Documentation/config.txt | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 015346c417..61a863adeb 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -334,6 +334,10 @@ core.trustctime::\n \tcrawlers and some backup systems).\n \tSee linkgit:git-update-index[1]. True by default.\n \n+core.splitIndex::\n+\tIf true, the split-index feature of the index will be used.\n+\tSee linkgit:git-update-index[1]. False by default.\n+\n core.untrackedCache::\n \tDetermines what to do about the untracked cache feature of the\n \tindex. It will be kept, if this variable is unset or set to\n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312739","messageId":"20170227180019.18666-10-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 09/22] Documentation/git-update-index: talk about core.splitIndex config var","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T18:00:06Z","receivedAt":"2017-02-27T18:02:31Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n Documentation/git-update-index.txt | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex 7386c93162..e091b2a409 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -171,6 +171,12 @@ may not support it yet.\n \tgiven again, all changes in $GIT_DIR/index are pushed back to\n \tthe shared index file. This mode is designed for very large\n \tindexes that take a significant amount of time to read or write.\n++\n+These options take effect whatever the value of the `core.splitIndex`\n+configuration variable (see linkgit:git-config[1]). But a warning is\n+emitted when the change goes against the configured value, as the\n+configured value will take effect next time the index is read and this\n+will remove the intended effect of the option.\n \n --untracked-cache::\n --no-untracked-cache::\n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312740","messageId":"20170227180019.18666-1-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":null,"subject":"[PATCH v4 00/22] Add configuration options for split-index","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T17:59:57Z","receivedAt":"2017-02-27T18:02:34Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Goal\n~~~~\n\nWe want to make it possible to use the split-index feature\nautomatically by just setting a new \"core.splitIndex\" configuration\nvariable to true.\n\nThis can be valuable as split-index can help significantly speed up\n`git rebase` especially along with the work to libify `git apply`\nthat has been merged to master\n(see https://github.com/git/git/commit/81358dc238372793b1590efa149cc1581d1fbd98)\nand is now in v2.11.\n\nDesign\n~~~~~~\n\nThe design is similar as the previous work that introduced\n\"core.untrackedCache\". \n\nThe new \"core.splitIndex\" configuration option can be either true,\nfalse or undefined which is the default.\n\nWhen it is true, the split index is created, if it does not already\nexists, when the index is read. When it is false, the split index is\nremoved if it exists, when the index is read. Otherwise it is left as\nis.\n\nAlong with this new configuration variable, the two following options\nare also introduced:\n\n    - splitIndex.maxPercentChange\n\n    This is to avoid having too many changes accumulating in the split\n    index while in split index mode. The git-update-index\n    documentation says:\n\n\tIf split-index mode is already enabled and `--split-index` is\n\tgiven again, all changes in $GIT_DIR/index are pushed back to\n\tthe shared index file.\n\n    but it is probably better to not expect the user to think about it\n    and to have a mechanism that pushes back all changes to the shared\n    index file automatically when some threshold is reached.\n\n    The default threshold is when the number of entries in the split\n    index file reaches 20% of the number of entries in the shared\n    index file. The new \"splitIndex.maxPercentChange\" config option\n    lets people tweak this value.\n\n    - splitIndex.sharedIndexExpire\n\n    To make sure that old sharedindex files are eventually removed\n    when a new one has been created, we \"touch\" the shared index file\n    every time a split index file using the shared index file is\n    either created or read from. Then we can delete shared indexes\n    with an mtime older than one week (by default), when we create a\n    new shared index file. The new \"splitIndex.sharedIndexExpire\"\n    config option lets people tweak this grace period.\n\n    This idea was suggested by Duy in:\n\n    https://public-inbox.org/git/CACsJy8BqMFASHf5kJgUh+bd7XG98CafNydE964VJyPXz-emEvA@mail.gmail.com/\n\n    and after some experiments, I agree that it is much simpler than\n    what I thought could be done during our discussion.\n\n    Junio also thinks that we have to do \"time-based GC\" in:\n \n    https://public-inbox.org/git/xmqqeg33ccjj.fsf@gitster.mtv.corp.google.com/\n\nNote that this patch series doesn't address a leak when removing a\nsplit-index, but Duy wrote that he has a patch to fix this leak:\n\nhttps://public-inbox.org/git/CACsJy8AisF2ZVs7JdnVFp5wdskkbVQQQ=DBq5UzE1MOsCfBMtQ@mail.gmail.com/\n\nHighlevel view of the patches in the series\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nExcept for patch 1/22 and 1/22, there are 3 big steps, one for each\nnew configuration variable introduced.\n\nThere only small differences between this patch series and the v3\npatch series sent a few months ago.\n\n    - Patch 1/22 marks a message for translation. It is not new and\n      can be applied separately.\n\n    - Patch 2/22 improves the existing indentation style of t1700 by\n      using different here document style. It is a new preparatory\n      patch suggested by Junio.\n\nStep 1 is:\n\n    - Patches 3/22 to 6/22 introduce the functions that are reading\n      the \"core.splitIndex\" configuration variable and tweaking the\n      split index depending on its value.\n\n    - Patch 7/22 adds a few tests for the new feature.\n\n    - Patches 8/22 and 9/22 add some documentation for the new\n      feature.\n\n    There is no change since v3 in this step.\n\nStep 2 is:\n\n    - Patches 10/22 and 11/22 introduce the functions that are reading\n      the \"splitIndex.maxPercentChange\" configuration variable and\n      regenerating a new shared index file depending on its value.\n\n      Patch 11/12 has a few changes suggested by Junio since v3, see\n      https://public-inbox.org/git/CAP8UFD3_1EN=0EsD12Cew1MuW8yhtPAZw0M_g3wmvKFk-uGXxw@mail.gmail.com/\n\n    - Patch 12/22 adds a few tests for the new feature.\n\n    - Patch 13/22 add some documentation for the new feature. The\n      added documentation has been reworded a little bit since v3 as\n      suggested by Junio.\n\nStep 3 is:\n\n    - Patches 14/22 to 17/22 introduce the functions that are reading\n      the \"splitIndex.sharedIndexExpire\" configuration variable and\n      expiring old shared index files depending on its value.\n\n      Patch 15/22 has a few changes suggested by Ramsay and Junio in\n      http://public-inbox.org/git/a1a44640-ff6c-2294-72ac-46322eff8505@ramsayjones.plus.com/\n      http://public-inbox.org/git/xmqqbmunq6mg.fsf@gitster.mtv.corp.google.com/\n\n      The main change is that the new freshen_shared_index() will now\n      warn if the split-index file has been written but the shared\n      index file could't be freshened.\n\n      Another change is that in 17/22 the new\n      \"splitIndex.sharedIndexExpire\" config variable now defaults to\n      \"2.weeks.ago\" instead of \"one.week.ago\" in v3.\n\n    - Patch 18/22 adds a few tests for the new feature. It is changed\n      a bit to account for the change in 17/22.\n\n      Also some flakyness in one of the tests has been fixed by using\n      a 5 second, instead of 1 second, delay, thanks to Ramsay and\n      Peff, see\n      http://public-inbox.org/git/818851a6-c3ef-618e-4146-518fbe6bd837@ramsayjones.plus.com/\n\n    - Patches 19/22 and 20/22 were new patches in v3. They update the\n      mtime of the shared index file when a split index based on the\n      shared index file is read from. 19/22 is a refactoring to make\n      the actual change in 20/22 easier.\n\n      Patch 20/22 has been changed a little bit since v3 to take into\n      account changes in 17/22.\n\n    - Patches 21/22 and 22/22 add some documentation for the new\n      feature. Patch 21/22 has been changed to avoid using \"mtime\" as\n      suggested by Junio.\n\nLinks\n~~~~~\n\nThis patch series is also available here:\n\n  https://github.com/chriscool/git/commits/config-split-index\n\nThe previous versions were:\n\n  RFC: https://github.com/chriscool/git/commits/config-split-index7\n  v1:  https://github.com/chriscool/git/commits/config-split-index72\n  v2:  https://github.com/chriscool/git/commits/config-split-index99\n  v3:  https://github.com/chriscool/git/commits/config-split-index102\n\nOn the mailing list the related patch series and discussions were:\n\n  RFC: https://public-inbox.org/git/20160711172254.13439-1-chriscool@tuxfamily.org/\n  v1:  https://public-inbox.org/git/20161023092648.12086-1-chriscool@tuxfamily.org/\n  v2:  https://public-inbox.org/git/20161217145547.11748-1-chriscool@tuxfamily.org/\n  v3:  https://public-inbox.org/git/20161226102222.17150-1-chriscool@tuxfamily.org/\n\nChristian Couder (22):\n  config: mark an error message up for translation\n  t1700: change here document style\n  config: add git_config_get_split_index()\n  split-index: add {add,remove}_split_index() functions\n  read-cache: add and then use tweak_split_index()\n  update-index: warn in case of split-index incoherency\n  t1700: add tests for core.splitIndex\n  Documentation/config: add information for core.splitIndex\n  Documentation/git-update-index: talk about core.splitIndex config var\n  config: add git_config_get_max_percent_split_change()\n  read-cache: regenerate shared index if necessary\n  t1700: add tests for splitIndex.maxPercentChange\n  Documentation/config: add splitIndex.maxPercentChange\n  sha1_file: make check_and_freshen_file() non static\n  read-cache: touch shared index files when used\n  config: add git_config_get_expiry() from gc.c\n  read-cache: unlink old sharedindex files\n  t1700: test shared index file expiration\n  read-cache: refactor read_index_from()\n  read-cache: use freshen_shared_index() in read_index_from()\n  Documentation/config: add splitIndex.sharedIndexExpire\n  Documentation/git-update-index: explain splitIndex.*\n\n Documentation/config.txt           |  29 ++++\n Documentation/git-update-index.txt |  43 ++++-\n builtin/gc.c                       |  15 +-\n builtin/update-index.c             |  25 +--\n cache.h                            |   8 +\n config.c                           |  42 ++++-\n read-cache.c                       | 157 ++++++++++++++++--\n sha1_file.c                        |   2 +-\n split-index.c                      |  22 +++\n split-index.h                      |   2 +\n t/t1700-split-index.sh             | 324 +++++++++++++++++++++++++++----------\n 11 files changed, 539 insertions(+), 130 deletions(-)\n\n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312741","messageId":"20170227180019.18666-3-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 02/22] t1700: change here document style","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T17:59:59Z","receivedAt":"2017-02-27T18:02:36Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This improves test indentation by getting rid of the outdated\nhere document style.\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n t/t1700-split-index.sh | 170 ++++++++++++++++++++++++-------------------------\n 1 file changed, 85 insertions(+), 85 deletions(-)\n\ndiff --git a/t/t1700-split-index.sh b/t/t1700-split-index.sh\nindex 292a0720fc..cb68b8dc1e 100755\n--- a/t/t1700-split-index.sh\n+++ b/t/t1700-split-index.sh\n@@ -19,12 +19,12 @@ test_expect_success 'enable split index' '\n \t\town=8299b0bcd1ac364e5f1d7768efb62fa2da79a339\n \t\tbase=39d890139ee5356c7ef572216cebcd27aa41f9df\n \tfi &&\n-\tcat >expect <<EOF &&\n-own $own\n-base $base\n-replacements:\n-deletions:\n-EOF\n+\tcat >expect <<-EOF &&\n+\town $own\n+\tbase $base\n+\treplacements:\n+\tdeletions:\n+\tEOF\n \ttest_cmp expect actual\n '\n \n@@ -32,51 +32,51 @@ test_expect_success 'add one file' '\n \t: >one &&\n \tgit update-index --add one &&\n \tgit ls-files --stage >ls-files.actual &&\n-\tcat >ls-files.expect <<EOF &&\n-100644 $EMPTY_BLOB 0\tone\n-EOF\n+\tcat >ls-files.expect <<-EOF &&\n+\t100644 $EMPTY_BLOB 0\tone\n+\tEOF\n \ttest_cmp ls-files.expect ls-files.actual &&\n \n \ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n-\tcat >expect <<EOF &&\n-base $base\n-100644 $EMPTY_BLOB 0\tone\n-replacements:\n-deletions:\n-EOF\n+\tcat >expect <<-EOF &&\n+\tbase $base\n+\t100644 $EMPTY_BLOB 0\tone\n+\treplacements:\n+\tdeletions:\n+\tEOF\n \ttest_cmp expect actual\n '\n \n test_expect_success 'disable split index' '\n \tgit update-index --no-split-index &&\n \tgit ls-files --stage >ls-files.actual &&\n-\tcat >ls-files.expect <<EOF &&\n-100644 $EMPTY_BLOB 0\tone\n-EOF\n+\tcat >ls-files.expect <<-EOF &&\n+\t100644 $EMPTY_BLOB 0\tone\n+\tEOF\n \ttest_cmp ls-files.expect ls-files.actual &&\n \n \tBASE=$(test-dump-split-index .git/index | grep \"^own\" | sed \"s/own/base/\") &&\n \ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n-\tcat >expect <<EOF &&\n-not a split index\n-EOF\n+\tcat >expect <<-EOF &&\n+\tnot a split index\n+\tEOF\n \ttest_cmp expect actual\n '\n \n test_expect_success 'enable split index again, \"one\" now belongs to base index\"' '\n \tgit update-index --split-index &&\n \tgit ls-files --stage >ls-files.actual &&\n-\tcat >ls-files.expect <<EOF &&\n-100644 $EMPTY_BLOB 0\tone\n-EOF\n+\tcat >ls-files.expect <<-EOF &&\n+\t100644 $EMPTY_BLOB 0\tone\n+\tEOF\n \ttest_cmp ls-files.expect ls-files.actual &&\n \n \ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n-\tcat >expect <<EOF &&\n-$BASE\n-replacements:\n-deletions:\n-EOF\n+\tcat >expect <<-EOF &&\n+\t$BASE\n+\treplacements:\n+\tdeletions:\n+\tEOF\n \ttest_cmp expect actual\n '\n \n@@ -84,18 +84,18 @@ test_expect_success 'modify original file, base index untouched' '\n \techo modified >one &&\n \tgit update-index one &&\n \tgit ls-files --stage >ls-files.actual &&\n-\tcat >ls-files.expect <<EOF &&\n-100644 2e0996000b7e9019eabcad29391bf0f5c7702f0b 0\tone\n-EOF\n+\tcat >ls-files.expect <<-EOF &&\n+\t100644 2e0996000b7e9019eabcad29391bf0f5c7702f0b 0\tone\n+\tEOF\n \ttest_cmp ls-files.expect ls-files.actual &&\n \n \ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n-\tq_to_tab >expect <<EOF &&\n-$BASE\n-100644 2e0996000b7e9019eabcad29391bf0f5c7702f0b 0Q\n-replacements: 0\n-deletions:\n-EOF\n+\tq_to_tab >expect <<-EOF &&\n+\t$BASE\n+\t100644 2e0996000b7e9019eabcad29391bf0f5c7702f0b 0Q\n+\treplacements: 0\n+\tdeletions:\n+\tEOF\n \ttest_cmp expect actual\n '\n \n@@ -103,54 +103,54 @@ test_expect_success 'add another file, which stays index' '\n \t: >two &&\n \tgit update-index --add two &&\n \tgit ls-files --stage >ls-files.actual &&\n-\tcat >ls-files.expect <<EOF &&\n-100644 2e0996000b7e9019eabcad29391bf0f5c7702f0b 0\tone\n-100644 $EMPTY_BLOB 0\ttwo\n-EOF\n+\tcat >ls-files.expect <<-EOF &&\n+\t100644 2e0996000b7e9019eabcad29391bf0f5c7702f0b 0\tone\n+\t100644 $EMPTY_BLOB 0\ttwo\n+\tEOF\n \ttest_cmp ls-files.expect ls-files.actual &&\n \n \ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n-\tq_to_tab >expect <<EOF &&\n-$BASE\n-100644 2e0996000b7e9019eabcad29391bf0f5c7702f0b 0Q\n-100644 $EMPTY_BLOB 0\ttwo\n-replacements: 0\n-deletions:\n-EOF\n+\tq_to_tab >expect <<-EOF &&\n+\t$BASE\n+\t100644 2e0996000b7e9019eabcad29391bf0f5c7702f0b 0Q\n+\t100644 $EMPTY_BLOB 0\ttwo\n+\treplacements: 0\n+\tdeletions:\n+\tEOF\n \ttest_cmp expect actual\n '\n \n test_expect_success 'remove file not in base index' '\n \tgit update-index --force-remove two &&\n \tgit ls-files --stage >ls-files.actual &&\n-\tcat >ls-files.expect <<EOF &&\n-100644 2e0996000b7e9019eabcad29391bf0f5c7702f0b 0\tone\n-EOF\n+\tcat >ls-files.expect <<-EOF &&\n+\t100644 2e0996000b7e9019eabcad29391bf0f5c7702f0b 0\tone\n+\tEOF\n \ttest_cmp ls-files.expect ls-files.actual &&\n \n \ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n-\tq_to_tab >expect <<EOF &&\n-$BASE\n-100644 2e0996000b7e9019eabcad29391bf0f5c7702f0b 0Q\n-replacements: 0\n-deletions:\n-EOF\n+\tq_to_tab >expect <<-EOF &&\n+\t$BASE\n+\t100644 2e0996000b7e9019eabcad29391bf0f5c7702f0b 0Q\n+\treplacements: 0\n+\tdeletions:\n+\tEOF\n \ttest_cmp expect actual\n '\n \n test_expect_success 'remove file in base index' '\n \tgit update-index --force-remove one &&\n \tgit ls-files --stage >ls-files.actual &&\n-\tcat >ls-files.expect <<EOF &&\n-EOF\n+\tcat >ls-files.expect <<-EOF &&\n+\tEOF\n \ttest_cmp ls-files.expect ls-files.actual &&\n \n \ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n-\tcat >expect <<EOF &&\n-$BASE\n-replacements:\n-deletions: 0\n-EOF\n+\tcat >expect <<-EOF &&\n+\t$BASE\n+\treplacements:\n+\tdeletions: 0\n+\tEOF\n \ttest_cmp expect actual\n '\n \n@@ -158,18 +158,18 @@ test_expect_success 'add original file back' '\n \t: >one &&\n \tgit update-index --add one &&\n \tgit ls-files --stage >ls-files.actual &&\n-\tcat >ls-files.expect <<EOF &&\n-100644 $EMPTY_BLOB 0\tone\n-EOF\n+\tcat >ls-files.expect <<-EOF &&\n+\t100644 $EMPTY_BLOB 0\tone\n+\tEOF\n \ttest_cmp ls-files.expect ls-files.actual &&\n \n \ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n-\tcat >expect <<EOF &&\n-$BASE\n-100644 $EMPTY_BLOB 0\tone\n-replacements:\n-deletions: 0\n-EOF\n+\tcat >expect <<-EOF &&\n+\t$BASE\n+\t100644 $EMPTY_BLOB 0\tone\n+\treplacements:\n+\tdeletions: 0\n+\tEOF\n \ttest_cmp expect actual\n '\n \n@@ -177,26 +177,26 @@ test_expect_success 'add new file' '\n \t: >two &&\n \tgit update-index --add two &&\n \tgit ls-files --stage >actual &&\n-\tcat >expect <<EOF &&\n-100644 $EMPTY_BLOB 0\tone\n-100644 $EMPTY_BLOB 0\ttwo\n-EOF\n+\tcat >expect <<-EOF &&\n+\t100644 $EMPTY_BLOB 0\tone\n+\t100644 $EMPTY_BLOB 0\ttwo\n+\tEOF\n \ttest_cmp expect actual\n '\n \n test_expect_success 'unify index, two files remain' '\n \tgit update-index --no-split-index &&\n \tgit ls-files --stage >ls-files.actual &&\n-\tcat >ls-files.expect <<EOF &&\n-100644 $EMPTY_BLOB 0\tone\n-100644 $EMPTY_BLOB 0\ttwo\n-EOF\n+\tcat >ls-files.expect <<-EOF &&\n+\t100644 $EMPTY_BLOB 0\tone\n+\t100644 $EMPTY_BLOB 0\ttwo\n+\tEOF\n \ttest_cmp ls-files.expect ls-files.actual &&\n \n \ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n-\tcat >expect <<EOF &&\n-not a split index\n-EOF\n+\tcat >expect <<-EOF &&\n+\tnot a split index\n+\tEOF\n \ttest_cmp expect actual\n '\n \n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312742","messageId":"20170227180019.18666-13-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 12/22] t1700: add tests for splitIndex.maxPercentChange","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T18:00:09Z","receivedAt":"2017-02-27T18:08:49Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n t/t1700-split-index.sh | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 72 insertions(+)\n\ndiff --git a/t/t1700-split-index.sh b/t/t1700-split-index.sh\nindex df19b812fd..21f43903f8 100755\n--- a/t/t1700-split-index.sh\n+++ b/t/t1700-split-index.sh\n@@ -238,4 +238,76 @@ test_expect_success 'set core.splitIndex config variable to false' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'set core.splitIndex config variable to true' '\n+\tgit config core.splitIndex true &&\n+\t: >three &&\n+\tgit update-index --add three &&\n+\tBASE=$(test-dump-split-index .git/index | grep \"^base\") &&\n+\ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n+\tcat >expect <<-EOF &&\n+\t$BASE\n+\treplacements:\n+\tdeletions:\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t: >four &&\n+\tgit update-index --add four &&\n+\ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n+\tcat >expect <<-EOF &&\n+\t$BASE\n+\t100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0\tfour\n+\treplacements:\n+\tdeletions:\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'check behavior with splitIndex.maxPercentChange unset' '\n+\tgit config --unset splitIndex.maxPercentChange &&\n+\t: >five &&\n+\tgit update-index --add five &&\n+\tBASE=$(test-dump-split-index .git/index | grep \"^base\") &&\n+\ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n+\tcat >expect <<-EOF &&\n+\t$BASE\n+\treplacements:\n+\tdeletions:\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t: >six &&\n+\tgit update-index --add six &&\n+\ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n+\tcat >expect <<-EOF &&\n+\t$BASE\n+\t100644 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 0\tsix\n+\treplacements:\n+\tdeletions:\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'check splitIndex.maxPercentChange set to 0' '\n+\tgit config splitIndex.maxPercentChange 0 &&\n+\t: >seven &&\n+\tgit update-index --add seven &&\n+\tBASE=$(test-dump-split-index .git/index | grep \"^base\") &&\n+\ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n+\tcat >expect <<-EOF &&\n+\t$BASE\n+\treplacements:\n+\tdeletions:\n+\tEOF\n+\ttest_cmp expect actual &&\n+\t: >eight &&\n+\tgit update-index --add eight &&\n+\tBASE=$(test-dump-split-index .git/index | grep \"^base\") &&\n+\ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n+\tcat >expect <<-EOF &&\n+\t$BASE\n+\treplacements:\n+\tdeletions:\n+\tEOF\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312743","messageId":"20170227180019.18666-15-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 14/22] sha1_file: make check_and_freshen_file() non static","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T18:00:11Z","receivedAt":"2017-02-27T18:08:51Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This function will be used in a commit soon, so let's make\nit available globally.\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n cache.h     | 3 +++\n sha1_file.c | 2 +-\n 2 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex 955e80913e..6b25b50aab 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1229,6 +1229,9 @@ extern int has_pack_index(const unsigned char *sha1);\n \n extern void assert_sha1_type(const unsigned char *sha1, enum object_type expect);\n \n+/* Helper to check and \"touch\" a file */\n+extern int check_and_freshen_file(const char *fn, int freshen);\n+\n extern const signed char hexval_table[256];\n static inline unsigned int hexval(unsigned char c)\n {\ndiff --git a/sha1_file.c b/sha1_file.c\nindex ec957db5e1..192073ea95 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -593,7 +593,7 @@ static int freshen_file(const char *fn)\n  * either does not exist on disk, or has a stale mtime and may be subject to\n  * pruning).\n  */\n-static int check_and_freshen_file(const char *fn, int freshen)\n+int check_and_freshen_file(const char *fn, int freshen)\n {\n \tif (access(fn, F_OK))\n \t\treturn 0;\n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312744","messageId":"20170227180019.18666-17-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 16/22] config: add git_config_get_expiry() from gc.c","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T18:00:13Z","receivedAt":"2017-02-27T18:11:43Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This function will be used in a following commit to get the expiration\ntime of the shared index files from the config, and it is generic\nenough to be put in \"config.c\".\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n builtin/gc.c | 15 ++-------------\n cache.h      |  3 +++\n config.c     | 13 +++++++++++++\n 3 files changed, 18 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 331f219260..66dff6a8af 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -62,17 +62,6 @@ static void report_pack_garbage(unsigned seen_bits, const char *path)\n \t\tstring_list_append(&pack_garbage, path);\n }\n \n-static void git_config_date_string(const char *key, const char **output)\n-{\n-\tif (git_config_get_string_const(key, output))\n-\t\treturn;\n-\tif (strcmp(*output, \"now\")) {\n-\t\tunsigned long now = approxidate(\"now\");\n-\t\tif (approxidate(*output) >= now)\n-\t\t\tgit_die_config(key, _(\"Invalid %s: '%s'\"), key, *output);\n-\t}\n-}\n-\n static void process_log_file(void)\n {\n \tstruct stat st;\n@@ -111,8 +100,8 @@ static void gc_config(void)\n \tgit_config_get_int(\"gc.auto\", &gc_auto_threshold);\n \tgit_config_get_int(\"gc.autopacklimit\", &gc_auto_pack_limit);\n \tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n-\tgit_config_date_string(\"gc.pruneexpire\", &prune_expire);\n-\tgit_config_date_string(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n+\tgit_config_get_expiry(\"gc.pruneexpire\", &prune_expire);\n+\tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n \tgit_config(git_default_config, NULL);\n }\n \ndiff --git a/cache.h b/cache.h\nindex 6b25b50aab..8994e7d373 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1888,6 +1888,9 @@ extern int git_config_get_untracked_cache(void);\n extern int git_config_get_split_index(void);\n extern int git_config_get_max_percent_split_change(void);\n \n+/* This dies if the configured or default date is in the future */\n+extern int git_config_get_expiry(const char *key, const char **output);\n+\n /*\n  * This is a hack for test programs like test-dump-untracked-cache to\n  * ensure that they do not modify the untracked cache when reading it.\ndiff --git a/config.c b/config.c\nindex 35b6f02960..f20d7d88f7 100644\n--- a/config.c\n+++ b/config.c\n@@ -1712,6 +1712,19 @@ int git_config_get_pathname(const char *key, const char **dest)\n \treturn ret;\n }\n \n+int git_config_get_expiry(const char *key, const char **output)\n+{\n+\tint ret = git_config_get_string_const(key, output);\n+\tif (ret)\n+\t\treturn ret;\n+\tif (strcmp(*output, \"now\")) {\n+\t\tunsigned long now = approxidate(\"now\");\n+\t\tif (approxidate(*output) >= now)\n+\t\t\tgit_die_config(key, _(\"Invalid %s: '%s'\"), key, *output);\n+\t}\n+\treturn ret;\n+}\n+\n int git_config_get_untracked_cache(void)\n {\n \tint val = -1;\n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312745","messageId":"20170227180019.18666-20-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 19/22] read-cache: refactor read_index_from()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T18:00:16Z","receivedAt":"2017-02-27T18:11:44Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"It looks better and is simpler to review when we don't compute\nthe same things many times in the function.\n\nIt will also help make the following commit simpler.\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n read-cache.c | 14 ++++++++------\n 1 file changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 45fc831010..3ea20701a3 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1691,6 +1691,8 @@ int read_index_from(struct index_state *istate, const char *path)\n {\n \tstruct split_index *split_index;\n \tint ret;\n+\tchar *base_sha1_hex;\n+\tconst char *base_path;\n \n \t/* istate->initialized covers both .git/index and .git/sharedindex.xxx */\n \tif (istate->initialized)\n@@ -1708,15 +1710,15 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tdiscard_index(split_index->base);\n \telse\n \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n-\tret = do_read_index(split_index->base,\n-\t\t\t    git_path(\"sharedindex.%s\",\n-\t\t\t\t     sha1_to_hex(split_index->base_sha1)), 1);\n+\n+\tbase_sha1_hex = sha1_to_hex(split_index->base_sha1);\n+\tbase_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n+\tret = do_read_index(split_index->base, base_path, 1);\n \tif (hashcmp(split_index->base_sha1, split_index->base->sha1))\n \t\tdie(\"broken index, expect %s in %s, got %s\",\n-\t\t    sha1_to_hex(split_index->base_sha1),\n-\t\t    git_path(\"sharedindex.%s\",\n-\t\t\t     sha1_to_hex(split_index->base_sha1)),\n+\t\t    base_sha1_hex, base_path,\n \t\t    sha1_to_hex(split_index->base->sha1));\n+\n \tmerge_base_index(istate);\n \tpost_read_index_from(istate);\n \treturn ret;\n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312746","messageId":"20170227180019.18666-23-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 22/22] Documentation/git-update-index: explain splitIndex.*","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T18:00:19Z","receivedAt":"2017-02-27T18:11:46Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n Documentation/config.txt           |  2 +-\n Documentation/git-update-index.txt | 37 +++++++++++++++++++++++++++++--------\n 2 files changed, 30 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 0e9982c5e3..2ccb053d92 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2853,7 +2853,7 @@ splitIndex.sharedIndexExpire::\n \tThe default value is \"2.weeks.ago\".\n \tNote that a shared index file is considered modified (for the\n \tpurpose of expiration) each time a new split-index file is\n-\tcreated based on it.\n+\teither created based on it or read from it.\n \tSee linkgit:git-update-index[1].\n \n status.relativePaths::\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex e091b2a409..1579abf3c3 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -163,14 +163,10 @@ may not support it yet.\n \n --split-index::\n --no-split-index::\n-\tEnable or disable split index mode. If enabled, the index is\n-\tsplit into two files, $GIT_DIR/index and $GIT_DIR/sharedindex.<SHA-1>.\n-\tChanges are accumulated in $GIT_DIR/index while the shared\n-\tindex file contains all index entries stays unchanged. If\n-\tsplit-index mode is already enabled and `--split-index` is\n-\tgiven again, all changes in $GIT_DIR/index are pushed back to\n-\tthe shared index file. This mode is designed for very large\n-\tindexes that take a significant amount of time to read or write.\n+\tEnable or disable split index mode. If split-index mode is\n+\talready enabled and `--split-index` is given again, all\n+\tchanges in $GIT_DIR/index are pushed back to the shared index\n+\tfile.\n +\n These options take effect whatever the value of the `core.splitIndex`\n configuration variable (see linkgit:git-config[1]). But a warning is\n@@ -394,6 +390,31 @@ Although this bit looks similar to assume-unchanged bit, its goal is\n different from assume-unchanged bit's. Skip-worktree also takes\n precedence over assume-unchanged bit when both are set.\n \n+Split index\n+-----------\n+\n+This mode is designed for repositories with very large indexes, and\n+aims at reducing the time it takes to repeatedly write these indexes.\n+\n+In this mode, the index is split into two files, $GIT_DIR/index and\n+$GIT_DIR/sharedindex.<SHA-1>. Changes are accumulated in\n+$GIT_DIR/index, the split index, while the shared index file contains\n+all index entries and stays unchanged.\n+\n+All changes in the split index are pushed back to the shared index\n+file when the number of entries in the split index reaches a level\n+specified by the splitIndex.maxPercentChange config variable (see\n+linkgit:git-config[1]).\n+\n+Each time a new shared index file is created, the old shared index\n+files are deleted if their modification time is older than what is\n+specified by the splitIndex.sharedIndexExpire config variable (see\n+linkgit:git-config[1]).\n+\n+To avoid deleting a shared index file that is still used, its\n+modification time is updated to the current time everytime a new split\n+index based on the shared index file is either created or read from.\n+\n Untracked cache\n ---------------\n \n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312747","messageId":"20170227180019.18666-21-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 20/22] read-cache: use freshen_shared_index() in read_index_from()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T18:00:17Z","receivedAt":"2017-02-27T18:11:47Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This way a share index file will not be garbage collected if\nwe still read from an index it is based from.\n\nAs we need to read the current index before creating a new\none, the tests have to be adjusted, so that we don't expect\nan old shared index file to be deleted right away when we\ncreate a new one.\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n read-cache.c           |  1 +\n t/t1700-split-index.sh | 14 +++++++-------\n 2 files changed, 8 insertions(+), 7 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 3ea20701a3..d1152161f6 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1719,6 +1719,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\t    base_sha1_hex, base_path,\n \t\t    sha1_to_hex(split_index->base->sha1));\n \n+\tfreshen_shared_index(base_sha1_hex, 0);\n \tmerge_base_index(istate);\n \tpost_read_index_from(istate);\n \treturn ret;\ndiff --git a/t/t1700-split-index.sh b/t/t1700-split-index.sh\nindex 480d3a8dc3..ea1aeacff0 100755\n--- a/t/t1700-split-index.sh\n+++ b/t/t1700-split-index.sh\n@@ -313,17 +313,17 @@ test_expect_success 'check splitIndex.maxPercentChange set to 0' '\n test_expect_success 'shared index files expire after 2 weeks by default' '\n \t: >ten &&\n \tgit update-index --add ten &&\n-\ttest $(ls .git/sharedindex.* | wc -l) -gt 1 &&\n+\ttest $(ls .git/sharedindex.* | wc -l) -gt 2 &&\n \tjust_under_2_weeks_ago=$((5-14*86400)) &&\n \ttest-chmtime =$just_under_2_weeks_ago .git/sharedindex.* &&\n \t: >eleven &&\n \tgit update-index --add eleven &&\n-\ttest $(ls .git/sharedindex.* | wc -l) -gt 1 &&\n+\ttest $(ls .git/sharedindex.* | wc -l) -gt 2 &&\n \tjust_over_2_weeks_ago=$((-1-14*86400)) &&\n \ttest-chmtime =$just_over_2_weeks_ago .git/sharedindex.* &&\n \t: >twelve &&\n \tgit update-index --add twelve &&\n-\ttest $(ls .git/sharedindex.* | wc -l) = 1\n+\ttest $(ls .git/sharedindex.* | wc -l) -le 2\n '\n \n test_expect_success 'check splitIndex.sharedIndexExpire set to 16 days' '\n@@ -331,12 +331,12 @@ test_expect_success 'check splitIndex.sharedIndexExpire set to 16 days' '\n \ttest-chmtime =$just_over_2_weeks_ago .git/sharedindex.* &&\n \t: >thirteen &&\n \tgit update-index --add thirteen &&\n-\ttest $(ls .git/sharedindex.* | wc -l) -gt 1 &&\n+\ttest $(ls .git/sharedindex.* | wc -l) -gt 2 &&\n \tjust_over_16_days_ago=$((-1-16*86400)) &&\n \ttest-chmtime =$just_over_16_days_ago .git/sharedindex.* &&\n \t: >fourteen &&\n \tgit update-index --add fourteen &&\n-\ttest $(ls .git/sharedindex.* | wc -l) = 1\n+\ttest $(ls .git/sharedindex.* | wc -l) -le 2\n '\n \n test_expect_success 'check splitIndex.sharedIndexExpire set to \"never\" and \"now\"' '\n@@ -345,13 +345,13 @@ test_expect_success 'check splitIndex.sharedIndexExpire set to \"never\" and \"now\"\n \ttest-chmtime =$just_10_years_ago .git/sharedindex.* &&\n \t: >fifteen &&\n \tgit update-index --add fifteen &&\n-\ttest $(ls .git/sharedindex.* | wc -l) -gt 1 &&\n+\ttest $(ls .git/sharedindex.* | wc -l) -gt 2 &&\n \tgit config splitIndex.sharedIndexExpire now &&\n \tjust_1_second_ago=-1 &&\n \ttest-chmtime =$just_1_second_ago .git/sharedindex.* &&\n \t: >sixteen &&\n \tgit update-index --add sixteen &&\n-\ttest $(ls .git/sharedindex.* | wc -l) = 1\n+\ttest $(ls .git/sharedindex.* | wc -l) -le 2\n '\n \n test_done\n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312750","messageId":"20170227180019.18666-19-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 18/22] t1700: test shared index file expiration","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T18:00:15Z","receivedAt":"2017-02-27T18:36:24Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n t/t1700-split-index.sh | 44 ++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 44 insertions(+)\n\ndiff --git a/t/t1700-split-index.sh b/t/t1700-split-index.sh\nindex 21f43903f8..480d3a8dc3 100755\n--- a/t/t1700-split-index.sh\n+++ b/t/t1700-split-index.sh\n@@ -310,4 +310,48 @@ test_expect_success 'check splitIndex.maxPercentChange set to 0' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'shared index files expire after 2 weeks by default' '\n+\t: >ten &&\n+\tgit update-index --add ten &&\n+\ttest $(ls .git/sharedindex.* | wc -l) -gt 1 &&\n+\tjust_under_2_weeks_ago=$((5-14*86400)) &&\n+\ttest-chmtime =$just_under_2_weeks_ago .git/sharedindex.* &&\n+\t: >eleven &&\n+\tgit update-index --add eleven &&\n+\ttest $(ls .git/sharedindex.* | wc -l) -gt 1 &&\n+\tjust_over_2_weeks_ago=$((-1-14*86400)) &&\n+\ttest-chmtime =$just_over_2_weeks_ago .git/sharedindex.* &&\n+\t: >twelve &&\n+\tgit update-index --add twelve &&\n+\ttest $(ls .git/sharedindex.* | wc -l) = 1\n+'\n+\n+test_expect_success 'check splitIndex.sharedIndexExpire set to 16 days' '\n+\tgit config splitIndex.sharedIndexExpire \"16.days.ago\" &&\n+\ttest-chmtime =$just_over_2_weeks_ago .git/sharedindex.* &&\n+\t: >thirteen &&\n+\tgit update-index --add thirteen &&\n+\ttest $(ls .git/sharedindex.* | wc -l) -gt 1 &&\n+\tjust_over_16_days_ago=$((-1-16*86400)) &&\n+\ttest-chmtime =$just_over_16_days_ago .git/sharedindex.* &&\n+\t: >fourteen &&\n+\tgit update-index --add fourteen &&\n+\ttest $(ls .git/sharedindex.* | wc -l) = 1\n+'\n+\n+test_expect_success 'check splitIndex.sharedIndexExpire set to \"never\" and \"now\"' '\n+\tgit config splitIndex.sharedIndexExpire never &&\n+\tjust_10_years_ago=$((-365*10*86400)) &&\n+\ttest-chmtime =$just_10_years_ago .git/sharedindex.* &&\n+\t: >fifteen &&\n+\tgit update-index --add fifteen &&\n+\ttest $(ls .git/sharedindex.* | wc -l) -gt 1 &&\n+\tgit config splitIndex.sharedIndexExpire now &&\n+\tjust_1_second_ago=-1 &&\n+\ttest-chmtime =$just_1_second_ago .git/sharedindex.* &&\n+\t: >sixteen &&\n+\tgit update-index --add sixteen &&\n+\ttest $(ls .git/sharedindex.* | wc -l) = 1\n+'\n+\n test_done\n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312753","messageId":"20170227180019.18666-14-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 13/22] Documentation/config: add splitIndex.maxPercentChange","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T18:00:10Z","receivedAt":"2017-02-27T18:55:52Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n Documentation/config.txt | 13 +++++++++++++\n 1 file changed, 13 insertions(+)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 61a863adeb..8e745bda52 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2831,6 +2831,19 @@ showbranch.default::\n \tThe default set of branches for linkgit:git-show-branch[1].\n \tSee linkgit:git-show-branch[1].\n \n+splitIndex.maxPercentChange::\n+\tWhen the split index feature is used, this specifies the\n+\tpercent of entries the split index can contain compared to the\n+\ttotal number of entries in both the split index and the shared\n+\tindex before a new shared index is written.\n+\tThe value should be between 0 and 100. If the value is 0 then\n+\ta new shared index is always written, if it is 100 a new\n+\tshared index is never written.\n+\tBy default the value is 20, so a new shared index is written\n+\tif the number of entries in the split index would be greater\n+\tthan 20 percent of the total number of entries.\n+\tSee linkgit:git-update-index[1].\n+\n status.relativePaths::\n \tBy default, linkgit:git-status[1] shows paths relative to the\n \tcurrent directory. Setting this variable to `false` shows paths\n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312756","messageId":"20170227180019.18666-16-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 15/22] read-cache: touch shared index files when used","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T18:00:12Z","receivedAt":"2017-02-27T18:59:21Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"When a split-index file is created, let's update the mtime of the\nshared index file that the split-index file is referencing.\n\nIn a following commit we will make shared index file expire\ndepending on their mtime, so updating the mtime makes sure that\nthe shared index file will not be deleted soon.\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n read-cache.c | 29 ++++++++++++++++++++++++++---\n 1 file changed, 26 insertions(+), 3 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex aeb413a508..5f295af4c6 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1674,6 +1674,19 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tdie(\"index file corrupt\");\n }\n \n+/*\n+ * Signal that the shared index is used by updating its mtime.\n+ *\n+ * This way, shared index can be removed if they have not been used\n+ * for some time.\n+ */\n+static void freshen_shared_index(char *base_sha1_hex, int warn)\n+{\n+\tconst char *shared_index = git_path(\"sharedindex.%s\", base_sha1_hex);\n+\tif (!check_and_freshen_file(shared_index, 1) && warn)\n+\t\twarning(\"Could not freshen shared index '%s'\", shared_index);\n+}\n+\n int read_index_from(struct index_state *istate, const char *path)\n {\n \tstruct split_index *split_index;\n@@ -2245,6 +2258,7 @@ static int too_many_not_shared_entries(struct index_state *istate)\n int write_locked_index(struct index_state *istate, struct lock_file *lock,\n \t\t       unsigned flags)\n {\n+\tint new_shared_index, ret;\n \tstruct split_index *si = istate->split_index;\n \n \tif (!si || alternate_index_output ||\n@@ -2261,13 +2275,22 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,\n \t}\n \tif (too_many_not_shared_entries(istate))\n \t\tistate->cache_changed |= SPLIT_INDEX_ORDERED;\n-\tif (istate->cache_changed & SPLIT_INDEX_ORDERED) {\n-\t\tint ret = write_shared_index(istate, lock, flags);\n+\n+\tnew_shared_index = istate->cache_changed & SPLIT_INDEX_ORDERED;\n+\n+\tif (new_shared_index) {\n+\t\tret = write_shared_index(istate, lock, flags);\n \t\tif (ret)\n \t\t\treturn ret;\n \t}\n \n-\treturn write_split_index(istate, lock, flags);\n+\tret = write_split_index(istate, lock, flags);\n+\n+\t/* Freshen the shared index only if the split-index was written */\n+\tif (!ret && !new_shared_index)\n+\t\tfreshen_shared_index(sha1_to_hex(si->base_sha1), 1);\n+\n+\treturn ret;\n }\n \n /*\n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312757","messageId":"20170227180019.18666-18-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 17/22] read-cache: unlink old sharedindex files","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T18:00:14Z","receivedAt":"2017-02-27T18:59:27Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Everytime split index is turned on, it creates a \"sharedindex.XXXX\"\nfile in the git directory. This change makes sure that shared index\nfiles that haven't been used for a long time are removed when a new\nshared index file is created.\n\nThe new \"splitIndex.sharedIndexExpire\" config variable is created\nto tell the delay after which an unused shared index file can be\ndeleted. It defaults to \"2.weeks.ago\".\n\nA previous commit made sure that each time a split index file is\ncreated the mtime of the shared index file it references is updated.\nThis makes sure that recently used shared index file will not be\ndeleted.\n\nSigned-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n read-cache.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 63 insertions(+), 1 deletion(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 5f295af4c6..45fc831010 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2199,6 +2199,65 @@ static int write_split_index(struct index_state *istate,\n \treturn ret;\n }\n \n+static const char *shared_index_expire = \"2.weeks.ago\";\n+\n+static unsigned long get_shared_index_expire_date(void)\n+{\n+\tstatic unsigned long shared_index_expire_date;\n+\tstatic int shared_index_expire_date_prepared;\n+\n+\tif (!shared_index_expire_date_prepared) {\n+\t\tgit_config_get_expiry(\"splitindex.sharedindexexpire\",\n+\t\t\t\t      &shared_index_expire);\n+\t\tshared_index_expire_date = approxidate(shared_index_expire);\n+\t\tshared_index_expire_date_prepared = 1;\n+\t}\n+\n+\treturn shared_index_expire_date;\n+}\n+\n+static int can_delete_shared_index(const char *shared_index_path)\n+{\n+\tstruct stat st;\n+\tunsigned long expiration;\n+\n+\t/* Check timestamp */\n+\texpiration = get_shared_index_expire_date();\n+\tif (!expiration)\n+\t\treturn 0;\n+\tif (stat(shared_index_path, &st))\n+\t\treturn error_errno(_(\"could not stat '%s\"), shared_index_path);\n+\tif (st.st_mtime > expiration)\n+\t\treturn 0;\n+\n+\treturn 1;\n+}\n+\n+static int clean_shared_index_files(const char *current_hex)\n+{\n+\tstruct dirent *de;\n+\tDIR *dir = opendir(get_git_dir());\n+\n+\tif (!dir)\n+\t\treturn error_errno(_(\"unable to open git dir: %s\"), get_git_dir());\n+\n+\twhile ((de = readdir(dir)) != NULL) {\n+\t\tconst char *sha1_hex;\n+\t\tconst char *shared_index_path;\n+\t\tif (!skip_prefix(de->d_name, \"sharedindex.\", &sha1_hex))\n+\t\t\tcontinue;\n+\t\tif (!strcmp(sha1_hex, current_hex))\n+\t\t\tcontinue;\n+\t\tshared_index_path = git_path(\"%s\", de->d_name);\n+\t\tif (can_delete_shared_index(shared_index_path) > 0 &&\n+\t\t    unlink(shared_index_path))\n+\t\t\terror_errno(_(\"unable to unlink: %s\"), shared_index_path);\n+\t}\n+\tclosedir(dir);\n+\n+\treturn 0;\n+}\n+\n static struct tempfile temporary_sharedindex;\n \n static int write_shared_index(struct index_state *istate,\n@@ -2220,8 +2279,11 @@ static int write_shared_index(struct index_state *istate,\n \t}\n \tret = rename_tempfile(&temporary_sharedindex,\n \t\t\t      git_path(\"sharedindex.%s\", sha1_to_hex(si->base->sha1)));\n-\tif (!ret)\n+\tif (!ret) {\n \t\thashcpy(si->base_sha1, si->base->sha1);\n+\t\tclean_shared_index_files(sha1_to_hex(si->base->sha1));\n+\t}\n+\n \treturn ret;\n }\n \n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312758","messageId":"20170227180019.18666-22-chriscool@tuxfamily.org","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"[PATCH v4 21/22] Documentation/config: add splitIndex.sharedIndexExpire","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2017-02-27T18:00:18Z","receivedAt":"2017-02-27T19:09:04Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder <chriscool@tuxfamily.org>\n---\n Documentation/config.txt | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 8e745bda52..0e9982c5e3 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2844,6 +2844,18 @@ splitIndex.maxPercentChange::\n \tthan 20 percent of the total number of entries.\n \tSee linkgit:git-update-index[1].\n \n+splitIndex.sharedIndexExpire::\n+\tWhen the split index feature is used, shared index files that\n+\twere not modified since the time this variable specifies will\n+\tbe removed when a new shared index file is created. The value\n+\t\"now\" expires all entries immediately, and \"never\" suppresses\n+\texpiration altogether.\n+\tThe default value is \"2.weeks.ago\".\n+\tNote that a shared index file is considered modified (for the\n+\tpurpose of expiration) each time a new split-index file is\n+\tcreated based on it.\n+\tSee linkgit:git-update-index[1].\n+\n status.relativePaths::\n \tBy default, linkgit:git-status[1] shows paths relative to the\n \tcurrent directory. Setting this variable to `false` shows paths\n-- \n2.12.0.22.g0672473d40\n\n"},{"id":"312980","messageId":"xmqqinnsfzqp.fsf@gitster.mtv.corp.google.com","threadId":"45240","inReplyTo":"20170227180019.18666-16-chriscool@tuxfamily.org","subject":"Re: [PATCH v4 15/22] read-cache: touch shared index files when used","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-03-01T21:34:06Z","receivedAt":"2017-03-01T21:34:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder <christian.couder@gmail.com> writes:\n\n> +/*\n> + * Signal that the shared index is used by updating its mtime.\n> + *\n> + * This way, shared index can be removed if they have not been used\n> + * for some time.\n> + */\n> +static void freshen_shared_index(char *base_sha1_hex, int warn)\n> +{\n> +\tconst char *shared_index = git_path(\"sharedindex.%s\", base_sha1_hex);\n> +\tif (!check_and_freshen_file(shared_index, 1) && warn)\n> +\t\twarning(\"Could not freshen shared index '%s'\", shared_index);\n\nI'll downcase \"Could\" to match existing convention (no need to\nresend only to update this one).\n"},{"id":"312981","messageId":"xmqqmvd4fzxr.fsf@gitster.mtv.corp.google.com","threadId":"45240","inReplyTo":"20170227180019.18666-1-chriscool@tuxfamily.org","subject":"Re: [PATCH v4 00/22] Add configuration options for split-index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-03-01T21:29:52Z","receivedAt":"2017-03-01T21:35:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder <christian.couder@gmail.com> writes:\n\n> Highlevel view of the patches in the series\n> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n>\n> Except for patch 1/22 and 1/22, there are 3 big steps, one for each\n> new configuration variable introduced.\n>\n> There only small differences between this patch series and the v3\n> patch series sent a few months ago.\n>\n>     - Patch 1/22 marks a message for translation. It is not new and\n>       can be applied separately.\n>\n>     - Patch 2/22 improves the existing indentation style of t1700 by\n>       using different here document style. It is a new preparatory\n>       patch suggested by Junio.\n\nOK.  I read interdiff against the previous round carefully, and\nskimmed all patches less carefully.  \n\nI may have comments on individual patches later, but this round\nlooked mostly sensible.\n\nThanks for following it through.\n\n"},{"id":"312983","messageId":"xmqqefygfzhn.fsf@gitster.mtv.corp.google.com","threadId":"45240","inReplyTo":"20170227180019.18666-18-chriscool@tuxfamily.org","subject":"Re: [PATCH v4 17/22] read-cache: unlink old sharedindex files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-03-01T21:39:32Z","receivedAt":"2017-03-01T21:45:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Christian Couder <christian.couder@gmail.com> writes:\n\n> +static int can_delete_shared_index(const char *shared_index_path)\n> +{\n> +\tstruct stat st;\n> +\tunsigned long expiration;\n> +\n> +\t/* Check timestamp */\n> +\texpiration = get_shared_index_expire_date();\n> +\tif (!expiration)\n> +\t\treturn 0;\n> +\tif (stat(shared_index_path, &st))\n> +\t\treturn error_errno(_(\"could not stat '%s\"), shared_index_path);\n> +\tif (st.st_mtime > expiration)\n> +\t\treturn 0;\n> +\n> +\treturn 1;\n> +}\n> +\n> +static int clean_shared_index_files(const char *current_hex)\n> +{\n> +\tstruct dirent *de;\n> +\tDIR *dir = opendir(get_git_dir());\n> +\n> +\tif (!dir)\n> +\t\treturn error_errno(_(\"unable to open git dir: %s\"), get_git_dir());\n> +\n> +\twhile ((de = readdir(dir)) != NULL) {\n> +\t\tconst char *sha1_hex;\n> +\t\tconst char *shared_index_path;\n> +\t\tif (!skip_prefix(de->d_name, \"sharedindex.\", &sha1_hex))\n> +\t\t\tcontinue;\n> +\t\tif (!strcmp(sha1_hex, current_hex))\n> +\t\t\tcontinue;\n> +\t\tshared_index_path = git_path(\"%s\", de->d_name);\n> +\t\tif (can_delete_shared_index(shared_index_path) > 0 &&\n\nIs this \"can\" or \"should\"?  This sounds like the latter.\n\n> +\t\t    unlink(shared_index_path))\n> +\t\t\terror_errno(_(\"unable to unlink: %s\"), shared_index_path);\n\nThis does not make the entire operation to fail (and I think the\nbehaviour you have here is preferrable--we just want to report\nwithout failing the main operation).\n\nBut should it be reported as \"error: unable to unlink\"?  It may be\nbetter to give this message as a warning.\n"}]}