{"thread":{"id":"44726","subject":"[PATCH v2 00/21] Add configuration options for split-index","startedAt":"2016-12-17T14:56:09Z","lastAt":"2016-12-26T10:32:41Z","messageCount":34,"participants":["Christian Couder","Duy Nguyen","Junio C Hamano"],"isPatch":true,"patchVersion":2,"patchTotal":21},"messages":[{"id":"307959","messageId":"20161217145547.11748-1-chriscool@tuxfamily.org","threadId":"44726","inReplyTo":null,"subject":"[PATCH v2 00/21] Add configuration options for split-index","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T14:55:26Z","receivedAt":"2016-12-17T14:56:09Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Goal\n~~~~\n\nWe want to make it possible to use the split-index feature\nautomatically by just setting a new \"core.splitIndex\" configuration\nvariable to true.\n\nThis can be valuable as split-index can help significantly speed up\n`git rebase` especially along with the work to libify `git apply`\nthat has been merged to master\n(see https://github.com/git/git/commit/81358dc238372793b1590efa149cc1581d1fbd98)\nand is now in v2.11.\n\nDesign\n~~~~~~\n\nThe design is similar as the previous work that introduced\n\"core.untrackedCache\". \n\nThe new \"core.splitIndex\" configuration option can be either true,\nfalse or undefined which is the default.\n\nWhen it is true, the split index is created, if it does not already\nexists, when the index is read. When it is false, the split index is\nremoved if it exists, when the index is read. Otherwise it is left as\nis.\n\nAlong with this new configuration variable, the two following options\nare also introduced:\n\n - splitIndex.maxPercentChange\n\n This is to avoid having too many changes accumulating in the split\n index while in split index mode. The git-update-index\n documentation says:\n\n\tIf split-index mode is already enabled and `--split-index` is\n\tgiven again, all changes in $GIT_DIR/index are pushed back to\n\tthe shared index file.\n\n but it is probably better to not expect the user to think about it\n and to have a mechanism that pushes back all changes to the shared\n index file automatically when some threshold is reached.\n\n The default threshold is when the number of entries in the split\n index file reaches 20% of the number of entries in the shared\n index file. The new \"splitIndex.maxPercentChange\" config option\n lets people tweak this value.\n\n - splitIndex.sharedIndexExpire\n\n To make sure that old sharedindex files are eventually removed\n when a new one has been created, we \"touch\" the shared index file\n every time a split index file using the shared index file is\n either created or read from. Then we can delete shared indexes\n with an mtime older than one week (by default), when we create a\n new shared index file. The new \"splitIndex.sharedIndexExpire\"\n config option lets people tweak this grace period.\n\n This idea was suggested by Duy in:\n\n https://public-inbox.org/git/CACsJy8BqMFASHf5kJgUh+bd7XG98CafNydE964VJyPXz-emEvA@mail.gmail.com/\n\n and after some experiments, I agree that it is much simpler than\n what I thought could be done during our discussion.\n\n Junio also thinks that we have to do \"time-based GC\" in:\n \n https://public-inbox.org/git/xmqqeg33ccjj.fsf@gitster.mtv.corp.google.com/\n\nHighlevel view of the patches in the series\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nExcept for patch 1/21, there are 3 big steps, one for each new\nconfiguration variable introduced.\n\nThe main difference between this patch series and the v1 patch series\nsent last October is that Step 3 has a few new commits to also update\nthe mtime of the shared index file when a split index based on the\nshared index file is read from.\n\n - Patch 1/21 marks a message for translation. It is a new patch\n and it can be applied separately. (The patch 1/19 in v1, which\n was a typo fix, has been merged separately into master already.)\n\nStep 1 is:\n\n - Patches 2/21 to 5/21 introduce the functions that are reading\n the \"core.splitIndex\" configuration variable and tweaking the\n split index depending on its value.\n\n - Patch 6/21 adds a few tests for the new feature.\n\n - Patches 7/21 and 8/21 add some documentation for the new\n feature.\n\n The only change since v1 in this step is that some warning\n messages in 5/21 have been marked for translation as suggested\n by Duy.\n\nStep 2 is:\n\n - Patches 9/21 and 10/21 introduce the functions that are reading\n the \"splitIndex.maxPercentChange\" configuration variable and\n regenerating a new shared index file depending on its value.\n\n - Patch 11/21 adds a few tests for the new feature.\n\n - Patch 12/21 add some documentation for the new feature.\n\n The changes since v1 in this step are:\n\n - an error message has been marked for translation in 9/21,\n - camelCase is used in the same error message as suggested by\n Duy in 9/21,\n\t- \"return error(...)\" is now used as suggested by Junio in\n 9/21,\n\t- too_many_not_shared_entries() is now \"static\" as suggested\n by Ramsay in 10/21,\n\t- changes made in write_locked_index() have been reorganized\n as suggested by Duy in 10/21,\n\nStep 3 is:\n\n - Patches 13/21 to 16/21 introduce the functions that are reading\n the \"splitIndex.sharedIndexExpire\" configuration variable and\n expiring old shared index files depending on its value.\n\n - Patch 17/21 adds a few tests for the new feature.\n\n - Patches 18/21 and 19/21 are new patches. They update the mtime\n of the shared index file when a split index based on the shared\n index file is read from. 18/21 is a refactoring to make the\n actual change in 19/21 easier.\n\n - Patches 20/21 and 21/21 add some documentation for the new\n feature.\n\n The changes since v1 in this step are:\n\n\t- a warning was removed from 14/21 as suggested by Junio,\n\t- code to touch the shared index file as been refactored into\n a function in 14/21,\n\t- a function has been renamed git_config_get_expiry() in 15/21\n as suggested by Junio,\n - error messages have been marked for translation in 16/21 as\n suggested by Duy,\n\t- \"return error_errno(...)\" is now used as suggested by Duy in\n 16/21,\n - patches 18/21 and 19/21 are new following Duy's suggestion,\n\t- patches 20/21 and 21/21 are updated to take changes made in\n\t 19/21 into account.\n\nLinks\n~~~~~\n\nThis patch series is also available here:\n\n https://github.com/chriscool/git/commits/config-split-index\n\nThe previous versions were:\n\n RFC: https://github.com/chriscool/git/commits/config-split-index7\n v1: https://github.com/chriscool/git/commits/config-split-index72\n\nOn the mailing list the related patch series and discussions were:\n\n RFC: https://public-inbox.org/git/20160711172254.13439-1-chriscool@tuxfamily.org/\n v1: https://public-inbox.org/git/20161023092648.12086-1-chriscool@tuxfamily.org/\n\nChristian Couder (21):\n config: mark an error message up for translation\n config: add git_config_get_split_index()\n split-index: add {add,remove}_split_index() functions\n read-cache: add and then use tweak_split_index()\n update-index: warn in case of split-index incoherency\n t1700: add tests for core.splitIndex\n Documentation/config: add information for core.splitIndex\n Documentation/git-update-index: talk about core.splitIndex config var\n config: add git_config_get_max_percent_split_change()\n read-cache: regenerate shared index if necessary\n t1700: add tests for splitIndex.maxPercentChange\n Documentation/config: add splitIndex.maxPercentChange\n sha1_file: make check_and_freshen_file() non static\n read-cache: touch shared index files when used\n config: add git_config_get_expiry() from gc.c\n read-cache: unlink old sharedindex files\n t1700: test shared index file expiration\n read-cache: refactor read_index_from()\n read-cache: use freshen_shared_index() in read_index_from()\n Documentation/config: add splitIndex.sharedIndexExpire\n Documentation/git-update-index: explain splitIndex.*\n\n Documentation/config.txt | 28 +++++++\n Documentation/git-update-index.txt | 43 +++++++++--\n builtin/gc.c | 15 +---\n builtin/update-index.c | 25 +++---\n cache.h | 8 ++\n config.c | 42 +++++++++-\n read-cache.c | 142 ++++++++++++++++++++++++++++++++--\n sha1_file.c | 2 +-\n split-index.c | 22 ++++++\n split-index.h | 2 +\n t/t1700-split-index.sh | 154 +++++++++++++++++++++++++++++++++++++\n 11 files changed, 441 insertions(+), 42 deletions(-)\n\n-- \n2.11.0.53.gb263787\n\n"},{"id":"307960","messageId":"20161217145547.11748-2-chriscool@tuxfamily.org","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"[PATCH v2 01/21] config: mark an error message up for translation","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T14:55:27Z","receivedAt":"2016-12-17T14:56:15Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n config.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/config.c b/config.c\nindex 83fdecb1bc..2eaf8ad77a 100644\n--- a/config.c\n+++ b/config.c\n@@ -1701,8 +1701,8 @@ int git_config_get_untracked_cache(void)\n \t\tif (!strcasecmp(v, \"keep\"))\n \t\t\treturn -1;\n \n-\t\terror(\"unknown core.untrackedCache value '%s'; \"\n-\t\t \"using 'keep' default value\", v);\n+\t\terror(_(\"unknown core.untrackedCache value '%s'; \"\n+\t\t\t\"using 'keep' default value\"), v);\n \t\treturn -1;\n \t}\n \n-- \n2.11.0.49.g2414764.dirty\n\n"},{"id":"307961","messageId":"20161217145547.11748-3-chriscool@tuxfamily.org","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"[PATCH v2 02/21] config: add git_config_get_split_index()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T14:55:28Z","receivedAt":"2016-12-17T14:56:18Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This new function will be used in a following commit to know\nif we want to use the split index feature or not.\n\nSigned-off-by: Christian Couder \n---\n cache.h | 1 +\n config.c | 10 ++++++++++\n 2 files changed, 11 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex a50a61a197..c126fe475e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1821,6 +1821,7 @@ extern int git_config_get_bool_or_int(const char *key, int *is_bool, int *dest);\n extern int git_config_get_maybe_bool(const char *key, int *dest);\n extern int git_config_get_pathname(const char *key, const char **dest);\n extern int git_config_get_untracked_cache(void);\n+extern int git_config_get_split_index(void);\n \n /*\n * This is a hack for test programs like test-dump-untracked-cache to\ndiff --git a/config.c b/config.c\nindex 2eaf8ad77a..c1343bbb3e 100644\n--- a/config.c\n+++ b/config.c\n@@ -1709,6 +1709,16 @@ int git_config_get_untracked_cache(void)\n \treturn -1; /* default value */\n }\n \n+int git_config_get_split_index(void)\n+{\n+\tint val = -1;\n+\n+\tif (!git_config_get_maybe_bool(\"core.splitindex\", &val))\n+\t\treturn val;\n+\n+\treturn -1; /* default value */\n+}\n+\n NORETURN\n void git_die_config_linenr(const char *key, const char *filename, int linenr)\n {\n-- \n2.11.0.49.g2414764.dirty\n\n"},{"id":"307962","messageId":"20161217145547.11748-7-chriscool@tuxfamily.org","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"[PATCH v2 06/21] t1700: add tests for core.splitIndex","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T14:55:32Z","receivedAt":"2016-12-17T14:56:21Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t1700-split-index.sh | 37 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 37 insertions(+)\n\ndiff --git a/t/t1700-split-index.sh b/t/t1700-split-index.sh\nindex 292a0720fc..db8c39f446 100755\n--- a/t/t1700-split-index.sh\n+++ b/t/t1700-split-index.sh\n@@ -200,4 +200,41 @@ EOF\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'set core.splitIndex config variable to true' '\n+\tgit config core.splitIndex true &&\n+\t: >three &&\n+\tgit update-index --add three &&\n+\tgit ls-files --stage >ls-files.actual &&\n+\tcat >ls-files.expect <actual &&\n+\tcat >expect <ls-files.actual &&\n+\tcat >ls-files.expect <actual &&\n+\tcat >expect <\n---\n cache.h | 1 +\n config.c | 15 +++++++++++++++\n 2 files changed, 16 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex c126fe475e..e15b421b6f 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1822,6 +1822,7 @@ extern int git_config_get_maybe_bool(const char *key, int *dest);\n extern int git_config_get_pathname(const char *key, const char **dest);\n extern int git_config_get_untracked_cache(void);\n extern int git_config_get_split_index(void);\n+extern int git_config_get_max_percent_split_change(void);\n \n /*\n * This is a hack for test programs like test-dump-untracked-cache to\ndiff --git a/config.c b/config.c\nindex c1343bbb3e..3e96c223f5 100644\n--- a/config.c\n+++ b/config.c\n@@ -1719,6 +1719,21 @@ int git_config_get_split_index(void)\n \treturn -1; /* default value */\n }\n \n+int git_config_get_max_percent_split_change(void)\n+{\n+\tint val = -1;\n+\n+\tif (!git_config_get_int(\"splitindex.maxpercentchange\", &val)) {\n+\t\tif (0 <= val && val <= 100)\n+\t\t\treturn val;\n+\n+\t\treturn error(_(\"splitIndex.maxPercentChange value '%d' \"\n+\t\t\t \"should be between 0 and 100\"), val);\n+\t}\n+\n+\treturn -1; /* default value */\n+}\n+\n NORETURN\n void git_die_config_linenr(const char *key, const char *filename, int linenr)\n {\n-- \n2.11.0.49.g2414764.dirty\n\n"},{"id":"307964","messageId":"20161217145547.11748-9-chriscool@tuxfamily.org","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"[PATCH v2 08/21] Documentation/git-update-index: talk about core.splitIndex config var","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T14:55:34Z","receivedAt":"2016-12-17T14:56:24Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n Documentation/git-update-index.txt | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex 7386c93162..e091b2a409 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -171,6 +171,12 @@ may not support it yet.\n \tgiven again, all changes in $GIT_DIR/index are pushed back to\n \tthe shared index file. This mode is designed for very large\n \tindexes that take a significant amount of time to read or write.\n++\n+These options take effect whatever the value of the `core.splitIndex`\n+configuration variable (see linkgit:git-config[1]). But a warning is\n+emitted when the change goes against the configured value, as the\n+configured value will take effect next time the index is read and this\n+will remove the intended effect of the option.\n \n --untracked-cache::\n --no-untracked-cache::\n-- \n2.11.0.49.g2414764.dirty\n\n"},{"id":"307965","messageId":"20161217145547.11748-13-chriscool@tuxfamily.org","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"[PATCH v2 12/21] Documentation/config: add splitIndex.maxPercentChange","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T14:55:38Z","receivedAt":"2016-12-17T14:56:27Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n Documentation/config.txt | 13 +++++++++++++\n 1 file changed, 13 insertions(+)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex dc44d8a417..08f638c65c 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2763,6 +2763,19 @@ showbranch.default::\n \tThe default set of branches for linkgit:git-show-branch[1].\n \tSee linkgit:git-show-branch[1].\n \n+splitIndex.maxPercentChange::\n+\tWhen the split index feature is used, this specifies the\n+\tpercent of entries the split index can contain compared to the\n+\twhole number of entries in both the split index and the shared\n+\tindex before a new shared index is written.\n+\tThe value should be between 0 and 100. If the value is 0 then\n+\ta new shared index is always written, if it is 100 a new\n+\tshared index is never written.\n+\tBy default the value is 20, so a new shared index is written\n+\tif the number of entries in the split index would be greater\n+\tthan 20 percent of the total number of entries.\n+\tSee linkgit:git-update-index[1].\n+\n status.relativePaths::\n \tBy default, linkgit:git-status[1] shows paths relative to the\n \tcurrent directory. Setting this variable to `false` shows paths\n-- \n2.11.0.49.g2414764.dirty\n\n"},{"id":"307966","messageId":"20161217145547.11748-15-chriscool@tuxfamily.org","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"[PATCH v2 14/21] read-cache: touch shared index files when used","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T14:55:40Z","receivedAt":"2016-12-17T14:56:29Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"When a split-index file is created, let's update the mtime of the\nshared index file that the split-index file is referencing.\n\nIn a following commit we will make shared index file expire\ndepending on their mtime, so updating the mtime makes sure that\nthe shared index file will not be deleted soon.\n\nSigned-off-by: Christian Couder \n---\n read-cache.c | 15 +++++++++++++++\n 1 file changed, 15 insertions(+)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 280b01bd6c..42688261f7 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1685,6 +1685,19 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n \tdie(\"index file corrupt\");\n }\n \n+/*\n+ * Signal that the shared index is used by updating its mtime.\n+ *\n+ * This way, shared index can be removed if they have not been used\n+ * for some time. It's ok to fail to update the mtime if we are on a\n+ * read only file system.\n+ */\n+void freshen_shared_index(char *base_sha1_hex)\n+{\n+\tconst char *shared_index = git_path(\"sharedindex.%s\", base_sha1_hex);\n+\tcheck_and_freshen_file(shared_index, 1);\n+}\n+\n int read_index_from(struct index_state *istate, const char *path)\n {\n \tstruct split_index *split_index;\n@@ -2276,6 +2289,8 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,\n \t\tint ret = write_shared_index(istate, lock, flags);\n \t\tif (ret)\n \t\t\treturn ret;\n+\t} else {\n+\t\tfreshen_shared_index(sha1_to_hex(si->base_sha1));\n \t}\n \n \treturn write_split_index(istate, lock, flags);\n-- \n2.11.0.49.g2414764.dirty\n\n"},{"id":"307967","messageId":"20161217145547.11748-17-chriscool@tuxfamily.org","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"[PATCH v2 16/21] read-cache: unlink old sharedindex files","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T14:55:42Z","receivedAt":"2016-12-17T14:56:33Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Everytime split index is turned on, it creates a \"sharedindex.XXXX\"\nfile in the git directory. This change makes sure that shared index\nfiles that haven't been used for a long time are removed when a new\nshared index file is created.\n\nThe new \"splitIndex.sharedIndexExpire\" config variable is created\nto tell the delay after which an unused shared index file can be\ndeleted. It defaults to \"1.week.ago\".\n\nA previous commit made sure that each time a split index file is\ncreated the mtime of the shared index file it references is updated.\nThis makes sure that recently used shared index file will not be\ndeleted.\n\nSigned-off-by: Christian Couder \n---\n read-cache.c | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 62 insertions(+), 1 deletion(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 42688261f7..8e99a7c325 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2210,6 +2210,64 @@ static int write_split_index(struct index_state *istate,\n \treturn ret;\n }\n \n+static const char *shared_index_expire = \"1.week.ago\";\n+\n+static unsigned long get_shared_index_expire_date(void)\n+{\n+\tstatic unsigned long shared_index_expire_date;\n+\tstatic int shared_index_expire_date_prepared;\n+\n+\tif (!shared_index_expire_date_prepared) {\n+\t\tgit_config_get_expiry(\"splitindex.sharedindexexpire\",\n+\t\t\t\t &shared_index_expire);\n+\t\tshared_index_expire_date = approxidate(shared_index_expire);\n+\t\tshared_index_expire_date_prepared = 1;\n+\t}\n+\n+\treturn shared_index_expire_date;\n+}\n+\n+static int can_delete_shared_index(const char *shared_sha1_hex)\n+{\n+\tstruct stat st;\n+\tunsigned long expiration;\n+\tconst char *shared_index = git_path(\"sharedindex.%s\", shared_sha1_hex);\n+\n+\t/* Check timestamp */\n+\texpiration = get_shared_index_expire_date();\n+\tif (!expiration)\n+\t\treturn 0;\n+\tif (stat(shared_index, &st))\n+\t\treturn error_errno(_(\"could not stat '%s\"), shared_index);\n+\tif (st.st_mtime > expiration)\n+\t\treturn 0;\n+\n+\treturn 1;\n+}\n+\n+static int clean_shared_index_files(const char *current_hex)\n+{\n+\tstruct dirent *de;\n+\tDIR *dir = opendir(get_git_dir());\n+\n+\tif (!dir)\n+\t\treturn error_errno(_(\"unable to open git dir: %s\"), get_git_dir());\n+\n+\twhile ((de = readdir(dir)) != NULL) {\n+\t\tconst char *sha1_hex;\n+\t\tif (!skip_prefix(de->d_name, \"sharedindex.\", &sha1_hex))\n+\t\t\tcontinue;\n+\t\tif (!strcmp(sha1_hex, current_hex))\n+\t\t\tcontinue;\n+\t\tif (can_delete_shared_index(sha1_hex) > 0 &&\n+\t\t unlink(git_path(\"%s\", de->d_name)))\n+\t\t\terror_errno(_(\"unable to unlink: %s\"), git_path(\"%s\", de->d_name));\n+\t}\n+\tclosedir(dir);\n+\n+\treturn 0;\n+}\n+\n static struct tempfile temporary_sharedindex;\n \n static int write_shared_index(struct index_state *istate,\n@@ -2231,8 +2289,11 @@ static int write_shared_index(struct index_state *istate,\n \t}\n \tret = rename_tempfile(&temporary_sharedindex,\n \t\t\t git_path(\"sharedindex.%s\", sha1_to_hex(si->base->sha1)));\n-\tif (!ret)\n+\tif (!ret) {\n \t\thashcpy(si->base_sha1, si->base->sha1);\n+\t\tclean_shared_index_files(sha1_to_hex(si->base->sha1));\n+\t}\n+\n \treturn ret;\n }\n \n-- \n2.11.0.49.g2414764.dirty\n\n"},{"id":"307968","messageId":"20161217145547.11748-18-chriscool@tuxfamily.org","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"[PATCH v2 17/21] t1700: test shared index file expiration","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T14:55:43Z","receivedAt":"2016-12-17T14:56:37Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t1700-split-index.sh | 44 ++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 44 insertions(+)\n\ndiff --git a/t/t1700-split-index.sh b/t/t1700-split-index.sh\nindex f03addf654..f448fc13cd 100755\n--- a/t/t1700-split-index.sh\n+++ b/t/t1700-split-index.sh\n@@ -310,4 +310,48 @@ EOF\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'shared index files expire after 7 days by default' '\n+\t: >ten &&\n+\tgit update-index --add ten &&\n+\ttest $(ls .git/sharedindex.* | wc -l) -gt 1 &&\n+\tjust_under_7_days_ago=$((1-7*86400)) &&\n+\ttest-chmtime =$just_under_7_days_ago .git/sharedindex.* &&\n+\t: >eleven &&\n+\tgit update-index --add eleven &&\n+\ttest $(ls .git/sharedindex.* | wc -l) -gt 1 &&\n+\tjust_over_7_days_ago=$((-1-7*86400)) &&\n+\ttest-chmtime =$just_over_7_days_ago .git/sharedindex.* &&\n+\t: >twelve &&\n+\tgit update-index --add twelve &&\n+\ttest $(ls .git/sharedindex.* | wc -l) = 1\n+'\n+\n+test_expect_success 'check splitIndex.sharedIndexExpire set to 8 days' '\n+\tgit config splitIndex.sharedIndexExpire \"8.days.ago\" &&\n+\ttest-chmtime =$just_over_7_days_ago .git/sharedindex.* &&\n+\t: >thirteen &&\n+\tgit update-index --add thirteen &&\n+\ttest $(ls .git/sharedindex.* | wc -l) -gt 1 &&\n+\tjust_over_8_days_ago=$((-1-8*86400)) &&\n+\ttest-chmtime =$just_over_8_days_ago .git/sharedindex.* &&\n+\t: >fourteen &&\n+\tgit update-index --add fourteen &&\n+\ttest $(ls .git/sharedindex.* | wc -l) = 1\n+'\n+\n+test_expect_success 'check splitIndex.sharedIndexExpire set to \"never\" and \"now\"' '\n+\tgit config splitIndex.sharedIndexExpire never &&\n+\tjust_10_years_ago=$((-365*10*86400)) &&\n+\ttest-chmtime =$just_10_years_ago .git/sharedindex.* &&\n+\t: >fifteen &&\n+\tgit update-index --add fifteen &&\n+\ttest $(ls .git/sharedindex.* | wc -l) -gt 1 &&\n+\tgit config splitIndex.sharedIndexExpire now &&\n+\tjust_1_second_ago=-1 &&\n+\ttest-chmtime =$just_1_second_ago .git/sharedindex.* &&\n+\t: >sixteen &&\n+\tgit update-index --add sixteen &&\n+\ttest $(ls .git/sharedindex.* | wc -l) = 1\n+'\n+\n test_done\n-- \n2.11.0.49.g2414764.dirty\n\n"},{"id":"307969","messageId":"20161217145547.11748-20-chriscool@tuxfamily.org","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"[PATCH v2 19/21] read-cache: use freshen_shared_index() in read_index_from()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T14:55:45Z","receivedAt":"2016-12-17T14:56:38Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This way a share index file will not be garbage collected if\nwe still read from an index it is based from.\n\nAs we need to read the current index before creating a new\none, the tests have to be adjusted, so that we don't expect\nan old shared index file to be deleted right away when we\ncreate a new one.\n\nSigned-off-by: Christian Couder \n---\n read-cache.c | 1 +\n t/t1700-split-index.sh | 14 +++++++-------\n 2 files changed, 8 insertions(+), 7 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 96e097e341..35377f0a3e 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1730,6 +1730,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\t base_sha1_hex, base_path,\n \t\t sha1_to_hex(split_index->base->sha1));\n \n+\tfreshen_shared_index(base_sha1_hex);\n \tmerge_base_index(istate);\n \tpost_read_index_from(istate);\n \treturn ret;\ndiff --git a/t/t1700-split-index.sh b/t/t1700-split-index.sh\nindex f448fc13cd..800f84a593 100755\n--- a/t/t1700-split-index.sh\n+++ b/t/t1700-split-index.sh\n@@ -313,17 +313,17 @@ EOF\n test_expect_success 'shared index files expire after 7 days by default' '\n \t: >ten &&\n \tgit update-index --add ten &&\n-\ttest $(ls .git/sharedindex.* | wc -l) -gt 1 &&\n+\ttest $(ls .git/sharedindex.* | wc -l) -gt 2 &&\n \tjust_under_7_days_ago=$((1-7*86400)) &&\n \ttest-chmtime =$just_under_7_days_ago .git/sharedindex.* &&\n \t: >eleven &&\n \tgit update-index --add eleven &&\n-\ttest $(ls .git/sharedindex.* | wc -l) -gt 1 &&\n+\ttest $(ls .git/sharedindex.* | wc -l) -gt 2 &&\n \tjust_over_7_days_ago=$((-1-7*86400)) &&\n \ttest-chmtime =$just_over_7_days_ago .git/sharedindex.* &&\n \t: >twelve &&\n \tgit update-index --add twelve &&\n-\ttest $(ls .git/sharedindex.* | wc -l) = 1\n+\ttest $(ls .git/sharedindex.* | wc -l) -le 2\n '\n \n test_expect_success 'check splitIndex.sharedIndexExpire set to 8 days' '\n@@ -331,12 +331,12 @@ test_expect_success 'check splitIndex.sharedIndexExpire set to 8 days' '\n \ttest-chmtime =$just_over_7_days_ago .git/sharedindex.* &&\n \t: >thirteen &&\n \tgit update-index --add thirteen &&\n-\ttest $(ls .git/sharedindex.* | wc -l) -gt 1 &&\n+\ttest $(ls .git/sharedindex.* | wc -l) -gt 2 &&\n \tjust_over_8_days_ago=$((-1-8*86400)) &&\n \ttest-chmtime =$just_over_8_days_ago .git/sharedindex.* &&\n \t: >fourteen &&\n \tgit update-index --add fourteen &&\n-\ttest $(ls .git/sharedindex.* | wc -l) = 1\n+\ttest $(ls .git/sharedindex.* | wc -l) -le 2\n '\n \n test_expect_success 'check splitIndex.sharedIndexExpire set to \"never\" and \"now\"' '\n@@ -345,13 +345,13 @@ test_expect_success 'check splitIndex.sharedIndexExpire set to \"never\" and \"now\"\n \ttest-chmtime =$just_10_years_ago .git/sharedindex.* &&\n \t: >fifteen &&\n \tgit update-index --add fifteen &&\n-\ttest $(ls .git/sharedindex.* | wc -l) -gt 1 &&\n+\ttest $(ls .git/sharedindex.* | wc -l) -gt 2 &&\n \tgit config splitIndex.sharedIndexExpire now &&\n \tjust_1_second_ago=-1 &&\n \ttest-chmtime =$just_1_second_ago .git/sharedindex.* &&\n \t: >sixteen &&\n \tgit update-index --add sixteen &&\n-\ttest $(ls .git/sharedindex.* | wc -l) = 1\n+\ttest $(ls .git/sharedindex.* | wc -l) -le 2\n '\n \n test_done\n-- \n2.11.0.49.g2414764.dirty\n\n"},{"id":"307970","messageId":"20161217145547.11748-22-chriscool@tuxfamily.org","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"[PATCH v2 21/21] Documentation/git-update-index: explain splitIndex.*","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T14:55:47Z","receivedAt":"2016-12-17T14:56:39Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n Documentation/config.txt | 6 +++---\n Documentation/git-update-index.txt | 37 +++++++++++++++++++++++++++++--------\n 2 files changed, 32 insertions(+), 11 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 8fbef25cb1..52a3cac4ff 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2782,9 +2782,9 @@ splitIndex.sharedIndexExpire::\n \tindex file is created. The value \"now\" expires all entries\n \timmediately, and \"never\" suppresses expiration altogether.\n \tThe default value is \"one.week.ago\".\n-\tNote that each time a new split-index file is created, the\n-\tmtime of the related shared index file is updated to the\n-\tcurrent time.\n+\tNote that each time a split index based on a shared index file\n+\tis either created or read from, the mtime of the shared index\n+\tfile is updated to the current time.\n \tSee linkgit:git-update-index[1].\n \n status.relativePaths::\ndiff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt\nindex e091b2a409..46c953b2f2 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -163,14 +163,10 @@ may not support it yet.\n \n --split-index::\n --no-split-index::\n-\tEnable or disable split index mode. If enabled, the index is\n-\tsplit into two files, $GIT_DIR/index and $GIT_DIR/sharedindex..\n-\tChanges are accumulated in $GIT_DIR/index while the shared\n-\tindex file contains all index entries stays unchanged. If\n-\tsplit-index mode is already enabled and `--split-index` is\n-\tgiven again, all changes in $GIT_DIR/index are pushed back to\n-\tthe shared index file. This mode is designed for very large\n-\tindexes that take a significant amount of time to read or write.\n+\tEnable or disable split index mode. If split-index mode is\n+\talready enabled and `--split-index` is given again, all\n+\tchanges in $GIT_DIR/index are pushed back to the shared index\n+\tfile.\n +\n These options take effect whatever the value of the `core.splitIndex`\n configuration variable (see linkgit:git-config[1]). But a warning is\n@@ -394,6 +390,31 @@ Although this bit looks similar to assume-unchanged bit, its goal is\n different from assume-unchanged bit's. Skip-worktree also takes\n precedence over assume-unchanged bit when both are set.\n \n+Split index\n+-----------\n+\n+This mode is designed for very large indexes that take a significant\n+amount of time to read or write.\n+\n+In this mode, the index is split into two files, $GIT_DIR/index and\n+$GIT_DIR/sharedindex.. Changes are accumulated in\n+$GIT_DIR/index, the split index, while the shared index file contains\n+all index entries and stays unchanged.\n+\n+All changes in the split index are pushed back to the shared index\n+file when the number of entries in the split index reaches a level\n+specified by the splitIndex.maxPercentChange config variable (see\n+linkgit:git-config[1]).\n+\n+Each time a new shared index file is created, the old shared index\n+files are deleted if their mtime is older than what is specified by\n+the splitIndex.sharedIndexExpire config variable (see\n+linkgit:git-config[1]).\n+\n+To avoid deleting a shared index file that is still used, its mtime is\n+updated to the current time everytime a new split index based on the\n+shared index file is either created or read from.\n+\n Untracked cache\n ---------------\n \n-- \n2.11.0.49.g2414764.dirty\n\n"},{"id":"307971","messageId":"20161217145547.11748-21-chriscool@tuxfamily.org","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"[PATCH v2 20/21] Documentation/config: add splitIndex.sharedIndexExpire","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T14:55:46Z","receivedAt":"2016-12-17T14:56:42Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n Documentation/config.txt | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 08f638c65c..8fbef25cb1 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2776,6 +2776,17 @@ splitIndex.maxPercentChange::\n \tthan 20 percent of the total number of entries.\n \tSee linkgit:git-update-index[1].\n \n+splitIndex.sharedIndexExpire::\n+\tWhen the split index feature is used, shared index files with\n+\ta mtime older than this time will be removed when a new shared\n+\tindex file is created. The value \"now\" expires all entries\n+\timmediately, and \"never\" suppresses expiration altogether.\n+\tThe default value is \"one.week.ago\".\n+\tNote that each time a new split-index file is created, the\n+\tmtime of the related shared index file is updated to the\n+\tcurrent time.\n+\tSee linkgit:git-update-index[1].\n+\n status.relativePaths::\n \tBy default, linkgit:git-status[1] shows paths relative to the\n \tcurrent directory. Setting this variable to `false` shows paths\n-- \n2.11.0.49.g2414764.dirty\n\n"},{"id":"307972","messageId":"20161217145547.11748-12-chriscool@tuxfamily.org","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"[PATCH v2 11/21] t1700: add tests for splitIndex.maxPercentChange","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T14:55:37Z","receivedAt":"2016-12-17T14:56:48Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n t/t1700-split-index.sh | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 72 insertions(+)\n\ndiff --git a/t/t1700-split-index.sh b/t/t1700-split-index.sh\nindex 507a1dd1ad..f03addf654 100755\n--- a/t/t1700-split-index.sh\n+++ b/t/t1700-split-index.sh\n@@ -238,4 +238,76 @@ EOF\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'set core.splitIndex config variable to true' '\n+\tgit config core.splitIndex true &&\n+\t: >three &&\n+\tgit update-index --add three &&\n+\tBASE=$(test-dump-split-index .git/index | grep \"^base\") &&\n+\ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n+\tcat >expect <four &&\n+\tgit update-index --add four &&\n+\ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n+\tcat >expect <five &&\n+\tgit update-index --add five &&\n+\tBASE=$(test-dump-split-index .git/index | grep \"^base\") &&\n+\ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n+\tcat >expect <six &&\n+\tgit update-index --add six &&\n+\ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n+\tcat >expect <seven &&\n+\tgit update-index --add seven &&\n+\tBASE=$(test-dump-split-index .git/index | grep \"^base\") &&\n+\ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n+\tcat >expect <eight &&\n+\tgit update-index --add eight &&\n+\tBASE=$(test-dump-split-index .git/index | grep \"^base\") &&\n+\ttest-dump-split-index .git/index | sed \"/^own/d\" >actual &&\n+\tcat >expect <\n---\n builtin/update-index.c | 11 ++++++++++-\n 1 file changed, 10 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex b75ea037dd..dc1fd0d44d 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1098,12 +1098,21 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (split_index > 0) {\n+\t\tif (git_config_get_split_index() == 0)\n+\t\t\twarning(_(\"core.splitIndex is set to false; \"\n+\t\t\t\t \"remove or change it, if you really want to \"\n+\t\t\t\t \"enable split index\"));\n \t\tif (the_index.split_index)\n \t\t\tthe_index.cache_changed |= SPLIT_INDEX_ORDERED;\n \t\telse\n \t\t\tadd_split_index(&the_index);\n-\t} else if (!split_index)\n+\t} else if (!split_index) {\n+\t\tif (git_config_get_split_index() == 1)\n+\t\t\twarning(_(\"core.splitIndex is set to true; \"\n+\t\t\t\t \"remove or change it, if you really want to \"\n+\t\t\t\t \"disable split index\"));\n \t\tremove_split_index(&the_index);\n+\t}\n \n \tswitch (untracked_cache) {\n \tcase UC_UNSPECIFIED:\n-- \n2.11.0.49.g2414764.dirty\n\n"},{"id":"307974","messageId":"20161217145547.11748-14-chriscool@tuxfamily.org","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"[PATCH v2 13/21] sha1_file: make check_and_freshen_file() non static","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T14:55:39Z","receivedAt":"2016-12-17T14:56:52Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This function will be used in a commit soon, so let's make\nit available globally.\n\nSigned-off-by: Christian Couder \n---\n cache.h | 3 +++\n sha1_file.c | 2 +-\n 2 files changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex e15b421b6f..f442f28189 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1170,6 +1170,9 @@ extern int has_pack_index(const unsigned char *sha1);\n \n extern void assert_sha1_type(const unsigned char *sha1, enum object_type expect);\n \n+/* Helper to check and \"touch\" a file */\n+extern int check_and_freshen_file(const char *fn, int freshen);\n+\n extern const signed char hexval_table[256];\n static inline unsigned int hexval(unsigned char c)\n {\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 9c86d1924a..7e5846d4f9 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -576,7 +576,7 @@ static int freshen_file(const char *fn)\n * either does not exist on disk, or has a stale mtime and may be subject to\n * pruning).\n */\n-static int check_and_freshen_file(const char *fn, int freshen)\n+int check_and_freshen_file(const char *fn, int freshen)\n {\n \tif (access(fn, F_OK))\n \t\treturn 0;\n-- \n2.11.0.49.g2414764.dirty\n\n"},{"id":"307975","messageId":"20161217145547.11748-19-chriscool@tuxfamily.org","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"[PATCH v2 18/21] read-cache: refactor read_index_from()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T14:55:44Z","receivedAt":"2016-12-17T14:56:56Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"It looks better and is simpler to review when we don't compute\nthe same things many times in the function.\n\nIt will also help make the following commit simpler.\n\nSigned-off-by: Christian Couder \n---\n read-cache.c | 14 ++++++++------\n 1 file changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 8e99a7c325..96e097e341 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1702,6 +1702,8 @@ int read_index_from(struct index_state *istate, const char *path)\n {\n \tstruct split_index *split_index;\n \tint ret;\n+\tchar *base_sha1_hex;\n+\tconst char *base_path;\n \n \t/* istate->initialized covers both .git/index and .git/sharedindex.xxx */\n \tif (istate->initialized)\n@@ -1719,15 +1721,15 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tdiscard_index(split_index->base);\n \telse\n \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n-\tret = do_read_index(split_index->base,\n-\t\t\t git_path(\"sharedindex.%s\",\n-\t\t\t\t sha1_to_hex(split_index->base_sha1)), 1);\n+\n+\tbase_sha1_hex = sha1_to_hex(split_index->base_sha1);\n+\tbase_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n+\tret = do_read_index(split_index->base, base_path, 1);\n \tif (hashcmp(split_index->base_sha1, split_index->base->sha1))\n \t\tdie(\"broken index, expect %s in %s, got %s\",\n-\t\t sha1_to_hex(split_index->base_sha1),\n-\t\t git_path(\"sharedindex.%s\",\n-\t\t\t sha1_to_hex(split_index->base_sha1)),\n+\t\t base_sha1_hex, base_path,\n \t\t sha1_to_hex(split_index->base->sha1));\n+\n \tmerge_base_index(istate);\n \tpost_read_index_from(istate);\n \treturn ret;\n-- \n2.11.0.49.g2414764.dirty\n\n"},{"id":"307976","messageId":"20161217145547.11748-16-chriscool@tuxfamily.org","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"[PATCH v2 15/21] config: add git_config_get_expiry() from gc.c","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T14:55:41Z","receivedAt":"2016-12-17T14:56:59Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This function will be used in a following commit to get the expiration\ntime of the shared index files from the config, and it is generic\nenough to be put in \"config.c\".\n\nSigned-off-by: Christian Couder \n---\n builtin/gc.c | 15 ++-------------\n cache.h | 3 +++\n config.c | 13 +++++++++++++\n 3 files changed, 18 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 069950d0b4..1e40d45aa2 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -62,17 +62,6 @@ static void report_pack_garbage(unsigned seen_bits, const char *path)\n \t\tstring_list_append(&pack_garbage, path);\n }\n \n-static void git_config_date_string(const char *key, const char **output)\n-{\n-\tif (git_config_get_string_const(key, output))\n-\t\treturn;\n-\tif (strcmp(*output, \"now\")) {\n-\t\tunsigned long now = approxidate(\"now\");\n-\t\tif (approxidate(*output) >= now)\n-\t\t\tgit_die_config(key, _(\"Invalid %s: '%s'\"), key, *output);\n-\t}\n-}\n-\n static void process_log_file(void)\n {\n \tstruct stat st;\n@@ -111,8 +100,8 @@ static void gc_config(void)\n \tgit_config_get_int(\"gc.auto\", &gc_auto_threshold);\n \tgit_config_get_int(\"gc.autopacklimit\", &gc_auto_pack_limit);\n \tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n-\tgit_config_date_string(\"gc.pruneexpire\", &prune_expire);\n-\tgit_config_date_string(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n+\tgit_config_get_expiry(\"gc.pruneexpire\", &prune_expire);\n+\tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n \tgit_config(git_default_config, NULL);\n }\n \ndiff --git a/cache.h b/cache.h\nindex f442f28189..279415afbd 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1827,6 +1827,9 @@ extern int git_config_get_untracked_cache(void);\n extern int git_config_get_split_index(void);\n extern int git_config_get_max_percent_split_change(void);\n \n+/* This dies if the configured or default date is in the future */\n+extern int git_config_get_expiry(const char *key, const char **output);\n+\n /*\n * This is a hack for test programs like test-dump-untracked-cache to\n * ensure that they do not modify the untracked cache when reading it.\ndiff --git a/config.c b/config.c\nindex 3e96c223f5..5c52cefd78 100644\n--- a/config.c\n+++ b/config.c\n@@ -1685,6 +1685,19 @@ int git_config_get_pathname(const char *key, const char **dest)\n \treturn ret;\n }\n \n+int git_config_get_expiry(const char *key, const char **output)\n+{\n+\tint ret = git_config_get_string_const(key, output);\n+\tif (ret)\n+\t\treturn ret;\n+\tif (strcmp(*output, \"now\")) {\n+\t\tunsigned long now = approxidate(\"now\");\n+\t\tif (approxidate(*output) >= now)\n+\t\t\tgit_die_config(key, _(\"Invalid %s: '%s'\"), key, *output);\n+\t}\n+\treturn ret;\n+}\n+\n int git_config_get_untracked_cache(void)\n {\n \tint val = -1;\n-- \n2.11.0.49.g2414764.dirty\n\n"},{"id":"307977","messageId":"20161217145547.11748-11-chriscool@tuxfamily.org","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"[PATCH v2 10/21] read-cache: regenerate shared index if necessary","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T14:55:36Z","receivedAt":"2016-12-17T14:57:00Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"When writing a new split-index and there is a big number of cache\nentries in the split-index compared to the shared index, it is a\ngood idea to regenerate the shared index.\n\nBy default when the ratio reaches 20%, we will push back all\nthe entries from the split-index into a new shared index file\ninstead of just creating a new split-index file.\n\nThe threshold can be configured using the\n\"splitIndex.maxPercentChange\" config variable.\n\nWe need to adjust the existing tests in t1700 by setting\n\"splitIndex.maxPercentChange\" to 100 at the beginning of t1700,\nas the existing tests are assuming that the shared index is\nregenerated only when `git update-index --split-index` is used.\n\nSigned-off-by: Christian Couder \n---\n read-cache.c | 32 ++++++++++++++++++++++++++++++++\n t/t1700-split-index.sh | 1 +\n 2 files changed, 33 insertions(+)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 79aae6bd20..280b01bd6c 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2223,6 +2223,36 @@ static int write_shared_index(struct index_state *istate,\n \treturn ret;\n }\n \n+static const int default_max_percent_split_change = 20;\n+\n+static int too_many_not_shared_entries(struct index_state *istate)\n+{\n+\tint i, not_shared = 0;\n+\tint max_split = git_config_get_max_percent_split_change();\n+\n+\tswitch (max_split) {\n+\tcase -1:\n+\t\t/* not or badly configured: use the default value */\n+\t\tmax_split = default_max_percent_split_change;\n+\t\tbreak;\n+\tcase 0:\n+\t\treturn 1; /* 0% means always write a new shared index */\n+\tcase 100:\n+\t\treturn 0; /* 100% means never write a new shared index */\n+\tdefault:\n+\t\t; /* do nothing: just use the configured value */\n+\t}\n+\n+\t/* Count not shared entries */\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = istate->cache[i];\n+\t\tif (!ce->index)\n+\t\t\tnot_shared++;\n+\t}\n+\n+\treturn istate->cache_nr * max_split < not_shared * 100;\n+}\n+\n int write_locked_index(struct index_state *istate, struct lock_file *lock,\n \t\t unsigned flags)\n {\n@@ -2240,6 +2270,8 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,\n \t\tif ((v & 15) < 6)\n \t\t\tistate->cache_changed |= SPLIT_INDEX_ORDERED;\n \t}\n+\tif (too_many_not_shared_entries(istate))\n+\t\tistate->cache_changed |= SPLIT_INDEX_ORDERED;\n \tif (istate->cache_changed & SPLIT_INDEX_ORDERED) {\n \t\tint ret = write_shared_index(istate, lock, flags);\n \t\tif (ret)\ndiff --git a/t/t1700-split-index.sh b/t/t1700-split-index.sh\nindex db8c39f446..507a1dd1ad 100755\n--- a/t/t1700-split-index.sh\n+++ b/t/t1700-split-index.sh\n@@ -8,6 +8,7 @@ test_description='split index mode tests'\n sane_unset GIT_TEST_SPLIT_INDEX\n \n test_expect_success 'enable split index' '\n+\tgit config splitIndex.maxPercentChange 100 &&\n \tgit update-index --split-index &&\n \ttest-dump-split-index .git/index >actual &&\n \tindexversion=$(test-index-version <.git/index) &&\n-- \n2.11.0.49.g2414764.dirty\n\n"},{"id":"307978","messageId":"20161217145547.11748-5-chriscool@tuxfamily.org","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"[PATCH v2 04/21] read-cache: add and then use tweak_split_index()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T14:55:30Z","receivedAt":"2016-12-17T14:57:03Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"This will make us use the split-index feature or not depending\non the value of the \"core.splitIndex\" config variable.\n\nSigned-off-by: Christian Couder \n---\n read-cache.c | 17 +++++++++++++++++\n 1 file changed, 17 insertions(+)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex db5d910642..79aae6bd20 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1569,10 +1569,27 @@ static void tweak_untracked_cache(struct index_state *istate)\n \t}\n }\n \n+static void tweak_split_index(struct index_state *istate)\n+{\n+\tswitch (git_config_get_split_index()) {\n+\tcase -1: /* unset: do nothing */\n+\t\tbreak;\n+\tcase 0: /* false */\n+\t\tremove_split_index(istate);\n+\t\tbreak;\n+\tcase 1: /* true */\n+\t\tadd_split_index(istate);\n+\t\tbreak;\n+\tdefault: /* unknown value: do nothing */\n+\t\tbreak;\n+\t}\n+}\n+\n static void post_read_index_from(struct index_state *istate)\n {\n \tcheck_ce_order(istate);\n \ttweak_untracked_cache(istate);\n+\ttweak_split_index(istate);\n }\n \n /* remember to discard_cache() before reading a different cache! */\n-- \n2.11.0.49.g2414764.dirty\n\n"},{"id":"307979","messageId":"20161217145547.11748-4-chriscool@tuxfamily.org","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"[PATCH v2 03/21] split-index: add {add,remove}_split_index() functions","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T14:55:29Z","receivedAt":"2016-12-17T14:57:04Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Also use the functions in cmd_update_index() in\nbuiltin/update-index.c.\n\nThese functions will be used in a following commit to tweak\nour use of the split-index feature depending on the setting\nof a configuration variable.\n\nSigned-off-by: Christian Couder \n---\n builtin/update-index.c | 18 ++++++------------\n split-index.c | 22 ++++++++++++++++++++++\n split-index.h | 2 ++\n 3 files changed, 30 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex f3f07e7f1c..b75ea037dd 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1098,18 +1098,12 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (split_index > 0) {\n-\t\tinit_split_index(&the_index);\n-\t\tthe_index.cache_changed |= SPLIT_INDEX_ORDERED;\n-\t} else if (!split_index && the_index.split_index) {\n-\t\t/*\n-\t\t * can't discard_split_index(&the_index); because that\n-\t\t * will destroy split_index->base->cache[], which may\n-\t\t * be shared with the_index.cache[]. So yeah we're\n-\t\t * leaking a bit here.\n-\t\t */\n-\t\tthe_index.split_index = NULL;\n-\t\tthe_index.cache_changed |= SOMETHING_CHANGED;\n-\t}\n+\t\tif (the_index.split_index)\n+\t\t\tthe_index.cache_changed |= SPLIT_INDEX_ORDERED;\n+\t\telse\n+\t\t\tadd_split_index(&the_index);\n+\t} else if (!split_index)\n+\t\tremove_split_index(&the_index);\n \n \tswitch (untracked_cache) {\n \tcase UC_UNSPECIFIED:\ndiff --git a/split-index.c b/split-index.c\nindex 615f4cac05..f519e60f87 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -317,3 +317,25 @@ void replace_index_entry_in_base(struct index_state *istate,\n \t\tistate->split_index->base->cache[new->index - 1] = new;\n \t}\n }\n+\n+void add_split_index(struct index_state *istate)\n+{\n+\tif (!istate->split_index) {\n+\t\tinit_split_index(istate);\n+\t\tistate->cache_changed |= SPLIT_INDEX_ORDERED;\n+\t}\n+}\n+\n+void remove_split_index(struct index_state *istate)\n+{\n+\tif (istate->split_index) {\n+\t\t/*\n+\t\t * can't discard_split_index(&the_index); because that\n+\t\t * will destroy split_index->base->cache[], which may\n+\t\t * be shared with the_index.cache[]. So yeah we're\n+\t\t * leaking a bit here.\n+\t\t */\n+\t\tistate->split_index = NULL;\n+\t\tistate->cache_changed |= SOMETHING_CHANGED;\n+\t}\n+}\ndiff --git a/split-index.h b/split-index.h\nindex c1324f521a..df91c1bda8 100644\n--- a/split-index.h\n+++ b/split-index.h\n@@ -31,5 +31,7 @@ void merge_base_index(struct index_state *istate);\n void prepare_to_write_split_index(struct index_state *istate);\n void finish_writing_split_index(struct index_state *istate);\n void discard_split_index(struct index_state *istate);\n+void add_split_index(struct index_state *istate);\n+void remove_split_index(struct index_state *istate);\n \n #endif\n-- \n2.11.0.49.g2414764.dirty\n\n"},{"id":"307980","messageId":"20161217145547.11748-8-chriscool@tuxfamily.org","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"[PATCH v2 07/21] Documentation/config: add information for core.splitIndex","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T14:55:33Z","receivedAt":"2016-12-17T14:57:05Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Signed-off-by: Christian Couder \n---\n Documentation/config.txt | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex a0ab66aae7..dc44d8a417 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -331,6 +331,10 @@ core.trustctime::\n \tcrawlers and some backup systems).\n \tSee linkgit:git-update-index[1]. True by default.\n \n+core.splitIndex::\n+\tIf true, the split-index feature of the index will be used.\n+\tSee linkgit:git-update-index[1]. False by default.\n+\n core.untrackedCache::\n \tDetermines what to do about the untracked cache feature of the\n \tindex. It will be kept, if this variable is unset or set to\n-- \n2.11.0.49.g2414764.dirty\n\n"},{"id":"307981","messageId":"CAP8UFD0Z7UquU0qiGq9y5AXtR-BiMEme9i=MnKRRzD_zyDpdzA@mail.gmail.com","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"Re: [PATCH v2 00/21] Add configuration options for split-index","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-17T15:35:45Z","receivedAt":"2016-12-17T15:35:54Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"> The previous versions were:\n>\n> RFC: https://github.com/chriscool/git/commits/config-split-index7\n> v1: https://github.com/chriscool/git/commits/config-split-index72\n\nThe diff since v1 is:\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 8fbef25cb1..52a3cac4ff 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2782,9 +2782,9 @@ splitIndex.sharedIndexExpire::\n index file is created. The value \"now\" expires all entries\n immediately, and \"never\" suppresses expiration altogether.\n The default value is \"one.week.ago\".\n- Note that each time a new split-index file is created, the\n- mtime of the related shared index file is updated to the\n- current time.\n+ Note that each time a split index based on a shared index file\n+ is either created or read from, the mtime of the shared index\n+ file is updated to the current time.\n See linkgit:git-update-index[1].\n\n status.relativePaths::\ndiff --git a/Documentation/git-update-index.txt\nb/Documentation/git-update-index.txt\nindex 635d1574b2..46c953b2f2 100644\n--- a/Documentation/git-update-index.txt\n+++ b/Documentation/git-update-index.txt\n@@ -407,10 +407,14 @@ specified by the splitIndex.maxPercentChange\nconfig variable (see\n linkgit:git-config[1]).\n\n Each time a new shared index file is created, the old shared index\n-files are deleted if they are older than what is specified by the\n-splitIndex.sharedIndexExpire config variable (see\n+files are deleted if their mtime is older than what is specified by\n+the splitIndex.sharedIndexExpire config variable (see\n linkgit:git-config[1]).\n\n+To avoid deleting a shared index file that is still used, its mtime is\n+updated to the current time everytime a new split index based on the\n+shared index file is either created or read from.\n+\n Untracked cache\n ---------------\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex c1e9602892..1e40d45aa2 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -100,8 +100,8 @@ static void gc_config(void)\n git_config_get_int(\"gc.auto\", &gc_auto_threshold);\n git_config_get_int(\"gc.autopacklimit\", &gc_auto_pack_limit);\n git_config_get_bool(\"gc.autodetach\", &detach_auto);\n- git_config_get_date_string(\"gc.pruneexpire\", &prune_expire);\n- git_config_get_date_string(\"gc.worktreepruneexpire\",\n&prune_worktrees_expire);\n+ git_config_get_expiry(\"gc.pruneexpire\", &prune_expire);\n+ git_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n git_config(git_default_config, NULL);\n }\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex a14dbf2612..dc1fd0d44d 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -1099,18 +1099,18 @@ int cmd_update_index(int argc, const char\n**argv, const char *prefix)\n\n if (split_index > 0) {\n if (git_config_get_split_index() == 0)\n- warning(\"core.splitIndex is set to false; \"\n- \"remove or change it, if you really want to \"\n- \"enable split index\");\n+ warning(_(\"core.splitIndex is set to false; \"\n+ \"remove or change it, if you really want to \"\n+ \"enable split index\"));\n if (the_index.split_index)\n the_index.cache_changed |= SPLIT_INDEX_ORDERED;\n else\n add_split_index(&the_index);\n } else if (!split_index) {\n if (git_config_get_split_index() == 1)\n- warning(\"core.splitIndex is set to true; \"\n- \"remove or change it, if you really want to \"\n- \"disable split index\");\n+ warning(_(\"core.splitIndex is set to true; \"\n+ \"remove or change it, if you really want to \"\n+ \"disable split index\"));\n remove_split_index(&the_index);\n }\n\ndiff --git a/cache.h b/cache.h\nindex 8e26aaf05e..279415afbd 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1823,11 +1823,13 @@ extern int git_config_get_bool(const char\n*key, int *dest);\n extern int git_config_get_bool_or_int(const char *key, int *is_bool,\nint *dest);\n extern int git_config_get_maybe_bool(const char *key, int *dest);\n extern int git_config_get_pathname(const char *key, const char **dest);\n-extern int git_config_get_date_string(const char *key, const char **output);\n extern int git_config_get_untracked_cache(void);\n extern int git_config_get_split_index(void);\n extern int git_config_get_max_percent_split_change(void);\n\n+/* This dies if the configured or default date is in the future */\n+extern int git_config_get_expiry(const char *key, const char **output);\n+\n /*\n * This is a hack for test programs like test-dump-untracked-cache to\n * ensure that they do not modify the untracked cache when reading it.\ndiff --git a/config.c b/config.c\nindex f88c61bb30..5c52cefd78 100644\n--- a/config.c\n+++ b/config.c\n@@ -1685,7 +1685,7 @@ int git_config_get_pathname(const char *key,\nconst char **dest)\n return ret;\n }\n\n-int git_config_get_date_string(const char *key, const char **output)\n+int git_config_get_expiry(const char *key, const char **output)\n {\n int ret = git_config_get_string_const(key, output);\n if (ret)\n@@ -1714,8 +1714,8 @@ int git_config_get_untracked_cache(void)\n if (!strcasecmp(v, \"keep\"))\n return -1;\n\n- error(\"unknown core.untrackedCache value '%s'; \"\n- \"using 'keep' default value\", v);\n+ error(_(\"unknown core.untrackedCache value '%s'; \"\n+ \"using 'keep' default value\"), v);\n return -1;\n }\n\n@@ -1740,9 +1740,8 @@ int git_config_get_max_percent_split_change(void)\n if (0 <= val && val <= 100)\n return val;\n\n- error(\"splitindex.maxpercentchange value '%d' \"\n- \"should be between 0 and 100\", val);\n- return -1;\n+ return error(_(\"splitIndex.maxPercentChange value '%d' \"\n+ \"should be between 0 and 100\"), val);\n }\n\n return -1; /* default value */\ndiff --git a/read-cache.c b/read-cache.c\nindex 9727efcb5b..35377f0a3e 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1685,10 +1685,25 @@ int do_read_index(struct index_state *istate,\nconst char *path, int must_exist)\n die(\"index file corrupt\");\n }\n\n+/*\n+ * Signal that the shared index is used by updating its mtime.\n+ *\n+ * This way, shared index can be removed if they have not been used\n+ * for some time. It's ok to fail to update the mtime if we are on a\n+ * read only file system.\n+ */\n+void freshen_shared_index(char *base_sha1_hex)\n+{\n+ const char *shared_index = git_path(\"sharedindex.%s\", base_sha1_hex);\n+ check_and_freshen_file(shared_index, 1);\n+}\n+\n int read_index_from(struct index_state *istate, const char *path)\n {\n struct split_index *split_index;\n int ret;\n+ char *base_sha1_hex;\n+ const char *base_path;\n\n /* istate->initialized covers both .git/index and .git/sharedindex.xxx */\n if (istate->initialized)\n@@ -1706,15 +1721,16 @@ int read_index_from(struct index_state\n*istate, const char *path)\n discard_index(split_index->base);\n else\n split_index->base = xcalloc(1, sizeof(*split_index->base));\n- ret = do_read_index(split_index->base,\n- git_path(\"sharedindex.%s\",\n- sha1_to_hex(split_index->base_sha1)), 1);\n+\n+ base_sha1_hex = sha1_to_hex(split_index->base_sha1);\n+ base_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n+ ret = do_read_index(split_index->base, base_path, 1);\n if (hashcmp(split_index->base_sha1, split_index->base->sha1))\n die(\"broken index, expect %s in %s, got %s\",\n- sha1_to_hex(split_index->base_sha1),\n- git_path(\"sharedindex.%s\",\n- sha1_to_hex(split_index->base_sha1)),\n+ base_sha1_hex, base_path,\n sha1_to_hex(split_index->base->sha1));\n+\n+ freshen_shared_index(base_sha1_hex);\n merge_base_index(istate);\n post_read_index_from(istate);\n return ret;\n@@ -2205,8 +2221,8 @@ static unsigned long get_shared_index_expire_date(void)\n static int shared_index_expire_date_prepared;\n\n if (!shared_index_expire_date_prepared) {\n- git_config_get_date_string(\"splitindex.sharedindexexpire\",\n- &shared_index_expire);\n+ git_config_get_expiry(\"splitindex.sharedindexexpire\",\n+ &shared_index_expire);\n shared_index_expire_date = approxidate(shared_index_expire);\n shared_index_expire_date_prepared = 1;\n }\n@@ -2225,22 +2241,20 @@ static int can_delete_shared_index(const char\n*shared_sha1_hex)\n if (!expiration)\n return 0;\n if (stat(shared_index, &st))\n- return error_errno(\"could not stat '%s\", shared_index);\n+ return error_errno(_(\"could not stat '%s\"), shared_index);\n if (st.st_mtime > expiration)\n return 0;\n\n return 1;\n }\n\n-static void clean_shared_index_files(const char *current_hex)\n+static int clean_shared_index_files(const char *current_hex)\n {\n struct dirent *de;\n DIR *dir = opendir(get_git_dir());\n\n- if (!dir) {\n- error_errno(\"unable to open git dir: %s\", get_git_dir());\n- return;\n- }\n+ if (!dir)\n+ return error_errno(_(\"unable to open git dir: %s\"), get_git_dir());\n\n while ((de = readdir(dir)) != NULL) {\n const char *sha1_hex;\n@@ -2250,9 +2264,11 @@ static void clean_shared_index_files(const char\n*current_hex)\n continue;\n if (can_delete_shared_index(sha1_hex) > 0 &&\n unlink(git_path(\"%s\", de->d_name)))\n- error_errno(\"unable to unlink: %s\", git_path(\"%s\", de->d_name));\n+ error_errno(_(\"unable to unlink: %s\"), git_path(\"%s\", de->d_name));\n }\n closedir(dir);\n+\n+ return 0;\n }\n\n static struct tempfile temporary_sharedindex;\n@@ -2286,7 +2302,7 @@ static int write_shared_index(struct index_state *istate,\n\n static const int default_max_percent_split_change = 20;\n\n-int too_many_not_shared_entries(struct index_state *istate)\n+static int too_many_not_shared_entries(struct index_state *istate)\n {\n int i, not_shared = 0;\n int max_split = git_config_get_max_percent_split_change();\n@@ -2331,17 +2347,14 @@ int write_locked_index(struct index_state\n*istate, struct lock_file *lock,\n if ((v & 15) < 6)\n istate->cache_changed |= SPLIT_INDEX_ORDERED;\n }\n- if (istate->cache_changed & SPLIT_INDEX_ORDERED ||\n- too_many_not_shared_entries(istate)) {\n+ if (too_many_not_shared_entries(istate))\n+ istate->cache_changed |= SPLIT_INDEX_ORDERED;\n+ if (istate->cache_changed & SPLIT_INDEX_ORDERED) {\n int ret = write_shared_index(istate, lock, flags);\n if (ret)\n return ret;\n } else {\n- /* Signal that the shared index is used */\n- const char *shared_index = git_path(\"sharedindex.%s\",\n- sha1_to_hex(si->base_sha1));\n- if (!check_and_freshen_file(shared_index, 1))\n- warning(\"could not freshen '%s'\", shared_index);\n+ freshen_shared_index(sha1_to_hex(si->base_sha1));\n }\n\n return write_split_index(istate, lock, flags);\ndiff --git a/t/t1700-split-index.sh b/t/t1700-split-index.sh\nindex f448fc13cd..800f84a593 100755\n--- a/t/t1700-split-index.sh\n+++ b/t/t1700-split-index.sh\n@@ -313,17 +313,17 @@ EOF\n test_expect_success 'shared index files expire after 7 days by default' '\n : >ten &&\n git update-index --add ten &&\n- test $(ls .git/sharedindex.* | wc -l) -gt 1 &&\n+ test $(ls .git/sharedindex.* | wc -l) -gt 2 &&\n just_under_7_days_ago=$((1-7*86400)) &&\n test-chmtime =$just_under_7_days_ago .git/sharedindex.* &&\n : >eleven &&\n git update-index --add eleven &&\n- test $(ls .git/sharedindex.* | wc -l) -gt 1 &&\n+ test $(ls .git/sharedindex.* | wc -l) -gt 2 &&\n just_over_7_days_ago=$((-1-7*86400)) &&\n test-chmtime =$just_over_7_days_ago .git/sharedindex.* &&\n : >twelve &&\n git update-index --add twelve &&\n- test $(ls .git/sharedindex.* | wc -l) = 1\n+ test $(ls .git/sharedindex.* | wc -l) -le 2\n '\n\n test_expect_success 'check splitIndex.sharedIndexExpire set to 8 days' '\n@@ -331,12 +331,12 @@ test_expect_success 'check\nsplitIndex.sharedIndexExpire set to 8 days' '\n test-chmtime =$just_over_7_days_ago .git/sharedindex.* &&\n : >thirteen &&\n git update-index --add thirteen &&\n- test $(ls .git/sharedindex.* | wc -l) -gt 1 &&\n+ test $(ls .git/sharedindex.* | wc -l) -gt 2 &&\n just_over_8_days_ago=$((-1-8*86400)) &&\n test-chmtime =$just_over_8_days_ago .git/sharedindex.* &&\n : >fourteen &&\n git update-index --add fourteen &&\n- test $(ls .git/sharedindex.* | wc -l) = 1\n+ test $(ls .git/sharedindex.* | wc -l) -le 2\n '\n\n test_expect_success 'check splitIndex.sharedIndexExpire set to\n\"never\" and \"now\"' '\n@@ -345,13 +345,13 @@ test_expect_success 'check\nsplitIndex.sharedIndexExpire set to \"never\" and \"now\"\n test-chmtime =$just_10_years_ago .git/sharedindex.* &&\n : >fifteen &&\n git update-index --add fifteen &&\n- test $(ls .git/sharedindex.* | wc -l) -gt 1 &&\n+ test $(ls .git/sharedindex.* | wc -l) -gt 2 &&\n git config splitIndex.sharedIndexExpire now &&\n just_1_second_ago=-1 &&\n test-chmtime =$just_1_second_ago .git/sharedindex.* &&\n : >sixteen &&\n git update-index --add sixteen &&\n- test $(ls .git/sharedindex.* | wc -l) = 1\n+ test $(ls .git/sharedindex.* | wc -l) -le 2\n '\n\n test_done\n"},{"id":"308011","messageId":"20161219111442.GA24125@ash","threadId":"44726","inReplyTo":"20161217145547.11748-3-chriscool@tuxfamily.org","subject":"Re: [PATCH v2 02/21] config: add git_config_get_split_index()","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-12-19T11:14:42Z","receivedAt":"2016-12-19T11:15:50Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Dec 17, 2016 at 03:55:28PM +0100, Christian Couder wrote:\n> diff --git a/config.c b/config.c\n> index 2eaf8ad77a..c1343bbb3e 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -1709,6 +1709,16 @@ int git_config_get_untracked_cache(void)\n> \treturn -1; /* default value */\n> }\n> \n> +int git_config_get_split_index(void)\n> +{\n> +\tint val = -1;\n\nIs it redundant to set default value here because it's not used\nanywhere? The \"return val;\" will always have the new value from\ngit_config_. And you don't use \"val\" in error case.\n\n> +\n> +\tif (!git_config_get_maybe_bool(\"core.splitindex\", &val))\n> +\t\treturn val;\n> +\n> +\treturn -1; /* default value */\n> +}\n> +\n> NORETURN\n> void git_die_config_linenr(const char *key, const char *filename, int linenr)\n> {\n> -- \n> 2.11.0.49.g2414764.dirty\n> \n"},{"id":"308012","messageId":"20161219111843.GB24125@ash","threadId":"44726","inReplyTo":"20161217145547.11748-5-chriscool@tuxfamily.org","subject":"Re: [PATCH v2 04/21] read-cache: add and then use tweak_split_index()","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-12-19T11:18:43Z","receivedAt":"2016-12-19T11:19:48Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Dec 17, 2016 at 03:55:30PM +0100, Christian Couder wrote:\n> +static void tweak_split_index(struct index_state *istate)\n> +{\n> +\tswitch (git_config_get_split_index()) {\n> +\tcase -1: /* unset: do nothing */\n> +\t\tbreak;\n> +\tcase 0: /* false */\n> +\t\tremove_split_index(istate);\n> +\t\tbreak;\n> +\tcase 1: /* true */\n> +\t\tadd_split_index(istate);\n> +\t\tbreak;\n> +\tdefault: /* unknown value: do nothing */\n\nProbably should die(\"BUG:\") here since it looks to me like\ngit_config_maybe_bool() (inside git_config_get_split_index) in this\ncase has been updated to return more values than we can handle. And we\nneed to know about that so we can handle it properly.\n\n> +\t\tbreak;\n> +\t}\n> +}\n"},{"id":"308016","messageId":"20161219114807.GC24125@ash","threadId":"44726","inReplyTo":"20161217145547.11748-11-chriscool@tuxfamily.org","subject":"Re: [PATCH v2 10/21] read-cache: regenerate shared index if necessary","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-12-19T11:48:07Z","receivedAt":"2016-12-19T11:49:15Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Dec 17, 2016 at 03:55:36PM +0100, Christian Couder wrote:\n> +static const int default_max_percent_split_change = 20;\n> +\n> +static int too_many_not_shared_entries(struct index_state *istate)\n> +{\n> +\tint i, not_shared = 0;\n> +\tint max_split = git_config_get_max_percent_split_change();\n> +\n> +\tswitch (max_split) {\n> +\tcase -1:\n> +\t\t/* not or badly configured: use the default value */\n> +\t\tmax_split = default_max_percent_split_change;\n> +\t\tbreak;\n> +\tcase 0:\n> +\t\treturn 1; /* 0% means always write a new shared index */\n> +\tcase 100:\n> +\t\treturn 0; /* 100% means never write a new shared index */\n\nI wonder if we really need to special case these here. If I read it\ncorrectly, the expression at the end of this function will return 1\nwhen max_split is 0, and 0 when max_split is 100 (not counting the\ncase when cache_nr is zero).\n\nPerhaps it's good for documentation purpose. Though I find it hard to\nsee a use case for max_split == 0. Always creating a new shared index\nsounds crazy.\n\n> +\tdefault:\n> +\t\t; /* do nothing: just use the configured value */\n> +\t}\n> +\n> +\t/* Count not shared entries */\n> +\tfor (i = 0; i < istate->cache_nr; i++) {\n> +\t\tstruct cache_entry *ce = istate->cache[i];\n> +\t\tif (!ce->index)\n> +\t\t\tnot_shared++;\n> +\t}\n> +\n> +\treturn istate->cache_nr * max_split < not_shared * 100;\n> +}\n"},{"id":"308017","messageId":"20161219115812.GD24125@ash","threadId":"44726","inReplyTo":"20161217145547.11748-17-chriscool@tuxfamily.org","subject":"Re: [PATCH v2 16/21] read-cache: unlink old sharedindex files","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-12-19T11:58:13Z","receivedAt":"2016-12-19T11:59:34Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Dec 17, 2016 at 03:55:42PM +0100, Christian Couder wrote:\n> +static int clean_shared_index_files(const char *current_hex)\n> +{\n> +\tstruct dirent *de;\n> +\tDIR *dir = opendir(get_git_dir());\n> +\n> +\tif (!dir)\n> +\t\treturn error_errno(_(\"unable to open git dir: %s\"), get_git_dir());\n> +\n> +\twhile ((de = readdir(dir)) != NULL) {\n> +\t\tconst char *sha1_hex;\n> +\t\tif (!skip_prefix(de->d_name, \"sharedindex.\", &sha1_hex))\n> +\t\t\tcontinue;\n> +\t\tif (!strcmp(sha1_hex, current_hex))\n\nfspathcmp since we're dealing fs paths?\n\nIn theory we should be ok using strcmp, even on Windows because case\nis preserved, I think. It's only a problem when we write path 'abc'\ndown and read 'ABC' back.\n\nOh well.. your call because if you go with fspathcmp, skip_prefix can't\nbe used either. A lot more changes for a very rare case.\n\n> +\t\t\tcontinue;\n> +\t\tif (can_delete_shared_index(sha1_hex) > 0 &&\n\nProbably shorter to pass full d->name here so you don't have to do\nanother git_path() in can_delete_\n\n> +\t\t unlink(git_path(\"%s\", de->d_name)))\n> +\t\t\terror_errno(_(\"unable to unlink: %s\"), git_path(\"%s\", de->d_name));\n> +\t}\n> +\tclosedir(dir);\n> +\n> +\treturn 0;\n> +}\n"},{"id":"308018","messageId":"20161219120246.GE24125@ash","threadId":"44726","inReplyTo":"20161217145547.11748-1-chriscool@tuxfamily.org","subject":"Re: [PATCH v2 00/21] Add configuration options for split-index","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-12-19T12:02:46Z","receivedAt":"2016-12-19T12:03:47Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Dec 17, 2016 at 03:55:26PM +0100, Christian Couder wrote:\n> Goal\n> ~~~~\n> \n> We want to make it possible to use the split-index feature\n> automatically by just setting a new \"core.splitIndex\" configuration\n> variable to true.\n> \n> This can be valuable as split-index can help significantly speed up\n> `git rebase` especially along with the work to libify `git apply`\n> that has been merged to master\n> (see https://github.com/git/git/commit/81358dc238372793b1590efa149cc1581d1fbd98)\n> and is now in v2.11.\n\nI've read through the series (*) and I think it looks good, just a few\nminor comments here and there.\n\n(*) guiltily admit that I only skimmed through tests, not giving them\n as much attention as I should have\n--\nDuy\n"},{"id":"308095","messageId":"xmqq7f6voclq.fsf@gitster.mtv.corp.google.com","threadId":"44726","inReplyTo":"20161219120246.GE24125@ash","subject":"Re: [PATCH v2 00/21] Add configuration options for split-index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-12-19T21:03:45Z","receivedAt":"2016-12-19T21:04:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> I've read through the series (*) and I think it looks good, just a few\n> minor comments here and there.\n>\n> (*) guiltily admit that I only skimmed through tests, not giving them\n> as much attention as I should have\n\nOK. I'd still want to see them get reviewed, though. Perhaps I'll\ndo so myself once I run out of things to do, but hopefully somebody\nelse gets there first ;-)\n\nThanks.\n"},{"id":"308332","messageId":"CAP8UFD0MWm_VNC=8Cyz4UgD1-DWM3D1JUY2YPh=9AwnXJRjufA@mail.gmail.com","threadId":"44726","inReplyTo":"20161219111442.GA24125@ash","subject":"Re: [PATCH v2 02/21] config: add git_config_get_split_index()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-26T08:15:08Z","receivedAt":"2016-12-26T08:15:18Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Dec 19, 2016 at 12:14 PM, Duy Nguyen wrote:\n> On Sat, Dec 17, 2016 at 03:55:28PM +0100, Christian Couder wrote:\n>> diff --git a/config.c b/config.c\n>> index 2eaf8ad77a..c1343bbb3e 100644\n>> --- a/config.c\n>> +++ b/config.c\n>> @@ -1709,6 +1709,16 @@ int git_config_get_untracked_cache(void)\n>> return -1; /* default value */\n>> }\n>>\n>> +int git_config_get_split_index(void)\n>> +{\n>> + int val = -1;\n>\n> Is it redundant to set default value here because it's not used\n> anywhere? The \"return val;\" will always have the new value from\n> git_config_. And you don't use \"val\" in error case.\n\nYeah, it is redundant, so I will not set the default value here in the\nnext version.\n"},{"id":"308333","messageId":"CAP8UFD0KwbQo4QgC9cevcdHWBwedy-qCUXcDqaeissv4U=5RyQ@mail.gmail.com","threadId":"44726","inReplyTo":"20161219111843.GB24125@ash","subject":"Re: [PATCH v2 04/21] read-cache: add and then use tweak_split_index()","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-26T08:20:25Z","receivedAt":"2016-12-26T08:20:34Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Dec 19, 2016 at 12:18 PM, Duy Nguyen wrote:\n> On Sat, Dec 17, 2016 at 03:55:30PM +0100, Christian Couder wrote:\n>> +static void tweak_split_index(struct index_state *istate)\n>> +{\n>> + switch (git_config_get_split_index()) {\n>> + case -1: /* unset: do nothing */\n>> + break;\n>> + case 0: /* false */\n>> + remove_split_index(istate);\n>> + break;\n>> + case 1: /* true */\n>> + add_split_index(istate);\n>> + break;\n>> + default: /* unknown value: do nothing */\n>\n> Probably should die(\"BUG:\") here since it looks to me like\n> git_config_maybe_bool() (inside git_config_get_split_index) in this\n> case has been updated to return more values than we can handle. And we\n> need to know about that so we can handle it properly.\n\nIf we later add another value to the possible ones, like for example\n\"auto\", we might not want old versions of Git to fail with a \"BUG:...\"\nmessage when they read config files tweaked for newer Git versions, so\nI don't think we should die() here.\n\nA warning would be ok, but given that in tweak_untracked_cache() we\ndecided to do nothing, I am thinking that it would be more consistent\nto just do nothing here too.\n"},{"id":"308334","messageId":"CAP8UFD0FmvqiY4ACSbThHBvVPmLySifBWjsgGeA=14jEfnkfAw@mail.gmail.com","threadId":"44726","inReplyTo":"20161219114807.GC24125@ash","subject":"Re: [PATCH v2 10/21] read-cache: regenerate shared index if necessary","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-26T08:33:45Z","receivedAt":"2016-12-26T08:33:52Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Dec 19, 2016 at 12:48 PM, Duy Nguyen wrote:\n> On Sat, Dec 17, 2016 at 03:55:36PM +0100, Christian Couder wrote:\n>> +static const int default_max_percent_split_change = 20;\n>> +\n>> +static int too_many_not_shared_entries(struct index_state *istate)\n>> +{\n>> + int i, not_shared = 0;\n>> + int max_split = git_config_get_max_percent_split_change();\n>> +\n>> + switch (max_split) {\n>> + case -1:\n>> + /* not or badly configured: use the default value */\n>> + max_split = default_max_percent_split_change;\n>> + break;\n>> + case 0:\n>> + return 1; /* 0% means always write a new shared index */\n>> + case 100:\n>> + return 0; /* 100% means never write a new shared index */\n>\n> I wonder if we really need to special case these here. If I read it\n> correctly, the expression at the end of this function will return 1\n> when max_split is 0, and 0 when max_split is 100 (not counting the\n> case when cache_nr is zero).\n\nIt's better for performance if we can avoid computing the number of\nunshared entries, which we can in case of 0 or 100.\n\n> Perhaps it's good for documentation purpose.\n\nYeah, I think it's also good for documentation purpose.\n\n> Though I find it hard to\n> see a use case for max_split == 0. Always creating a new shared index\n> sounds crazy.\n\nYeah, but perhaps to test shared index writing performance people\nmight want to use it. And I don't see any good way or any good reason\nto disallow it.\n"},{"id":"308335","messageId":"CAP8UFD2hE5nLXKpSEPaS_1XYqQw+Wn=nMT4paV+FtGFrHzHsFw@mail.gmail.com","threadId":"44726","inReplyTo":"20161219115812.GD24125@ash","subject":"Re: [PATCH v2 16/21] read-cache: unlink old sharedindex files","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-26T09:54:29Z","receivedAt":"2016-12-26T09:54:36Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Dec 19, 2016 at 12:58 PM, Duy Nguyen wrote:\n> On Sat, Dec 17, 2016 at 03:55:42PM +0100, Christian Couder wrote:\n>> +static int clean_shared_index_files(const char *current_hex)\n>> +{\n>> + struct dirent *de;\n>> + DIR *dir = opendir(get_git_dir());\n>> +\n>> + if (!dir)\n>> + return error_errno(_(\"unable to open git dir: %s\"), get_git_dir());\n>> +\n>> + while ((de = readdir(dir)) != NULL) {\n>> + const char *sha1_hex;\n>> + if (!skip_prefix(de->d_name, \"sharedindex.\", &sha1_hex))\n>> + continue;\n>> + if (!strcmp(sha1_hex, current_hex))\n>\n> fspathcmp since we're dealing fs paths?\n>\n> In theory we should be ok using strcmp, even on Windows because case\n> is preserved, I think. It's only a problem when we write path 'abc'\n> down and read 'ABC' back.\n>\n> Oh well.. your call because if you go with fspathcmp, skip_prefix can't\n> be used either. A lot more changes for a very rare case.\n\nI'd rather keep using skip_prefix() and strcmp().\n\nI could find no place in the code where fspathcmp() is used to compare\ndir entries, but a few where other *cmp() functions are used:\n\n$ git grep '>d_name' | grep cmp\nbuiltin/repack.c: if (strncmp(e->d_name, buf.buf +\ndirlen, prefixlen))\ndir.c: if (is_dot_or_dotdot(de->d_name) || !strcmp(de->d_name, \".git\"))\nsha1_name.c: if (memcmp(de->d_name, ds->hex_pfx +\n2, ds->len - 2))\n\nand also a few where skip_prefix() is used:\n\n$ git grep -n '>d_name' | grep prefix\nbuiltin/repack.c:60: if (strncmp(e->d_name, buf.buf +\ndirlen, prefixlen))\nhelp.c:147: if (!skip_prefix(de->d_name, prefix, &ent))\n\nso I think it is ok to use skip_prefix() and strcmp().\n\n>> + continue;\n>> + if (can_delete_shared_index(sha1_hex) > 0 &&\n>\n> Probably shorter to pass full d->name here so you don't have to do\n> another git_path() in can_delete_\n\nYeah, you are right, I will do that.\n"},{"id":"308359","messageId":"CAP8UFD0DSB2o24EX4GSY38M1cfWB3b6ZkeniVdVP4y+t=xZUcA@mail.gmail.com","threadId":"44726","inReplyTo":"20161219120246.GE24125@ash","subject":"Re: [PATCH v2 00/21] Add configuration options for split-index","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2016-12-26T10:32:35Z","receivedAt":"2016-12-26T10:32:41Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Dec 19, 2016 at 1:02 PM, Duy Nguyen wrote:\n> On Sat, Dec 17, 2016 at 03:55:26PM +0100, Christian Couder wrote:\n>> Goal\n>> ~~~~\n>>\n>> We want to make it possible to use the split-index feature\n>> automatically by just setting a new \"core.splitIndex\" configuration\n>> variable to true.\n>>\n>> This can be valuable as split-index can help significantly speed up\n>> `git rebase` especially along with the work to libify `git apply`\n>> that has been merged to master\n>> (see https://github.com/git/git/commit/81358dc238372793b1590efa149cc1581d1fbd98)\n>> and is now in v2.11.\n>\n> I've read through the series (*) and I think it looks good, just a few\n> minor comments here and there.\n\nThanks for your review.\n\nI think I addressed all the minor points left in the v3 and the emails\nI just sent.\n\n> (*) guiltily admit that I only skimmed through tests, not giving them\n> as much attention as I should have\n"}]}