{"thread":{"id":"47421","subject":"[PATCH 0/3] fixes for split index mode","startedAt":"2017-12-10T21:20:39Z","lastAt":"2018-02-07T22:41:34Z","messageCount":95,"participants":["Thomas Gummerer","Eric Sunshine","Brandon Williams","SZEDER Gábor","Lars Schneider","Junio C Hamano","Duy Nguyen","Nguyễn Thái Ngọc Duy","Jonathan Nieder","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"334576","messageId":"20171210212202.28231-1-t.gummerer@gmail.com","threadId":"47421","inReplyTo":null,"subject":"[PATCH 0/3] fixes for split index mode","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2017-12-10T21:21:59Z","receivedAt":"2017-12-10T21:20:39Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On the current master branch, 95ec6b1b33 (\"RelNotes: the eighth\nbatch\", 2017-12-06) , the test suite fails a few tests when\nGIT_TEST_SPLIT_INDEX is set:\n\nTest Summary Report\n-------------------\nt3007-ls-files-recurse-submodules.sh (Wstat: 256 Tests: 21 Failed: 13)\n Failed tests: 2-14\n Non-zero exit status: 1\nt7009-filter-branch-null-sha1.sh (Wstat: 256 Tests: 6 Failed: 1)\n Failed test: 4\n Non-zero exit status: 1\nt5304-prune.sh (Wstat: 256 Tests: 25 Failed: 3)\n Failed tests: 23-25\n Non-zero exit status: 1\nt7814-grep-recurse-submodules.sh (Wstat: 256 Tests: 22 Failed: 13)\n Failed tests: 2-3, 5-10, 12-15, 22\n Non-zero exit status: 1\n\nThis series fixes these and makes travis run the test suite with\nGIT_TEST_SPLIT_INDEX to avoid similar breakages in the future.\n\nThomas Gummerer (3):\n repository: fix repo_read_index with submodules\n prune: fix pruning with multiple worktrees and split index\n travis: run tests with GIT_TEST_SPLIT_INDEX\n\n .travis.yml | 2 +-\n cache.h | 1 +\n read-cache.c | 19 +++++++++++++++++--\n repository.c | 13 ++++++++++++-\n repository.h | 2 ++\n revision.c | 13 ++++++++-----\n 6 files changed, 41 insertions(+), 9 deletions(-)\n\n-- \n2.15.1.504.g5279b80103\n\n"},{"id":"334577","messageId":"20171210212202.28231-2-t.gummerer@gmail.com","threadId":"47421","inReplyTo":"20171210212202.28231-1-t.gummerer@gmail.com","subject":"[PATCH 1/3] repository: fix repo_read_index with submodules","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2017-12-10T21:22:00Z","receivedAt":"2017-12-10T21:20:42Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"repo_read_index calls read_index_from, which takes an path argument for\nthe location of the index file. For the split index however it relies\non the current working directory to construct the path using git_path.\n\nrepo_read_index calls read_index_from with the full path for the index\nfile, however it doesn't change the cwd, so when split index mode is\nturned on, read_index_from can't find the file for the split index.\n\nFor example t3007-ls-files-recurse-submodules.sh was broken with\nGIT_TEST_SPLIT_INDEX set in 188dce131f (\"ls-files: use repository\nobject\", 2017-06-22), and t7814-grep-recurse-submodules.sh was also\nbroken in a similar manner, probably by introducing struct repository\nthere, although I didn't track down the exact commit for that.\n\nFix this by introducing a new read_index_for_repo function, which knows\nabout the correct paths for the submodules.\n\nThe alternative would have been to make the callers pass in the base\npath for the split index, however that ended up being more complicated,\nand I think we want to converge towards using struct repository for\nthings like these anyway.\n\nSigned-off-by: Thomas Gummerer \n---\n cache.h | 1 +\n read-cache.c | 19 +++++++++++++++++--\n repository.c | 2 +-\n 3 files changed, 19 insertions(+), 3 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex cb5db7bf83..d42bea1ef7 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -614,6 +614,7 @@ extern int read_index_preload(struct index_state *, const struct pathspec *paths\n extern int do_read_index(struct index_state *istate, const char *path,\n \t\t\t int must_exist); /* for testting only! */\n extern int read_index_from(struct index_state *, const char *path);\n+extern int read_index_for_repo(const struct repository *);\n extern int is_index_unborn(struct index_state *);\n extern int read_index_unmerged(struct index_state *);\n \ndiff --git a/read-cache.c b/read-cache.c\nindex 2eb81a66b9..4d5c4ad79b 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -20,6 +20,7 @@\n #include \"split-index.h\"\n #include \"utf8.h\"\n #include \"fsmonitor.h\"\n+#include \"repository.h\"\n \n /* Mask for the name length in ce_flags in the on-disk index */\n \n@@ -1871,7 +1872,8 @@ static void freshen_shared_index(char *base_sha1_hex, int warn)\n \tfree(shared_index);\n }\n \n-int read_index_from(struct index_state *istate, const char *path)\n+static int do_read_index_from(struct index_state *istate, const char *path,\n+\t\t\t const struct repository *repo)\n {\n \tstruct split_index *split_index;\n \tint ret;\n@@ -1896,7 +1898,10 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n \n \tbase_sha1_hex = sha1_to_hex(split_index->base_sha1);\n-\tbase_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n+\tif (repo)\n+\t\tbase_path = repo_git_path(repo, \"sharedindex.%s\", base_sha1_hex);\n+\telse\n+\t\tbase_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n \tret = do_read_index(split_index->base, base_path, 1);\n \tif (hashcmp(split_index->base_sha1, split_index->base->sha1))\n \t\tdie(\"broken index, expect %s in %s, got %s\",\n@@ -1909,6 +1914,16 @@ int read_index_from(struct index_state *istate, const char *path)\n \treturn ret;\n }\n \n+int read_index_for_repo(const struct repository *repo)\n+{\n+\treturn do_read_index_from(repo->index, repo->index_file, repo);\n+}\n+\n+int read_index_from(struct index_state *istate, const char *path)\n+{\n+\treturn do_read_index_from(istate, path, NULL);\n+}\n+\n int is_index_unborn(struct index_state *istate)\n {\n \treturn (!istate->cache_nr && !istate->timestamp.sec);\ndiff --git a/repository.c b/repository.c\nindex bb2fae5446..928b1f553d 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -229,5 +229,5 @@ int repo_read_index(struct repository *repo)\n \tif (!repo->index)\n \t\trepo->index = xcalloc(1, sizeof(*repo->index));\n \n-\treturn read_index_from(repo->index, repo->index_file);\n+\treturn read_index_for_repo(repo);\n }\n-- \n2.15.1.504.g5279b80103\n\n"},{"id":"334578","messageId":"20171210212202.28231-3-t.gummerer@gmail.com","threadId":"47421","inReplyTo":"20171210212202.28231-1-t.gummerer@gmail.com","subject":"[PATCH 2/3] prune: fix pruning with multiple worktrees and split index","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2017-12-10T21:22:01Z","receivedAt":"2017-12-10T21:20:44Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"be489d02d2 (\"revision.c: --indexed-objects add objects from all\nworktrees\", 2017-08-23) made sure that pruning takes objects from all\nworktrees into account.\n\nIt did that by reading the index of every worktree and adding the\nnecessary index objects to the set of pending objects. The index is\nread by read_index_from. As mentioned in the previous commit,\nread_index_from depends on the CWD for the location of the split index,\nand add_index_objects_to_pending doesn't set that before using\nread_index_from.\n\nInstead of using read_index_from, use repo_read_index, which is aware of\nthe proper paths for the worktree.\n\nThis fixes t5304-prune when ran with GIT_TEST_SPLIT_INDEX set.\n\nSigned-off-by: Thomas Gummerer \n---\n\nThis also fixes t7009 when ran with GIT_TEST_SPLIT_INDEX. I'm not\nquite sure why it is fixed by this. Either way I tracked the failure\ndown to f767178a5a (\"Merge branch 'jk/no-null-sha1-in-cache-tree'\",\n2017-05-16). Maybe Peff has an idea why this fixes that test?\n\n repository.c | 11 +++++++++++\n repository.h | 2 ++\n revision.c | 13 ++++++++-----\n 3 files changed, 21 insertions(+), 5 deletions(-)\n\ndiff --git a/repository.c b/repository.c\nindex 928b1f553d..3c9bfbd1b8 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -2,6 +2,7 @@\n #include \"repository.h\"\n #include \"config.h\"\n #include \"submodule-config.h\"\n+#include \"worktree.h\"\n \n /* The main repository */\n static struct repository the_repo = {\n@@ -146,6 +147,16 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)\n \treturn -1;\n }\n \n+/*\n+ * Initialize 'repo' based on the provided worktree\n+ * Return 0 upon success and a non-zero value upon failure.\n+ */\n+int repo_worktree_init(struct repository *repo, struct worktree *worktree)\n+{\n+\treturn repo_init(repo, get_worktree_git_dir(worktree),\n+\t\t\t worktree->path);\n+}\n+\n /*\n * Initialize 'submodule' as the submodule given by 'path' in parent repository\n * 'superproject'.\ndiff --git a/repository.h b/repository.h\nindex 7f5e24a0a2..2adeb05bf4 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -4,6 +4,7 @@\n struct config_set;\n struct index_state;\n struct submodule_cache;\n+struct worktree;\n \n struct repository {\n \t/* Environment */\n@@ -87,6 +88,7 @@ extern struct repository *the_repository;\n extern void repo_set_gitdir(struct repository *repo, const char *path);\n extern void repo_set_worktree(struct repository *repo, const char *path);\n extern int repo_init(struct repository *repo, const char *gitdir, const char *worktree);\n+extern int repo_worktree_init(struct repository *repo, struct worktree *worktree);\n extern int repo_submodule_init(struct repository *submodule,\n \t\t\t struct repository *superproject,\n \t\t\t const char *path);\ndiff --git a/revision.c b/revision.c\nindex e2e691dd5a..9d8d9b96d1 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -22,6 +22,7 @@\n #include \"packfile.h\"\n #include \"worktree.h\"\n #include \"argv-array.h\"\n+#include \"repository.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -1346,15 +1347,17 @@ void add_index_objects_to_pending(struct rev_info *revs, unsigned int flags)\n \tworktrees = get_worktrees(0);\n \tfor (p = worktrees; *p; p++) {\n \t\tstruct worktree *wt = *p;\n-\t\tstruct index_state istate = { NULL };\n+\t\tstruct repository *repo;\n \n+\t\trepo = xmalloc(sizeof(struct repository));\n \t\tif (wt->is_current)\n \t\t\tcontinue; /* current index already taken care of */\n+\t\tif (repo_worktree_init(repo, wt))\n+\t\t\tBUG(\"couldn't initialize repository object from worktree\");\n \n-\t\tif (read_index_from(&istate,\n-\t\t\t\t worktree_git_path(wt, \"index\")) > 0)\n-\t\t\tdo_add_index_objects_to_pending(revs, &istate);\n-\t\tdiscard_index(&istate);\n+\t\tif (repo_read_index(repo) > 0)\n+\t\t\tdo_add_index_objects_to_pending(revs, repo->index);\n+\t\tdiscard_index(repo->index);\n \t}\n \tfree_worktrees(worktrees);\n }\n-- \n2.15.1.504.g5279b80103\n\n"},{"id":"334579","messageId":"20171210212202.28231-4-t.gummerer@gmail.com","threadId":"47421","inReplyTo":"20171210212202.28231-1-t.gummerer@gmail.com","subject":"[PATCH 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2017-12-10T21:22:02Z","receivedAt":"2017-12-10T21:20:46Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Make sure that split index doesn't get broken, by running it on travis\nCI.\n\nRun the test suite with split index enabled in linux 64 bit mode, and\nleave split index turned off in 32-bit mode. The laternative would be\nto add an extra target in the matrix, enabling split index mode, but\nthat would only use additional cycles on travis and would not bring that\nmuch benefit, as we are still running the test suite in the \"vanilla\"\nversion in the 32-bit mode.\n\nSigned-off-by: Thomas Gummerer \n---\n .travis.yml | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/.travis.yml b/.travis.yml\nindex 281f101f31..c83c766dee 100644\n--- a/.travis.yml\n+++ b/.travis.yml\n@@ -39,7 +39,7 @@ env:\n \n matrix:\n include:\n- - env: GETTEXT_POISON=YesPlease\n+ - env: GETTEXT_POISON=YesPlease GIT_TEST_SPLIT_INDEX=YesPlease\n os: linux\n compiler:\n addons:\n-- \n2.15.1.504.g5279b80103\n\n"},{"id":"334584","messageId":"CAPig+cRG2BcD3DRuPcyxOsoMV6HuZOXXoyc+CQZm6gBLpsSCaQ@mail.gmail.com","threadId":"47421","inReplyTo":"20171210212202.28231-4-t.gummerer@gmail.com","subject":"Re: [PATCH 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2017-12-10T23:37:16Z","receivedAt":"2017-12-10T23:37:22Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Dec 10, 2017 at 4:22 PM, Thomas Gummerer wrote:\n> Make sure that split index doesn't get broken, by running it on travis\n> CI.\n>\n> Run the test suite with split index enabled in linux 64 bit mode, and\n> leave split index turned off in 32-bit mode. The laternative would be\n\ns/laternative/alternative/\n\n> to add an extra target in the matrix, enabling split index mode, but\n> that would only use additional cycles on travis and would not bring that\n> much benefit, as we are still running the test suite in the \"vanilla\"\n> version in the 32-bit mode.\n>\n> Signed-off-by: Thomas Gummerer \n"},{"id":"334618","messageId":"20171211185406.GA177995@google.com","threadId":"47421","inReplyTo":"20171210212202.28231-2-t.gummerer@gmail.com","subject":"Re: [PATCH 1/3] repository: fix repo_read_index with submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-12-11T18:54:06Z","receivedAt":"2017-12-11T18:54:16Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 12/10, Thomas Gummerer wrote:\n> repo_read_index calls read_index_from, which takes an path argument for\n> the location of the index file. For the split index however it relies\n> on the current working directory to construct the path using git_path.\n> \n> repo_read_index calls read_index_from with the full path for the index\n> file, however it doesn't change the cwd, so when split index mode is\n> turned on, read_index_from can't find the file for the split index.\n> \n> For example t3007-ls-files-recurse-submodules.sh was broken with\n> GIT_TEST_SPLIT_INDEX set in 188dce131f (\"ls-files: use repository\n> object\", 2017-06-22), and t7814-grep-recurse-submodules.sh was also\n> broken in a similar manner, probably by introducing struct repository\n> there, although I didn't track down the exact commit for that.\n> \n> Fix this by introducing a new read_index_for_repo function, which knows\n> about the correct paths for the submodules.\n> \n> The alternative would have been to make the callers pass in the base\n> path for the split index, however that ended up being more complicated,\n> and I think we want to converge towards using struct repository for\n> things like these anyway.\n\nThanks for catching this, I'm not a user of split index myself which is\nwhy I unfortunately overlooked this. Definitely a good change. I\nreally only have one nit below.\n\n> \n> Signed-off-by: Thomas Gummerer \n> ---\n> cache.h | 1 +\n> read-cache.c | 19 +++++++++++++++++--\n> repository.c | 2 +-\n> 3 files changed, 19 insertions(+), 3 deletions(-)\n> \n> diff --git a/cache.h b/cache.h\n> index cb5db7bf83..d42bea1ef7 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -614,6 +614,7 @@ extern int read_index_preload(struct index_state *, const struct pathspec *paths\n> extern int do_read_index(struct index_state *istate, const char *path,\n> \t\t\t int must_exist); /* for testting only! */\n> extern int read_index_from(struct index_state *, const char *path);\n> +extern int read_index_for_repo(const struct repository *);\n> extern int is_index_unborn(struct index_state *);\n> extern int read_index_unmerged(struct index_state *);\n> \n> diff --git a/read-cache.c b/read-cache.c\n> index 2eb81a66b9..4d5c4ad79b 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -20,6 +20,7 @@\n> #include \"split-index.h\"\n> #include \"utf8.h\"\n> #include \"fsmonitor.h\"\n> +#include \"repository.h\"\n> \n> /* Mask for the name length in ce_flags in the on-disk index */\n> \n> @@ -1871,7 +1872,8 @@ static void freshen_shared_index(char *base_sha1_hex, int warn)\n> \tfree(shared_index);\n> }\n> \n> -int read_index_from(struct index_state *istate, const char *path)\n> +static int do_read_index_from(struct index_state *istate, const char *path,\n> +\t\t\t const struct repository *repo)\n> {\n> \tstruct split_index *split_index;\n> \tint ret;\n> @@ -1896,7 +1898,10 @@ int read_index_from(struct index_state *istate, const char *path)\n> \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n> \n> \tbase_sha1_hex = sha1_to_hex(split_index->base_sha1);\n> -\tbase_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n> +\tif (repo)\n> +\t\tbase_path = repo_git_path(repo, \"sharedindex.%s\", base_sha1_hex);\n> +\telse\n> +\t\tbase_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n> \tret = do_read_index(split_index->base, base_path, 1);\n> \tif (hashcmp(split_index->base_sha1, split_index->base->sha1))\n> \t\tdie(\"broken index, expect %s in %s, got %s\",\n> @@ -1909,6 +1914,16 @@ int read_index_from(struct index_state *istate, const char *path)\n> \treturn ret;\n> }\n> \n> +int read_index_for_repo(const struct repository *repo)\n> +{\n> +\treturn do_read_index_from(repo->index, repo->index_file, repo);\n> +}\n> +\n> +int read_index_from(struct index_state *istate, const char *path)\n> +{\n> +\treturn do_read_index_from(istate, path, NULL);\n> +}\n\nInstead of passing NULL and having to special case it in\n'do_read_index_from()', how about we pass in 'the_repository'?\n\n> +\n> int is_index_unborn(struct index_state *istate)\n> {\n> \treturn (!istate->cache_nr && !istate->timestamp.sec);\n> diff --git a/repository.c b/repository.c\n> index bb2fae5446..928b1f553d 100644\n> --- a/repository.c\n> +++ b/repository.c\n> @@ -229,5 +229,5 @@ int repo_read_index(struct repository *repo)\n> \tif (!repo->index)\n> \t\trepo->index = xcalloc(1, sizeof(*repo->index));\n> \n> -\treturn read_index_from(repo->index, repo->index_file);\n> +\treturn read_index_for_repo(repo);\n> }\n> -- \n> 2.15.1.504.g5279b80103\n> \n\n-- \nBrandon Williams\n"},{"id":"334619","messageId":"20171211190946.GB177995@google.com","threadId":"47421","inReplyTo":"20171210212202.28231-3-t.gummerer@gmail.com","subject":"Re: [PATCH 2/3] prune: fix pruning with multiple worktrees and split index","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-12-11T19:09:46Z","receivedAt":"2017-12-11T19:09:56Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 12/10, Thomas Gummerer wrote:\n> be489d02d2 (\"revision.c: --indexed-objects add objects from all\n> worktrees\", 2017-08-23) made sure that pruning takes objects from all\n> worktrees into account.\n> \n> It did that by reading the index of every worktree and adding the\n> necessary index objects to the set of pending objects. The index is\n> read by read_index_from. As mentioned in the previous commit,\n> read_index_from depends on the CWD for the location of the split index,\n> and add_index_objects_to_pending doesn't set that before using\n> read_index_from.\n> \n> Instead of using read_index_from, use repo_read_index, which is aware of\n> the proper paths for the worktree.\n> \n> This fixes t5304-prune when ran with GIT_TEST_SPLIT_INDEX set.\n> \n\nI'm on the fence about this change. I understand that this will ensure\nthat the proper objects aren't pruned when using a split index in the\npresence of worktrees but I think the solution needs to be thought\nthrough a bit more.\n\nMy big concern right now is the interaction of 'struct worktree's and\n'struct repository'. I'll try to highlight my concerns below.\n\n> Signed-off-by: Thomas Gummerer \n> ---\n> \n> This also fixes t7009 when ran with GIT_TEST_SPLIT_INDEX. I'm not\n> quite sure why it is fixed by this. Either way I tracked the failure\n> down to f767178a5a (\"Merge branch 'jk/no-null-sha1-in-cache-tree'\",\n> 2017-05-16). Maybe Peff has an idea why this fixes that test?\n> \n> repository.c | 11 +++++++++++\n> repository.h | 2 ++\n> revision.c | 13 ++++++++-----\n> 3 files changed, 21 insertions(+), 5 deletions(-)\n> \n> diff --git a/repository.c b/repository.c\n> index 928b1f553d..3c9bfbd1b8 100644\n> --- a/repository.c\n> +++ b/repository.c\n> @@ -2,6 +2,7 @@\n> #include \"repository.h\"\n> #include \"config.h\"\n> #include \"submodule-config.h\"\n> +#include \"worktree.h\"\n> \n> /* The main repository */\n> static struct repository the_repo = {\n> @@ -146,6 +147,16 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)\n> \treturn -1;\n> }\n> \n> +/*\n> + * Initialize 'repo' based on the provided worktree\n> + * Return 0 upon success and a non-zero value upon failure.\n> + */\n> +int repo_worktree_init(struct repository *repo, struct worktree *worktree)\n> +{\n> +\treturn repo_init(repo, get_worktree_git_dir(worktree),\n> +\t\t\t worktree->path);\n> +}\n\nMy first concern is the use of 'get_worktree_git_dir()'. Under the hood\nit calls 'get_git_dir()', 'get_git_common_dir()', and\n'git_common_path()' which rely on global state as stored in\n'the_repository'. So how does one initialize a repository struct (using\nthis initializer) using a worktree from a repository other than the\nglobal 'the_repository' struct? I'm not sure I have an answer right\nnow, but its an issue that needs to be thought through before we head\ndown this road.\n\nJust thinking to myself, Does it make sense to have worktree's as a\nseparate struct or to have them stored in 'struct repository' in some\nway? Shouldn't a repository struct have a way to interact with all of\nits worktrees? How would initializing a repository struct for every\nworktree work once we migrate the object store to be stored in 'struct\nrepoisotry'? Shouldn't every worktree share the same object store\nin-memory like they do on-disk?\n\n> +\n> /*\n> * Initialize 'submodule' as the submodule given by 'path' in parent repository\n> * 'superproject'.\n> diff --git a/repository.h b/repository.h\n> index 7f5e24a0a2..2adeb05bf4 100644\n> --- a/repository.h\n> +++ b/repository.h\n> @@ -4,6 +4,7 @@\n> struct config_set;\n> struct index_state;\n> struct submodule_cache;\n> +struct worktree;\n> \n> struct repository {\n> \t/* Environment */\n> @@ -87,6 +88,7 @@ extern struct repository *the_repository;\n> extern void repo_set_gitdir(struct repository *repo, const char *path);\n> extern void repo_set_worktree(struct repository *repo, const char *path);\n> extern int repo_init(struct repository *repo, const char *gitdir, const char *worktree);\n> +extern int repo_worktree_init(struct repository *repo, struct worktree *worktree);\n> extern int repo_submodule_init(struct repository *submodule,\n> \t\t\t struct repository *superproject,\n> \t\t\t const char *path);\n> diff --git a/revision.c b/revision.c\n> index e2e691dd5a..9d8d9b96d1 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -22,6 +22,7 @@\n> #include \"packfile.h\"\n> #include \"worktree.h\"\n> #include \"argv-array.h\"\n> +#include \"repository.h\"\n> \n> volatile show_early_output_fn_t show_early_output;\n> \n> @@ -1346,15 +1347,17 @@ void add_index_objects_to_pending(struct rev_info *revs, unsigned int flags)\n> \tworktrees = get_worktrees(0);\n> \tfor (p = worktrees; *p; p++) {\n> \t\tstruct worktree *wt = *p;\n> -\t\tstruct index_state istate = { NULL };\n> +\t\tstruct repository *repo;\n> \n> +\t\trepo = xmalloc(sizeof(struct repository));\n\nThis was allocated but never freed, was that intentional?\n\n> \t\tif (wt->is_current)\n> \t\t\tcontinue; /* current index already taken care of */\n> +\t\tif (repo_worktree_init(repo, wt))\n> +\t\t\tBUG(\"couldn't initialize repository object from worktree\");\n> \n> -\t\tif (read_index_from(&istate,\n> -\t\t\t\t worktree_git_path(wt, \"index\")) > 0)\n> -\t\t\tdo_add_index_objects_to_pending(revs, &istate);\n> -\t\tdiscard_index(&istate);\n> +\t\tif (repo_read_index(repo) > 0)\n> +\t\t\tdo_add_index_objects_to_pending(revs, repo->index);\n> +\t\tdiscard_index(repo->index);\n\nOne we have separate object stores per-repository how would we handle\nthis since this pruning should only work on a single repository's object\nstore?\n\n> \t}\n> \tfree_worktrees(worktrees);\n> }\n> -- \n> 2.15.1.504.g5279b80103\n> \n\n-- \nBrandon Williams\n"},{"id":"334626","messageId":"20171211203707.GB25616@hank","threadId":"47421","inReplyTo":"20171211185406.GA177995@google.com","subject":"Re: [PATCH 1/3] repository: fix repo_read_index with submodules","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2017-12-11T20:37:07Z","receivedAt":"2017-12-11T20:35:31Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 12/11, Brandon Williams wrote:\n> On 12/10, Thomas Gummerer wrote:\n> > repo_read_index calls read_index_from, which takes an path argument for\n> > the location of the index file. For the split index however it relies\n> > on the current working directory to construct the path using git_path.\n> > \n> > repo_read_index calls read_index_from with the full path for the index\n> > file, however it doesn't change the cwd, so when split index mode is\n> > turned on, read_index_from can't find the file for the split index.\n> > \n> > For example t3007-ls-files-recurse-submodules.sh was broken with\n> > GIT_TEST_SPLIT_INDEX set in 188dce131f (\"ls-files: use repository\n> > object\", 2017-06-22), and t7814-grep-recurse-submodules.sh was also\n> > broken in a similar manner, probably by introducing struct repository\n> > there, although I didn't track down the exact commit for that.\n> > \n> > Fix this by introducing a new read_index_for_repo function, which knows\n> > about the correct paths for the submodules.\n> > \n> > The alternative would have been to make the callers pass in the base\n> > path for the split index, however that ended up being more complicated,\n> > and I think we want to converge towards using struct repository for\n> > things like these anyway.\n> \n> Thanks for catching this, I'm not a user of split index myself which is\n> why I unfortunately overlooked this. Definitely a good change. I\n> really only have one nit below.\n\nMe neither, I just remember to run the split index tests every once in\na while, which is also why it's taken so long to catch this.\nHopefully after we fixed this we can get travis to run the tests,\nwhich should help :)\n\n> > \n> > Signed-off-by: Thomas Gummerer \n> > ---\n> > cache.h | 1 +\n> > read-cache.c | 19 +++++++++++++++++--\n> > repository.c | 2 +-\n> > 3 files changed, 19 insertions(+), 3 deletions(-)\n> > \n> > diff --git a/cache.h b/cache.h\n> > index cb5db7bf83..d42bea1ef7 100644\n> > --- a/cache.h\n> > +++ b/cache.h\n> > @@ -614,6 +614,7 @@ extern int read_index_preload(struct index_state *, const struct pathspec *paths\n> > extern int do_read_index(struct index_state *istate, const char *path,\n> > \t\t\t int must_exist); /* for testting only! */\n> > extern int read_index_from(struct index_state *, const char *path);\n> > +extern int read_index_for_repo(const struct repository *);\n> > extern int is_index_unborn(struct index_state *);\n> > extern int read_index_unmerged(struct index_state *);\n> > \n> > diff --git a/read-cache.c b/read-cache.c\n> > index 2eb81a66b9..4d5c4ad79b 100644\n> > --- a/read-cache.c\n> > +++ b/read-cache.c\n> > @@ -20,6 +20,7 @@\n> > #include \"split-index.h\"\n> > #include \"utf8.h\"\n> > #include \"fsmonitor.h\"\n> > +#include \"repository.h\"\n> > \n> > /* Mask for the name length in ce_flags in the on-disk index */\n> > \n> > @@ -1871,7 +1872,8 @@ static void freshen_shared_index(char *base_sha1_hex, int warn)\n> > \tfree(shared_index);\n> > }\n> > \n> > -int read_index_from(struct index_state *istate, const char *path)\n> > +static int do_read_index_from(struct index_state *istate, const char *path,\n> > +\t\t\t const struct repository *repo)\n> > {\n> > \tstruct split_index *split_index;\n> > \tint ret;\n> > @@ -1896,7 +1898,10 @@ int read_index_from(struct index_state *istate, const char *path)\n> > \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n> > \n> > \tbase_sha1_hex = sha1_to_hex(split_index->base_sha1);\n> > -\tbase_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n> > +\tif (repo)\n> > +\t\tbase_path = repo_git_path(repo, \"sharedindex.%s\", base_sha1_hex);\n> > +\telse\n> > +\t\tbase_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n> > \tret = do_read_index(split_index->base, base_path, 1);\n> > \tif (hashcmp(split_index->base_sha1, split_index->base->sha1))\n> > \t\tdie(\"broken index, expect %s in %s, got %s\",\n> > @@ -1909,6 +1914,16 @@ int read_index_from(struct index_state *istate, const char *path)\n> > \treturn ret;\n> > }\n> > \n> > +int read_index_for_repo(const struct repository *repo)\n> > +{\n> > +\treturn do_read_index_from(repo->index, repo->index_file, repo);\n> > +}\n> > +\n> > +int read_index_from(struct index_state *istate, const char *path)\n> > +{\n> > +\treturn do_read_index_from(istate, path, NULL);\n> > +}\n> \n> Instead of passing NULL and having to special case it in\n> 'do_read_index_from()', how about we pass in 'the_repository'?\n\nI think that makes sense the only way this function used to work with\nsplit index turned on is if it's called from the main repository, so\njust passing through 'the_repository' would have the function behave\nthe exact same way as before.\n\n> > +\n> > int is_index_unborn(struct index_state *istate)\n> > {\n> > \treturn (!istate->cache_nr && !istate->timestamp.sec);\n> > diff --git a/repository.c b/repository.c\n> > index bb2fae5446..928b1f553d 100644\n> > --- a/repository.c\n> > +++ b/repository.c\n> > @@ -229,5 +229,5 @@ int repo_read_index(struct repository *repo)\n> > \tif (!repo->index)\n> > \t\trepo->index = xcalloc(1, sizeof(*repo->index));\n> > \n> > -\treturn read_index_from(repo->index, repo->index_file);\n> > +\treturn read_index_for_repo(repo);\n> > }\n> > -- \n> > 2.15.1.504.g5279b80103\n> > \n> \n> -- \n> Brandon Williams\n"},{"id":"334631","messageId":"20171211210948.6867-1-szeder.dev@gmail.com","threadId":"47421","inReplyTo":"20171210212202.28231-4-t.gummerer@gmail.com","subject":"Re: [PATCH 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2017-12-11T21:09:48Z","receivedAt":"2017-12-11T21:10:09Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"> Make sure that split index doesn't get broken, by running it on travis\n> CI.\n> \n> Run the test suite with split index enabled in linux 64 bit mode, and\n> leave split index turned off in 32-bit mode.\n\nThis doesn't accurately describe what the patch does.\nTravis CI runs three 64 bit Linux build jobs for us: one compiled with\nClang, one with GCC, and one with GETTEXT_POISON enabled. This patch\nenables split index only in the latter build job, but not in the Clang\nand GCC build jobs.\n\n> The laternative would be\n> to add an extra target in the matrix, enabling split index mode, but\n> that would only use additional cycles on travis and would not bring that\n> much benefit, as we are still running the test suite in the \"vanilla\"\n> version in the 32-bit mode.\n> \n> Signed-off-by: Thomas Gummerer \n> ---\n> .travis.yml | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n> \n> diff --git a/.travis.yml b/.travis.yml\n> index 281f101f31..c83c766dee 100644\n> --- a/.travis.yml\n> +++ b/.travis.yml\n> @@ -39,7 +39,7 @@ env:\n> \n> matrix:\n> include:\n> - - env: GETTEXT_POISON=YesPlease\n> + - env: GETTEXT_POISON=YesPlease GIT_TEST_SPLIT_INDEX=YesPlease\n> os: linux\n> compiler:\n> addons:\n> -- \n> 2.15.1.504.g5279b80103\n"},{"id":"334634","messageId":"20171211213948.GC25616@hank","threadId":"47421","inReplyTo":"20171211190946.GB177995@google.com","subject":"Re: [PATCH 2/3] prune: fix pruning with multiple worktrees and split index","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2017-12-11T21:39:48Z","receivedAt":"2017-12-11T21:38:10Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 12/11, Brandon Williams wrote:\n> On 12/10, Thomas Gummerer wrote:\n> > be489d02d2 (\"revision.c: --indexed-objects add objects from all\n> > worktrees\", 2017-08-23) made sure that pruning takes objects from all\n> > worktrees into account.\n> > \n> > It did that by reading the index of every worktree and adding the\n> > necessary index objects to the set of pending objects. The index is\n> > read by read_index_from. As mentioned in the previous commit,\n> > read_index_from depends on the CWD for the location of the split index,\n> > and add_index_objects_to_pending doesn't set that before using\n> > read_index_from.\n> > \n> > Instead of using read_index_from, use repo_read_index, which is aware of\n> > the proper paths for the worktree.\n> > \n> > This fixes t5304-prune when ran with GIT_TEST_SPLIT_INDEX set.\n> > \n> \n> I'm on the fence about this change. I understand that this will ensure\n> that the proper objects aren't pruned when using a split index in the\n> presence of worktrees but I think the solution needs to be thought\n> through a bit more.\n> \n> My big concern right now is the interaction of 'struct worktree's and\n> 'struct repository'. I'll try to highlight my concerns below.\n\nThanks for the review! My thoughts on the concerns are below.\n\n> > Signed-off-by: Thomas Gummerer \n> > ---\n> > \n> > This also fixes t7009 when ran with GIT_TEST_SPLIT_INDEX. I'm not\n> > quite sure why it is fixed by this. Either way I tracked the failure\n> > down to f767178a5a (\"Merge branch 'jk/no-null-sha1-in-cache-tree'\",\n> > 2017-05-16). Maybe Peff has an idea why this fixes that test?\n> > \n> > repository.c | 11 +++++++++++\n> > repository.h | 2 ++\n> > revision.c | 13 ++++++++-----\n> > 3 files changed, 21 insertions(+), 5 deletions(-)\n> > \n> > diff --git a/repository.c b/repository.c\n> > index 928b1f553d..3c9bfbd1b8 100644\n> > --- a/repository.c\n> > +++ b/repository.c\n> > @@ -2,6 +2,7 @@\n> > #include \"repository.h\"\n> > #include \"config.h\"\n> > #include \"submodule-config.h\"\n> > +#include \"worktree.h\"\n> > \n> > /* The main repository */\n> > static struct repository the_repo = {\n> > @@ -146,6 +147,16 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)\n> > \treturn -1;\n> > }\n> > \n> > +/*\n> > + * Initialize 'repo' based on the provided worktree\n> > + * Return 0 upon success and a non-zero value upon failure.\n> > + */\n> > +int repo_worktree_init(struct repository *repo, struct worktree *worktree)\n> > +{\n> > +\treturn repo_init(repo, get_worktree_git_dir(worktree),\n> > +\t\t\t worktree->path);\n> > +}\n> \n> My first concern is the use of 'get_worktree_git_dir()'. Under the hood\n> it calls 'get_git_dir()', 'get_git_common_dir()', and\n> 'git_common_path()' which rely on global state as stored in\n> 'the_repository'. So how does one initialize a repository struct (using\n> this initializer) using a worktree from a repository other than the\n> global 'the_repository' struct? I'm not sure I have an answer right\n> now, but its an issue that needs to be thought through before we head\n> down this road.\n\nThe main reason for just using 'get_worktree_git_dir()' was because it\nexists and it does exactly what I needed. If we ever have the need to\ninitialize a repository struct for a worktree from another repository\nstruct, I would imagine we introduce a new function\n'repo_get_worktree_git_dir()', which takes a struct repository as\ninput and returns the worktree git dir based on that.\n\nAnd then we'd have to add a different initializer\n'repo_worktree_init_from_repo()' or something like that, where we\ncould initialize a worktree from a repository struct different from\n'the_repository'. But maybe there are some complications there that I\ndidn't think about?\n\n> Just thinking to myself, Does it make sense to have worktree's as a\n> separate struct or to have them stored in 'struct repository' in some\n> way?\n\nI'm not sure I have a good answer for this. Looking at the libgit2\nsource, which has a repository struct for (I assume at least, I'm not\nvery familiar with the libgit2 source :)), they have a repository\nstruct for each worktree, with a flag specifying whether the struct is\nfor a worktree, or a \"normal\" git repository/the main worktree.\n\nObviously that doesn't mean we should do it the same way, but it\nprobably also doesn't hurt to look at prior art.\n\n> Shouldn't a repository struct have a way to interact with all of\n> its worktrees?\n\nIt could go either way I guess depending on how we want to design it.\nIf we want the repository struct to interact with all worktrees, it\nseems to me like we would almost need some kind of 'struct\nsuper_repository', which then contains a list of 'struct repository's\nrepresenting each worktree, probably including the main worktree.\n\nBut then again in normal operation we probably only want the\nrepository struct for the current repository we're working with, not\nall of the worktrees. I suspect it's only in some special cases where\nwe want to have all worktrees in memory at once (such as in the prune\ncase).\n\nOr how would you haven envisioned this?\n\n> How would initializing a repository struct for every\n> worktree work once we migrate the object store to be stored in 'struct\n> repoisotry'? Shouldn't every worktree share the same object store\n> in-memory like they do on-disk?\n\nI haven't thought this far. They probably want to share the same\nobject store, if they all want the object store to be loaded. But\nwe can do that either way, whether we have one 'struct repository'\nthat has links to the worktrees, or we have multiple 'struct\nrepository's, which each have a ref-counted pointer to the object\nstore or something like that.\n\n> > +\n> > /*\n> > * Initialize 'submodule' as the submodule given by 'path' in parent repository\n> > * 'superproject'.\n> > diff --git a/repository.h b/repository.h\n> > index 7f5e24a0a2..2adeb05bf4 100644\n> > --- a/repository.h\n> > +++ b/repository.h\n> > @@ -4,6 +4,7 @@\n> > struct config_set;\n> > struct index_state;\n> > struct submodule_cache;\n> > +struct worktree;\n> > \n> > struct repository {\n> > \t/* Environment */\n> > @@ -87,6 +88,7 @@ extern struct repository *the_repository;\n> > extern void repo_set_gitdir(struct repository *repo, const char *path);\n> > extern void repo_set_worktree(struct repository *repo, const char *path);\n> > extern int repo_init(struct repository *repo, const char *gitdir, const char *worktree);\n> > +extern int repo_worktree_init(struct repository *repo, struct worktree *worktree);\n> > extern int repo_submodule_init(struct repository *submodule,\n> > \t\t\t struct repository *superproject,\n> > \t\t\t const char *path);\n> > diff --git a/revision.c b/revision.c\n> > index e2e691dd5a..9d8d9b96d1 100644\n> > --- a/revision.c\n> > +++ b/revision.c\n> > @@ -22,6 +22,7 @@\n> > #include \"packfile.h\"\n> > #include \"worktree.h\"\n> > #include \"argv-array.h\"\n> > +#include \"repository.h\"\n> > \n> > volatile show_early_output_fn_t show_early_output;\n> > \n> > @@ -1346,15 +1347,17 @@ void add_index_objects_to_pending(struct rev_info *revs, unsigned int flags)\n> > \tworktrees = get_worktrees(0);\n> > \tfor (p = worktrees; *p; p++) {\n> > \t\tstruct worktree *wt = *p;\n> > -\t\tstruct index_state istate = { NULL };\n> > +\t\tstruct repository *repo;\n> > \n> > +\t\trepo = xmalloc(sizeof(struct repository));\n> \n> This was allocated but never freed, was that intentional?\n\nNo, that was an oversight, will fix, thanks!\n\n> > \t\tif (wt->is_current)\n> > \t\t\tcontinue; /* current index already taken care of */\n> > +\t\tif (repo_worktree_init(repo, wt))\n> > +\t\t\tBUG(\"couldn't initialize repository object from worktree\");\n> > \n> > -\t\tif (read_index_from(&istate,\n> > -\t\t\t\t worktree_git_path(wt, \"index\")) > 0)\n> > -\t\t\tdo_add_index_objects_to_pending(revs, &istate);\n> > -\t\tdiscard_index(&istate);\n> > +\t\tif (repo_read_index(repo) > 0)\n> > +\t\t\tdo_add_index_objects_to_pending(revs, repo->index);\n> > +\t\tdiscard_index(repo->index);\n> \n> One we have separate object stores per-repository how would we handle\n> this since this pruning should only work on a single repository's object\n> store?\n\nThat depends a lot on how we design the object store and how it's held\nby the repository struct. If we only have one in-memory\nrepresentation of it, the answer is obvious. If we have multiple\nin-memory representations of it at the same time, things will get a\nbit more hairy. But until we have at least a design of how that would\nlook like it's quite hard to tell I'd say.\n\n> > \t}\n> > \tfree_worktrees(worktrees);\n> > }\n> > -- \n> > 2.15.1.504.g5279b80103\n> > \n> \n> -- \n> Brandon Williams\n"},{"id":"334635","messageId":"20171211214255.GA24597@hank","threadId":"47421","inReplyTo":"20171211210948.6867-1-szeder.dev@gmail.com","subject":"Re: [PATCH 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2017-12-11T21:42:55Z","receivedAt":"2017-12-11T21:41:17Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 12/11, SZEDER Gábor wrote:\n> > Make sure that split index doesn't get broken, by running it on travis\n> > CI.\n> > \n> > Run the test suite with split index enabled in linux 64 bit mode, and\n> > leave split index turned off in 32-bit mode.\n> \n> This doesn't accurately describe what the patch does.\n> Travis CI runs three 64 bit Linux build jobs for us: one compiled with\n> Clang, one with GCC, and one with GETTEXT_POISON enabled. This patch\n> enables split index only in the latter build job, but not in the Clang\n> and GCC build jobs.\n\nYou're right, it's my first time using travis CI and I got confused\nabout how the .travis.yml works, thanks for catching that. Will\nre-phrase the commit message.\n\n> > The laternative would be\n> > to add an extra target in the matrix, enabling split index mode, but\n> > that would only use additional cycles on travis and would not bring that\n> > much benefit, as we are still running the test suite in the \"vanilla\"\n> > version in the 32-bit mode.\n> > \n> > Signed-off-by: Thomas Gummerer \n> > ---\n> > .travis.yml | 2 +-\n> > 1 file changed, 1 insertion(+), 1 deletion(-)\n> > \n> > diff --git a/.travis.yml b/.travis.yml\n> > index 281f101f31..c83c766dee 100644\n> > --- a/.travis.yml\n> > +++ b/.travis.yml\n> > @@ -39,7 +39,7 @@ env:\n> > \n> > matrix:\n> > include:\n> > - - env: GETTEXT_POISON=YesPlease\n> > + - env: GETTEXT_POISON=YesPlease GIT_TEST_SPLIT_INDEX=YesPlease\n> > os: linux\n> > compiler:\n> > addons:\n> > -- \n> > 2.15.1.504.g5279b80103\n"},{"id":"334681","messageId":"C5F93DFF-D3DF-4507-8546-941BFA768705@gmail.com","threadId":"47421","inReplyTo":"20171211214255.GA24597@hank","subject":"Re: [PATCH 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-12-12T15:54:04Z","receivedAt":"2017-12-12T15:54:17Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 11 Dec 2017, at 22:42, Thomas Gummerer wrote:\n> \n> On 12/11, SZEDER Gábor wrote:\n>>> Make sure that split index doesn't get broken, by running it on travis\n>>> CI.\n>>> \n>>> Run the test suite with split index enabled in linux 64 bit mode, and\n>>> leave split index turned off in 32-bit mode.\n>> \n>> This doesn't accurately describe what the patch does.\n>> Travis CI runs three 64 bit Linux build jobs for us: one compiled with\n>> Clang, one with GCC, and one with GETTEXT_POISON enabled. This patch\n>> enables split index only in the latter build job, but not in the Clang\n>> and GCC build jobs.\n> \n> You're right, it's my first time using travis CI and I got confused\n> about how the .travis.yml works, thanks for catching that. Will\n> re-phrase the commit message.\n\nSzeder is spot on. If you fix up the message, then this patch looks\nperfect! Pragmatic (= very good!) idea to combine GITTEXT_POISON with \nGIT_TEST_SPLIT_INDEX :-)\n\nThanks,\nLars\n\n\n> \n>>> The laternative would be\n>>> to add an extra target in the matrix, enabling split index mode, but\n>>> that would only use additional cycles on travis and would not bring that\n>>> much benefit, as we are still running the test suite in the \"vanilla\"\n>>> version in the 32-bit mode.\n>>> \n>>> Signed-off-by: Thomas Gummerer \n>>> ---\n>>> .travis.yml | 2 +-\n>>> 1 file changed, 1 insertion(+), 1 deletion(-)\n>>> \n>>> diff --git a/.travis.yml b/.travis.yml\n>>> index 281f101f31..c83c766dee 100644\n>>> --- a/.travis.yml\n>>> +++ b/.travis.yml\n>>> @@ -39,7 +39,7 @@ env:\n>>> \n>>> matrix:\n>>> include:\n>>> - - env: GETTEXT_POISON=YesPlease\n>>> + - env: GETTEXT_POISON=YesPlease GIT_TEST_SPLIT_INDEX=YesPlease\n>>> os: linux\n>>> compiler:\n>>> addons:\n>>> -- \n>>> 2.15.1.504.g5279b80103\n\n"},{"id":"334693","messageId":"xmqq609b3gi3.fsf@gitster.mtv.corp.google.com","threadId":"47421","inReplyTo":"C5F93DFF-D3DF-4507-8546-941BFA768705@gmail.com","subject":"Re: [PATCH 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-12-12T19:15:00Z","receivedAt":"2017-12-12T19:15:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Schneider writes:\n\n>> You're right, it's my first time using travis CI and I got confused\n>> about how the .travis.yml works, thanks for catching that. Will\n>> re-phrase the commit message.\n>\n> Szeder is spot on. If you fix up the message, then this patch looks\n> perfect! Pragmatic (= very good!) idea to combine GITTEXT_POISON with \n> GIT_TEST_SPLIT_INDEX :-)\n\nI am failing to guess the real intent of the smiley here.\n\nIf split-index code is so easy to break, I do not think it is a good\nidea to combine it into the poison build. In fact, the poison test\nis useless on a codebase where other/real breakages are expected to\nexist, because it is about seeing messages meant for non-humans are\nnot passed to the _() mechanism by sloppy coding, and the way it\ndoes so is to corrupt all the messages that come through the _()\nmechanism. If we do not even produce a message when a correct code\n_should_ produce one, poison test would catch nothing useful.\n\nI wonder if it makes more sense to update ci/run-tests.sh so that\nits final step is run twice with different settings, like so?\n\n ci/run-tests.sh | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/ci/run-tests.sh b/ci/run-tests.sh\nindex f0c743de94..15a5f5a6cc 100755\n--- a/ci/run-tests.sh\n+++ b/ci/run-tests.sh\n@@ -8,3 +8,4 @@\n mkdir -p $HOME/travis-cache\n ln -s $HOME/travis-cache/.prove t/.prove\n make --quiet test\n+GIT_TEST_SPLIT_INDEX=LetsTryIt make --quiet test\n"},{"id":"334706","messageId":"20171212201558.GB24597@hank","threadId":"47421","inReplyTo":"xmqq609b3gi3.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2017-12-12T20:15:58Z","receivedAt":"2017-12-12T20:14:19Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 12/12, Junio C Hamano wrote:\n> Lars Schneider writes:\n> \n> >> You're right, it's my first time using travis CI and I got confused\n> >> about how the .travis.yml works, thanks for catching that. Will\n> >> re-phrase the commit message.\n> >\n> > Szeder is spot on. If you fix up the message, then this patch looks\n> > perfect! Pragmatic (= very good!) idea to combine GITTEXT_POISON with \n> > GIT_TEST_SPLIT_INDEX :-)\n> \n> I am failing to guess the real intent of the smiley here.\n> \n> If split-index code is so easy to break, I do not think it is a good\n> idea to combine it into the poison build. In fact, the poison test\n> is useless on a codebase where other/real breakages are expected to\n> exist, because it is about seeing messages meant for non-humans are\n> not passed to the _() mechanism by sloppy coding, and the way it\n> does so is to corrupt all the messages that come through the _()\n> mechanism. If we do not even produce a message when a correct code\n> _should_ produce one, poison test would catch nothing useful.\n\nThe breakages wen the split-index code fails tend to break things in\nmuch more obvious manners than a wrong message, usually git ends up\ndying if it gets broken. Both of the bugs that were fixed here would\nhave been caught with the change in my patch.\n\nBut yeah I can see the argument that it doesn't give us a guarantee\nthat it catches all things the test suite could catch.\n\n> I wonder if it makes more sense to update ci/run-tests.sh so that\n> its final step is run twice with different settings, like so?\n\nI kind of wanted to avoid that, because it ends up running the test\nsuite twice on travis for every test, which seems a bit overkill. But\nI don't exactly how worried we are about cycles on travis (I don't\ncheck it very often personally, so I don't really know). If we aren't\nworried about cycles what you have below would certainly make sense.\n\n> ci/run-tests.sh | 1 +\n> 1 file changed, 1 insertion(+)\n> \n> diff --git a/ci/run-tests.sh b/ci/run-tests.sh\n> index f0c743de94..15a5f5a6cc 100755\n> --- a/ci/run-tests.sh\n> +++ b/ci/run-tests.sh\n> @@ -8,3 +8,4 @@\n> mkdir -p $HOME/travis-cache\n> ln -s $HOME/travis-cache/.prove t/.prove\n> make --quiet test\n> +GIT_TEST_SPLIT_INDEX=LetsTryIt make --quiet test\n"},{"id":"334710","messageId":"xmqqvahbzn3c.fsf@gitster.mtv.corp.google.com","threadId":"47421","inReplyTo":"20171212201558.GB24597@hank","subject":"Re: [PATCH 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-12-12T20:51:35Z","receivedAt":"2017-12-12T20:51:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Gummerer writes:\n\n>\n> The breakages wen the split-index code fails tend to break things in\n> much more obvious manners than a wrong message, usually git ends up\n> dying if it gets broken. Both of the bugs that were fixed here would\n> have been caught with the change in my patch.\n>\n> But yeah I can see the argument that it doesn't give us a guarantee\n> that it catches all things the test suite could catch.\n\nI think you misunderstood me. When split index is much easier to\nbreak than poison tests, combining them together would hurt the test\ncoverage of poison tests. If you value poison tests much more than\nhow well split index mode works, that is a worse outcome.\n\n>> I wonder if it makes more sense to update ci/run-tests.sh so that\n>> its final step is run twice with different settings, like so?\n>\n> I kind of wanted to avoid that, because it ends up running the test\n> suite twice on travis for every test, which seems a bit overkill. But\n> I don't exactly how worried we are about cycles on travis (I don't\n> check it very often personally, so I don't really know). If we aren't\n> worried about cycles what you have below would certainly make sense.\n\nI think 64-bit gcc/clang builds tend to cost us about 10-20 minutes\neach, and 32-bit linux builds about 10 minutes or so. I wonder if\nit makes sense to do the \"run twice\" for only one of 64-bit builds\n(perhaps the clang one, as I suspect 32-bit linux one uses gcc) and\nthe 32-bit linux build, and nowhere else.\n"},{"id":"334761","messageId":"09E575DE-ED7A-40FA-9A47-0ABB61452C28@gmail.com","threadId":"47421","inReplyTo":"xmqq609b3gi3.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-12-13T17:21:52Z","receivedAt":"2017-12-13T17:22:03Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 12 Dec 2017, at 20:15, Junio C Hamano wrote:\n> \n> Lars Schneider writes:\n> \n>>> You're right, it's my first time using travis CI and I got confused\n>>> about how the .travis.yml works, thanks for catching that. Will\n>>> re-phrase the commit message.\n>> \n>> Szeder is spot on. If you fix up the message, then this patch looks\n>> perfect! Pragmatic (= very good!) idea to combine GITTEXT_POISON with \n>> GIT_TEST_SPLIT_INDEX :-)\n> \n> I am failing to guess the real intent of the smiley here.\n\nNo real reason. I was just happy to see that Travis CI seems to\nbe useful for the Git project.\n\n\n> If split-index code is so easy to break, I do not think it is a good\n> idea to combine it into the poison build. In fact, the poison test\n> is useless on a codebase where other/real breakages are expected to\n> exist, because it is about seeing messages meant for non-humans are\n> not passed to the _() mechanism by sloppy coding, and the way it\n> does so is to corrupt all the messages that come through the _()\n> mechanism. If we do not even produce a message when a correct code\n> _should_ produce one, poison test would catch nothing useful.\n> \n> I wonder if it makes more sense to update ci/run-tests.sh so that\n> its final step is run twice with different settings, like so?\n\nAgreed - I didn't think it through. Let's keep it separate then.\n\nI think your solution points into the right direction.\nRight now we have the following test matrix:\n\n1. Linux - clang\n2. Linux - gcc\n3. Mac - clang\n4. Mac - gcc\n5. Linux - gcc - GET_TEXT_POISION\n6. Linux - gcc - 32bit\n7. Windows\n\nAFAIK your solution would run the split index test for \n1, 2, 3, and 4. I think that is too much.\n\n1 runs the fastest and I would like to keep it that way\nto get a quick \"general\" result. I think only 2 should be\nextended in the way you are suggesting. We could run\nthe tests with different env variables there. What else\ndo we have besides GIT_TEST_SPLIT_INDEX?\n\nWould that work for everyone?\n\n- Lars\n\n\n> ci/run-tests.sh | 1 +\n> 1 file changed, 1 insertion(+)\n> \n> diff --git a/ci/run-tests.sh b/ci/run-tests.sh\n> index f0c743de94..15a5f5a6cc 100755\n> --- a/ci/run-tests.sh\n> +++ b/ci/run-tests.sh\n> @@ -8,3 +8,4 @@\n> mkdir -p $HOME/travis-cache\n> ln -s $HOME/travis-cache/.prove t/.prove\n> make --quiet test\n> +GIT_TEST_SPLIT_INDEX=LetsTryIt make --quiet test\n\n"},{"id":"334764","messageId":"xmqq4louy1d9.fsf@gitster.mtv.corp.google.com","threadId":"47421","inReplyTo":"09E575DE-ED7A-40FA-9A47-0ABB61452C28@gmail.com","subject":"Re: [PATCH 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-12-13T17:38:26Z","receivedAt":"2017-12-13T17:38:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Schneider writes:\n\n> I think your solution points into the right direction.\n> Right now we have the following test matrix:\n>\n> 1. Linux - clang\n> 2. Linux - gcc\n> 3. Mac - clang\n> 4. Mac - gcc\n> 5. Linux - gcc - GET_TEXT_POISION\n> 6. Linux - gcc - 32bit\n> 7. Windows\n>\n> AFAIK your solution would run the split index test for \n> 1, 2, 3, and 4. I think that is too much.\n\nNot that it matters too much, but I meant to add it to 1. and\n6. when I said \"only one of 64-bit build plus 32-bit one\". \n\n> 1 runs the fastest and I would like to keep it that way\n> to get a quick \"general\" result. I think only 2 should be\n> extended in the way you are suggesting. We could run\n> the tests with different env variables there. What else\n> do we have besides GIT_TEST_SPLIT_INDEX?\n>\n> Would that work for everyone?\n\nI am OK to make 2. use split-index (which unfortunately would mean\nwe lose tests without split-index under gcc), or add 2.5 that is a\ncopy of 2. plus split-index.\n"},{"id":"334765","messageId":"8BB646FF-A95A-434D-95E2-FC143B847AE0@gmail.com","threadId":"47421","inReplyTo":"xmqq4louy1d9.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-12-13T17:46:55Z","receivedAt":"2017-12-13T17:47:03Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 13 Dec 2017, at 18:38, Junio C Hamano wrote:\n> \n> Lars Schneider writes:\n> \n>> I think your solution points into the right direction.\n>> Right now we have the following test matrix:\n>> \n>> 1. Linux - clang\n>> 2. Linux - gcc\n>> 3. Mac - clang\n>> 4. Mac - gcc\n>> 5. Linux - gcc - GET_TEXT_POISION\n>> 6. Linux - gcc - 32bit\n>> 7. Windows\n>> \n>> AFAIK your solution would run the split index test for \n>> 1, 2, 3, and 4. I think that is too much.\n> \n> Not that it matters too much, but I meant to add it to 1. and\n> 6. when I said \"only one of 64-bit build plus 32-bit one\". \n\nAh. Sorry, I didn't get that.\n\n\n>> 1 runs the fastest and I would like to keep it that way\n>> to get a quick \"general\" result. I think only 2 should be\n>> extended in the way you are suggesting. We could run\n>> the tests with different env variables there. What else\n>> do we have besides GIT_TEST_SPLIT_INDEX?\n>> \n>> Would that work for everyone?\n> \n> I am OK to make 2. use split-index (which unfortunately would mean\n> we lose tests without split-index under gcc), or add 2.5 that is a\n> copy of 2. plus split-index.\n\n\nI think I experessed myself poorly. As far as I understand it,\nGIT_TEST_SPLIT_INDEX is an environment variable that only needs\nto be set at test time and not at compile time. Is this right?\n\nIf yes, my idea for 2. is as follows:\n- build Git with gcc\n- run tests with \"make --quiet test\"\n- run tests with \"GIT_TEST_SPLIT_INDEX=YesPlease make --quiet test\"\n\nThis should be quicker than a new 2.5 target because we don't need to\nspin up the machine and build Git. Plus, we could run the tests\na few more times with other test flags.\n\n- Lars\n"},{"id":"334787","messageId":"20171213232140.GE24597@hank","threadId":"47421","inReplyTo":"xmqqvahbzn3c.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2017-12-13T23:21:40Z","receivedAt":"2017-12-13T23:20:03Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 12/12, Junio C Hamano wrote:\n> Thomas Gummerer writes:\n> \n> >\n> > The breakages wen the split-index code fails tend to break things in\n> > much more obvious manners than a wrong message, usually git ends up\n> > dying if it gets broken. Both of the bugs that were fixed here would\n> > have been caught with the change in my patch.\n> >\n> > But yeah I can see the argument that it doesn't give us a guarantee\n> > that it catches all things the test suite could catch.\n> \n> I think you misunderstood me. When split index is much easier to\n> break than poison tests, combining them together would hurt the test\n> coverage of poison tests. If you value poison tests much more than\n> how well split index mode works, that is a worse outcome.\n\nSorry, indeed I misunderstood you. I do agree with the above.\n\n> >> I wonder if it makes more sense to update ci/run-tests.sh so that\n> >> its final step is run twice with different settings, like so?\n> >\n> > I kind of wanted to avoid that, because it ends up running the test\n> > suite twice on travis for every test, which seems a bit overkill. But\n> > I don't exactly how worried we are about cycles on travis (I don't\n> > check it very often personally, so I don't really know). If we aren't\n> > worried about cycles what you have below would certainly make sense.\n> \n> I think 64-bit gcc/clang builds tend to cost us about 10-20 minutes\n> each, and 32-bit linux builds about 10 minutes or so. I wonder if\n> it makes sense to do the \"run twice\" for only one of 64-bit builds\n> (perhaps the clang one, as I suspect 32-bit linux one uses gcc) and\n> the 32-bit linux build, and nowhere else.\n\nYeah I'd be happy with something like that. And I see there's some\nmore discussion about it in another part of the thread, will follow up\nthere. Thanks!\n"},{"id":"334788","messageId":"20171213232819.GF24597@hank","threadId":"47421","inReplyTo":"8BB646FF-A95A-434D-95E2-FC143B847AE0@gmail.com","subject":"Re: [PATCH 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2017-12-13T23:28:19Z","receivedAt":"2017-12-13T23:26:41Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 12/13, Lars Schneider wrote:\n> \n> > On 13 Dec 2017, at 18:38, Junio C Hamano wrote:\n> > \n> > Lars Schneider writes:\n> > \n> >> I think your solution points into the right direction.\n> >> Right now we have the following test matrix:\n> >> \n> >> 1. Linux - clang\n> >> 2. Linux - gcc\n> >> 3. Mac - clang\n> >> 4. Mac - gcc\n> >> 5. Linux - gcc - GET_TEXT_POISION\n> >> 6. Linux - gcc - 32bit\n> >> 7. Windows\n> >> \n> >> AFAIK your solution would run the split index test for \n> >> 1, 2, 3, and 4. I think that is too much.\n> > \n> > Not that it matters too much, but I meant to add it to 1. and\n> > 6. when I said \"only one of 64-bit build plus 32-bit one\". \n> \n> Ah. Sorry, I didn't get that.\n> \n> \n> >> 1 runs the fastest and I would like to keep it that way\n> >> to get a quick \"general\" result. I think only 2 should be\n> >> extended in the way you are suggesting. We could run\n> >> the tests with different env variables there. What else\n> >> do we have besides GIT_TEST_SPLIT_INDEX?\n> >> \n> >> Would that work for everyone?\n> > \n> > I am OK to make 2. use split-index (which unfortunately would mean\n> > we lose tests without split-index under gcc), or add 2.5 that is a\n> > copy of 2. plus split-index.\n> \n> \n> I think I experessed myself poorly. As far as I understand it,\n> GIT_TEST_SPLIT_INDEX is an environment variable that only needs\n> to be set at test time and not at compile time. Is this right?\n\nYes, that's correct.\n\n> If yes, my idea for 2. is as follows:\n> - build Git with gcc\n> - run tests with \"make --quiet test\"\n> - run tests with \"GIT_TEST_SPLIT_INDEX=YesPlease make --quiet test\"\n> \n> This should be quicker than a new 2.5 target because we don't need to\n> spin up the machine and build Git. Plus, we could run the tests\n> a few more times with other test flags.\n\nI'd be happy with that. Thanks for the discussion on this, will\nchange this in my re-roll.\n\n> - Lars\n"},{"id":"334952","messageId":"20171217225122.28941-1-t.gummerer@gmail.com","threadId":"47421","inReplyTo":"20171210212202.28231-1-t.gummerer@gmail.com","subject":"[PATCH v2 0/3] fixes for split index mode","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2017-12-17T22:51:19Z","receivedAt":"2017-12-17T22:49:45Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"The previous round was at <20171210212202.28231-1-t.gummerer@gmail.com>.\n\nThanks Brandon, Eric, Szeder, Lars and Junio for the comments on the\nprevious round. \n\nChanges since the previous round:\n\n- pass in the_repository to do_read_index_from, so we don't have to\n special case NULL anymore.\n- free struct repository properly after allocating it in revision.c\n- run the travis tests in the linux-gcc and linux 32-bit builds, to\n avoid clashes with the GETTEXT_POISON tests.\n\nThe travis tests now depend on sg/travis-fixes, as running the split\nindex tests separately gets simplified with the introduction of the\n'$jobname' variable. The script falls back gracefully to not runing\nthe split index test under linux with gcc and only running the\n\"normal\" tests there, while the split index tests would still get run\nin the linux 32-bit build.\n\nBecause it's a \"soft\" dependency I thought having this dependency\nwould be okay, but let me know if you it's getting to complicated and\nwe can use the environment variables set by travis for now, and then\nswitch to the '$jobname' variable when it's merged to next/master.\n\nI kept the usage of struct repository for worktrees, as I didn't hear\nanything back after my last reply in <20171211213948.GC25616@hank>,\nand don't see a better way of doing it.\n\nInterdiff below:\ndiff --git a/.travis.yml b/.travis.yml\nindex c83c766dee..281f101f31 100644\n--- a/.travis.yml\n+++ b/.travis.yml\n@@ -39,7 +39,7 @@ env:\n \n matrix:\n include:\n- - env: GETTEXT_POISON=YesPlease GIT_TEST_SPLIT_INDEX=YesPlease\n+ - env: GETTEXT_POISON=YesPlease\n os: linux\n compiler:\n addons:\ndiff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh\nindex e30fb2cddc..f173c9cf2a 100755\n--- a/ci/run-linux32-build.sh\n+++ b/ci/run-linux32-build.sh\n@@ -27,4 +27,5 @@ linux32 --32bit i386 su -m -l $CI_USER -c '\n cd /usr/src/git &&\n make --jobs=2 &&\n make --quiet test\n+ GIT_TEST_SPLIT_INDEX=YesPlease make --quiet test\n '\ndiff --git a/ci/run-tests.sh b/ci/run-tests.sh\nindex f0c743de94..c7aee5b9ff 100755\n--- a/ci/run-tests.sh\n+++ b/ci/run-tests.sh\n@@ -8,3 +8,7 @@\n mkdir -p $HOME/travis-cache\n ln -s $HOME/travis-cache/.prove t/.prove\n make --quiet test\n+if test \"$jobname\" = \"linux-gcc\"\n+then\n+\tGIT_TEST_SPLIT_INDEX=YesPlease make --quiet test\n+fi\ndiff --git a/read-cache.c b/read-cache.c\nindex 4d5c4ad79b..70357febdc 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1898,10 +1898,7 @@ static int do_read_index_from(struct index_state *istate, const char *path,\n \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n \n \tbase_sha1_hex = sha1_to_hex(split_index->base_sha1);\n-\tif (repo)\n-\t\tbase_path = repo_git_path(repo, \"sharedindex.%s\", base_sha1_hex);\n-\telse\n-\t\tbase_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n+\tbase_path = repo_git_path(repo, \"sharedindex.%s\", base_sha1_hex);\n \tret = do_read_index(split_index->base, base_path, 1);\n \tif (hashcmp(split_index->base_sha1, split_index->base->sha1))\n \t\tdie(\"broken index, expect %s in %s, got %s\",\n@@ -1921,7 +1918,7 @@ int read_index_for_repo(const struct repository *repo)\n \n int read_index_from(struct index_state *istate, const char *path)\n {\n-\treturn do_read_index_from(istate, path, NULL);\n+\treturn do_read_index_from(istate, path, the_repository);\n }\n \n int is_index_unborn(struct index_state *istate)\ndiff --git a/revision.c b/revision.c\nindex 9d8d9b96d1..34e1e4b799 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1358,6 +1358,7 @@ void add_index_objects_to_pending(struct rev_info *revs, unsigned int flags)\n \t\tif (repo_read_index(repo) > 0)\n \t\t\tdo_add_index_objects_to_pending(revs, repo->index);\n \t\tdiscard_index(repo->index);\n+\t\tfree(repo);\n \t}\n \tfree_worktrees(worktrees);\n }\n\nThomas Gummerer (3):\n repository: fix repo_read_index with submodules\n prune: fix pruning with multiple worktrees and split index\n travis: run tests with GIT_TEST_SPLIT_INDEX\n\n cache.h | 1 +\n ci/run-linux32-build.sh | 1 +\n ci/run-tests.sh | 1 +\n read-cache.c | 16 ++++++++++++++--\n repository.c | 13 ++++++++++++-\n repository.h | 2 ++\n revision.c | 14 +++++++++-----\n 7 files changed, 40 insertions(+), 8 deletions(-)\n\n-- \n2.15.1.620.gb9897f4670\n\n"},{"id":"334953","messageId":"20171217225122.28941-3-t.gummerer@gmail.com","threadId":"47421","inReplyTo":"20171217225122.28941-1-t.gummerer@gmail.com","subject":"[PATCH v2 2/3] prune: fix pruning with multiple worktrees and split index","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2017-12-17T22:51:21Z","receivedAt":"2017-12-17T22:49:48Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"be489d02d2 (\"revision.c: --indexed-objects add objects from all\nworktrees\", 2017-08-23) made sure that pruning takes objects from all\nworktrees into account.\n\nIt did that by reading the index of every worktree and adding the\nnecessary index objects to the set of pending objects. The index is\nread by read_index_from. As mentioned in the previous commit,\nread_index_from depends on the CWD for the location of the split index,\nand add_index_objects_to_pending doesn't set that before using\nread_index_from.\n\nInstead of using read_index_from, use repo_read_index, which is aware of\nthe proper paths for the worktree.\n\nThis fixes t5304-prune when ran with GIT_TEST_SPLIT_INDEX set.\n\nSigned-off-by: Thomas Gummerer \n---\n repository.c | 11 +++++++++++\n repository.h | 2 ++\n revision.c | 14 +++++++++-----\n 3 files changed, 22 insertions(+), 5 deletions(-)\n\ndiff --git a/repository.c b/repository.c\nindex 928b1f553d..3c9bfbd1b8 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -2,6 +2,7 @@\n #include \"repository.h\"\n #include \"config.h\"\n #include \"submodule-config.h\"\n+#include \"worktree.h\"\n \n /* The main repository */\n static struct repository the_repo = {\n@@ -146,6 +147,16 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)\n \treturn -1;\n }\n \n+/*\n+ * Initialize 'repo' based on the provided worktree\n+ * Return 0 upon success and a non-zero value upon failure.\n+ */\n+int repo_worktree_init(struct repository *repo, struct worktree *worktree)\n+{\n+\treturn repo_init(repo, get_worktree_git_dir(worktree),\n+\t\t\t worktree->path);\n+}\n+\n /*\n * Initialize 'submodule' as the submodule given by 'path' in parent repository\n * 'superproject'.\ndiff --git a/repository.h b/repository.h\nindex 7f5e24a0a2..2adeb05bf4 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -4,6 +4,7 @@\n struct config_set;\n struct index_state;\n struct submodule_cache;\n+struct worktree;\n \n struct repository {\n \t/* Environment */\n@@ -87,6 +88,7 @@ extern struct repository *the_repository;\n extern void repo_set_gitdir(struct repository *repo, const char *path);\n extern void repo_set_worktree(struct repository *repo, const char *path);\n extern int repo_init(struct repository *repo, const char *gitdir, const char *worktree);\n+extern int repo_worktree_init(struct repository *repo, struct worktree *worktree);\n extern int repo_submodule_init(struct repository *submodule,\n \t\t\t struct repository *superproject,\n \t\t\t const char *path);\ndiff --git a/revision.c b/revision.c\nindex e2e691dd5a..34e1e4b799 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -22,6 +22,7 @@\n #include \"packfile.h\"\n #include \"worktree.h\"\n #include \"argv-array.h\"\n+#include \"repository.h\"\n \n volatile show_early_output_fn_t show_early_output;\n \n@@ -1346,15 +1347,18 @@ void add_index_objects_to_pending(struct rev_info *revs, unsigned int flags)\n \tworktrees = get_worktrees(0);\n \tfor (p = worktrees; *p; p++) {\n \t\tstruct worktree *wt = *p;\n-\t\tstruct index_state istate = { NULL };\n+\t\tstruct repository *repo;\n \n+\t\trepo = xmalloc(sizeof(struct repository));\n \t\tif (wt->is_current)\n \t\t\tcontinue; /* current index already taken care of */\n+\t\tif (repo_worktree_init(repo, wt))\n+\t\t\tBUG(\"couldn't initialize repository object from worktree\");\n \n-\t\tif (read_index_from(&istate,\n-\t\t\t\t worktree_git_path(wt, \"index\")) > 0)\n-\t\t\tdo_add_index_objects_to_pending(revs, &istate);\n-\t\tdiscard_index(&istate);\n+\t\tif (repo_read_index(repo) > 0)\n+\t\t\tdo_add_index_objects_to_pending(revs, repo->index);\n+\t\tdiscard_index(repo->index);\n+\t\tfree(repo);\n \t}\n \tfree_worktrees(worktrees);\n }\n-- \n2.15.1.620.gb9897f4670\n\n"},{"id":"334954","messageId":"20171217225122.28941-2-t.gummerer@gmail.com","threadId":"47421","inReplyTo":"20171217225122.28941-1-t.gummerer@gmail.com","subject":"[PATCH v2 1/3] repository: fix repo_read_index with submodules","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2017-12-17T22:51:20Z","receivedAt":"2017-12-17T22:49:51Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"repo_read_index calls read_index_from, which takes an path argument for\nthe location of the index file. For the split index however it relies\non the current working directory to construct the path using git_path.\n\nrepo_read_index calls read_index_from with the full path for the index\nfile, however it doesn't change the cwd, so when split index mode is\nturned on, read_index_from can't find the file for the split index.\n\nFor example t3007-ls-files-recurse-submodules.sh was broken with\nGIT_TEST_SPLIT_INDEX set in 188dce131f (\"ls-files: use repository\nobject\", 2017-06-22), and t7814-grep-recurse-submodules.sh was also\nbroken in a similar manner, probably by introducing struct repository\nthere, although I didn't track down the exact commit for that.\n\nFix this by introducing a new read_index_for_repo function, which knows\nabout the correct paths for the submodules.\n\nThe alternative would have been to make the callers pass in the base\npath for the split index, however that ended up being more complicated,\nand I think we want to converge towards using struct repository for\nthings like these anyway.\n\nSigned-off-by: Thomas Gummerer \n---\n cache.h | 1 +\n read-cache.c | 16 ++++++++++++++--\n repository.c | 2 +-\n 3 files changed, 16 insertions(+), 3 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex cb5db7bf83..d42bea1ef7 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -614,6 +614,7 @@ extern int read_index_preload(struct index_state *, const struct pathspec *paths\n extern int do_read_index(struct index_state *istate, const char *path,\n \t\t\t int must_exist); /* for testting only! */\n extern int read_index_from(struct index_state *, const char *path);\n+extern int read_index_for_repo(const struct repository *);\n extern int is_index_unborn(struct index_state *);\n extern int read_index_unmerged(struct index_state *);\n \ndiff --git a/read-cache.c b/read-cache.c\nindex 2eb81a66b9..70357febdc 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -20,6 +20,7 @@\n #include \"split-index.h\"\n #include \"utf8.h\"\n #include \"fsmonitor.h\"\n+#include \"repository.h\"\n \n /* Mask for the name length in ce_flags in the on-disk index */\n \n@@ -1871,7 +1872,8 @@ static void freshen_shared_index(char *base_sha1_hex, int warn)\n \tfree(shared_index);\n }\n \n-int read_index_from(struct index_state *istate, const char *path)\n+static int do_read_index_from(struct index_state *istate, const char *path,\n+\t\t\t const struct repository *repo)\n {\n \tstruct split_index *split_index;\n \tint ret;\n@@ -1896,7 +1898,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n \n \tbase_sha1_hex = sha1_to_hex(split_index->base_sha1);\n-\tbase_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n+\tbase_path = repo_git_path(repo, \"sharedindex.%s\", base_sha1_hex);\n \tret = do_read_index(split_index->base, base_path, 1);\n \tif (hashcmp(split_index->base_sha1, split_index->base->sha1))\n \t\tdie(\"broken index, expect %s in %s, got %s\",\n@@ -1909,6 +1911,16 @@ int read_index_from(struct index_state *istate, const char *path)\n \treturn ret;\n }\n \n+int read_index_for_repo(const struct repository *repo)\n+{\n+\treturn do_read_index_from(repo->index, repo->index_file, repo);\n+}\n+\n+int read_index_from(struct index_state *istate, const char *path)\n+{\n+\treturn do_read_index_from(istate, path, the_repository);\n+}\n+\n int is_index_unborn(struct index_state *istate)\n {\n \treturn (!istate->cache_nr && !istate->timestamp.sec);\ndiff --git a/repository.c b/repository.c\nindex bb2fae5446..928b1f553d 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -229,5 +229,5 @@ int repo_read_index(struct repository *repo)\n \tif (!repo->index)\n \t\trepo->index = xcalloc(1, sizeof(*repo->index));\n \n-\treturn read_index_from(repo->index, repo->index_file);\n+\treturn read_index_for_repo(repo);\n }\n-- \n2.15.1.620.gb9897f4670\n\n"},{"id":"334955","messageId":"20171217225122.28941-4-t.gummerer@gmail.com","threadId":"47421","inReplyTo":"20171217225122.28941-1-t.gummerer@gmail.com","subject":"[PATCH v2 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2017-12-17T22:51:22Z","receivedAt":"2017-12-17T22:49:52Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Split index mode only has a few dedicated tests, but as the index is\ninvolved in nearly every git operation, this doesn't quite cover all the\nways repositories with split index can break. To use split index mode\nthroughout the test suite a GIT_TEST_SPLIT_INDEX environment variable\ncan be set, which makes git split the index at random and thus\nexcercises the functionality much more thoroughly.\n\nAs this is not turned on by default, it is not executed nearly as often\nas the test suite is run, so occationally breakages slip through. Try\nto counteract that by running the test suite with GIT_TEST_SPLIT_INDEX\nmode turned on on travis.\n\nTo avoid using too many cycles on travis only run split index mode in\nthe linux-gcc and the linux 32-bit gcc targets. The Linux builds were\nchosen over the Mac OS builds because they tend to be much faster to\ncomplete.\n\nThe linux gcc build was chosen over the linux clang build because the\nlinux clang build is the fastest build, so it can serve as an early\nindicator if something is broken and we want to avoid spending the extra\ncycles of running the test suite twice for that.\n\nHelped-by: Lars Schneider \nHelped-by: Junio C Hamano \nSigned-off-by: Thomas Gummerer \n---\n ci/run-linux32-build.sh | 1 +\n ci/run-tests.sh | 4 ++++\n 2 files changed, 5 insertions(+)\n\ndiff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh\nindex e30fb2cddc..f173c9cf2a 100755\n--- a/ci/run-linux32-build.sh\n+++ b/ci/run-linux32-build.sh\n@@ -27,4 +27,5 @@ linux32 --32bit i386 su -m -l $CI_USER -c '\n cd /usr/src/git &&\n make --jobs=2 &&\n make --quiet test\n+ GIT_TEST_SPLIT_INDEX=YesPlease make --quiet test\n '\ndiff --git a/ci/run-tests.sh b/ci/run-tests.sh\nindex f0c743de94..c7aee5b9ff 100755\n--- a/ci/run-tests.sh\n+++ b/ci/run-tests.sh\n@@ -8,3 +8,7 @@\n mkdir -p $HOME/travis-cache\n ln -s $HOME/travis-cache/.prove t/.prove\n make --quiet test\n+if test \"$jobname\" = \"linux-gcc\"\n+then\n+\tGIT_TEST_SPLIT_INDEX=YesPlease make --quiet test\n+fi\n-- \n2.15.1.620.gb9897f4670\n\n"},{"id":"334972","messageId":"20171218180126.GA147973@google.com","threadId":"47421","inReplyTo":"20171217225122.28941-2-t.gummerer@gmail.com","subject":"Re: [PATCH v2 1/3] repository: fix repo_read_index with submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-12-18T18:01:26Z","receivedAt":"2017-12-18T18:01:39Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 12/17, Thomas Gummerer wrote:\n> repo_read_index calls read_index_from, which takes an path argument for\n> the location of the index file. For the split index however it relies\n\n> on the current working directory to construct the path using git_path.\n\nThis line isn't actually true and should probably be fixed. git_path\ndoesn't rely on the CWD but rather it relies on the gitdir of the main\nrepository (the_repository).\n\n> \n> repo_read_index calls read_index_from with the full path for the index\n> file, however it doesn't change the cwd, so when split index mode is\n> turned on, read_index_from can't find the file for the split index.\n> \n> For example t3007-ls-files-recurse-submodules.sh was broken with\n> GIT_TEST_SPLIT_INDEX set in 188dce131f (\"ls-files: use repository\n> object\", 2017-06-22), and t7814-grep-recurse-submodules.sh was also\n> broken in a similar manner, probably by introducing struct repository\n> there, although I didn't track down the exact commit for that.\n> \n> Fix this by introducing a new read_index_for_repo function, which knows\n> about the correct paths for the submodules.\n> \n> The alternative would have been to make the callers pass in the base\n> path for the split index, however that ended up being more complicated,\n> and I think we want to converge towards using struct repository for\n> things like these anyway.\n> \n> Signed-off-by: Thomas Gummerer \n> ---\n> cache.h | 1 +\n> read-cache.c | 16 ++++++++++++++--\n> repository.c | 2 +-\n> 3 files changed, 16 insertions(+), 3 deletions(-)\n> \n> diff --git a/cache.h b/cache.h\n> index cb5db7bf83..d42bea1ef7 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -614,6 +614,7 @@ extern int read_index_preload(struct index_state *, const struct pathspec *paths\n> extern int do_read_index(struct index_state *istate, const char *path,\n> \t\t\t int must_exist); /* for testting only! */\n> extern int read_index_from(struct index_state *, const char *path);\n> +extern int read_index_for_repo(const struct repository *);\n> extern int is_index_unborn(struct index_state *);\n> extern int read_index_unmerged(struct index_state *);\n> \n> diff --git a/read-cache.c b/read-cache.c\n> index 2eb81a66b9..70357febdc 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -20,6 +20,7 @@\n> #include \"split-index.h\"\n> #include \"utf8.h\"\n> #include \"fsmonitor.h\"\n> +#include \"repository.h\"\n> \n> /* Mask for the name length in ce_flags in the on-disk index */\n> \n> @@ -1871,7 +1872,8 @@ static void freshen_shared_index(char *base_sha1_hex, int warn)\n> \tfree(shared_index);\n> }\n> \n> -int read_index_from(struct index_state *istate, const char *path)\n> +static int do_read_index_from(struct index_state *istate, const char *path,\n> +\t\t\t const struct repository *repo)\n> {\n> \tstruct split_index *split_index;\n> \tint ret;\n> @@ -1896,7 +1898,7 @@ int read_index_from(struct index_state *istate, const char *path)\n> \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n> \n> \tbase_sha1_hex = sha1_to_hex(split_index->base_sha1);\n> -\tbase_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n> +\tbase_path = repo_git_path(repo, \"sharedindex.%s\", base_sha1_hex);\n> \tret = do_read_index(split_index->base, base_path, 1);\n> \tif (hashcmp(split_index->base_sha1, split_index->base->sha1))\n> \t\tdie(\"broken index, expect %s in %s, got %s\",\n> @@ -1909,6 +1911,16 @@ int read_index_from(struct index_state *istate, const char *path)\n> \treturn ret;\n> }\n> \n> +int read_index_for_repo(const struct repository *repo)\n> +{\n> +\treturn do_read_index_from(repo->index, repo->index_file, repo);\n> +}\n\n> +\n> +int read_index_from(struct index_state *istate, const char *path)\n> +{\n> +\treturn do_read_index_from(istate, path, the_repository);\n> +}\n\nThis looks fine, though I wonder what the point of passing in the index\nfile even was since we end just ended up reading the 'sharedindex' file based\non the git path. I'm just curious about how this function evolved.\n\n> +\n> int is_index_unborn(struct index_state *istate)\n> {\n> \treturn (!istate->cache_nr && !istate->timestamp.sec);\n> diff --git a/repository.c b/repository.c\n> index bb2fae5446..928b1f553d 100644\n> --- a/repository.c\n> +++ b/repository.c\n> @@ -229,5 +229,5 @@ int repo_read_index(struct repository *repo)\n> \tif (!repo->index)\n> \t\trepo->index = xcalloc(1, sizeof(*repo->index));\n> \n> -\treturn read_index_from(repo->index, repo->index_file);\n> +\treturn read_index_for_repo(repo);\n> }\n> -- \n> 2.15.1.620.gb9897f4670\n> \n\n-- \nBrandon Williams\n"},{"id":"334974","messageId":"CA6C93AD-B24D-43A2-8AAA-DE98A4C9E719@gmail.com","threadId":"47421","inReplyTo":"20171217225122.28941-4-t.gummerer@gmail.com","subject":"Re: [PATCH v2 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-12-18T18:16:01Z","receivedAt":"2017-12-18T18:16:14Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 17 Dec 2017, at 23:51, Thomas Gummerer wrote:\n> \n> Split index mode only has a few dedicated tests, but as the index is\n> involved in nearly every git operation, this doesn't quite cover all the\n> ways repositories with split index can break. To use split index mode\n> throughout the test suite a GIT_TEST_SPLIT_INDEX environment variable\n> can be set, which makes git split the index at random and thus\n> excercises the functionality much more thoroughly.\n> \n> As this is not turned on by default, it is not executed nearly as often\n> as the test suite is run, so occationally breakages slip through. Try\n> to counteract that by running the test suite with GIT_TEST_SPLIT_INDEX\n> mode turned on on travis.\n> \n> To avoid using too many cycles on travis only run split index mode in\n> the linux-gcc and the linux 32-bit gcc targets.\n\nI am surprised to see the split index mode test for the linux 32-bit\ntarget. Is it likely that a split index bug appears only on 32-bit? \nWouldn't the linux-gcc target be sufficient to save resources/time?\n\n\n> The Linux builds were\n> chosen over the Mac OS builds because they tend to be much faster to\n> complete.\n> \n> The linux gcc build was chosen over the linux clang build because the\n> linux clang build is the fastest build, so it can serve as an early\n> indicator if something is broken and we want to avoid spending the extra\n> cycles of running the test suite twice for that.\n> \n> Helped-by: Lars Schneider \n> Helped-by: Junio C Hamano \n> Signed-off-by: Thomas Gummerer \n> ---\n> ci/run-linux32-build.sh | 1 +\n> ci/run-tests.sh | 4 ++++\n> 2 files changed, 5 insertions(+)\n> \n> diff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh\n> index e30fb2cddc..f173c9cf2a 100755\n> --- a/ci/run-linux32-build.sh\n> +++ b/ci/run-linux32-build.sh\n> @@ -27,4 +27,5 @@ linux32 --32bit i386 su -m -l $CI_USER -c '\n> cd /usr/src/git &&\n> make --jobs=2 &&\n> make --quiet test\n> + GIT_TEST_SPLIT_INDEX=YesPlease make --quiet test\n> '\n> diff --git a/ci/run-tests.sh b/ci/run-tests.sh\n> index f0c743de94..c7aee5b9ff 100755\n> --- a/ci/run-tests.sh\n> +++ b/ci/run-tests.sh\n> @@ -8,3 +8,7 @@\n> mkdir -p $HOME/travis-cache\n> ln -s $HOME/travis-cache/.prove t/.prove\n> make --quiet test\n> +if test \"$jobname\" = \"linux-gcc\"\n> +then\n> +\tGIT_TEST_SPLIT_INDEX=YesPlease make --quiet test\n> +fi\n\nFor now I think that looks good. Maybe we could define additional test \nconfigurations with an environment variable. That could be an array variable\ndefined in the lib-travis.ci \"case\" statement:\nhttps://github.com/git/git/blob/1229713f78cd2883798e95f33c19c81b523413fd/ci/lib-travisci.sh#L42-L65\n\n\n- Lars"},{"id":"334975","messageId":"20171218181913.GB147973@google.com","threadId":"47421","inReplyTo":"20171217225122.28941-3-t.gummerer@gmail.com","subject":"Re: [PATCH v2 2/3] prune: fix pruning with multiple worktrees and split index","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-12-18T18:19:13Z","receivedAt":"2017-12-18T18:19:27Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 12/17, Thomas Gummerer wrote:\n> be489d02d2 (\"revision.c: --indexed-objects add objects from all\n> worktrees\", 2017-08-23) made sure that pruning takes objects from all\n> worktrees into account.\n> \n> It did that by reading the index of every worktree and adding the\n> necessary index objects to the set of pending objects. The index is\n> read by read_index_from. As mentioned in the previous commit,\n> read_index_from depends on the CWD for the location of the split index,\n\nAs I mentioned before this doesn't actually depend on the CWD but\nrather the per-worktree gitdir.\n\n> and add_index_objects_to_pending doesn't set that before using\n> read_index_from.\n> \n> Instead of using read_index_from, use repo_read_index, which is aware of\n> the proper paths for the worktree.\n> \n> This fixes t5304-prune when ran with GIT_TEST_SPLIT_INDEX set.\n> \n> Signed-off-by: Thomas Gummerer \n> ---\n> repository.c | 11 +++++++++++\n> repository.h | 2 ++\n> revision.c | 14 +++++++++-----\n> 3 files changed, 22 insertions(+), 5 deletions(-)\n> \n> diff --git a/repository.c b/repository.c\n> index 928b1f553d..3c9bfbd1b8 100644\n> --- a/repository.c\n> +++ b/repository.c\n> @@ -2,6 +2,7 @@\n> #include \"repository.h\"\n> #include \"config.h\"\n> #include \"submodule-config.h\"\n> +#include \"worktree.h\"\n> \n> /* The main repository */\n> static struct repository the_repo = {\n> @@ -146,6 +147,16 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)\n> \treturn -1;\n> }\n> \n> +/*\n> + * Initialize 'repo' based on the provided worktree\n> + * Return 0 upon success and a non-zero value upon failure.\n> + */\n> +int repo_worktree_init(struct repository *repo, struct worktree *worktree)\n> +{\n> +\treturn repo_init(repo, get_worktree_git_dir(worktree),\n> +\t\t\t worktree->path);\n\nI still feel very unsettled about this and don't think its a good idea.\nget_worktree_git_dir depends implicitly on the global the_repository\nobject and I would like to avoid relying on it for an initialization\nfunction like this.\n\n> +}\n> +\n> /*\n> * Initialize 'submodule' as the submodule given by 'path' in parent repository\n> * 'superproject'.\n> diff --git a/repository.h b/repository.h\n> index 7f5e24a0a2..2adeb05bf4 100644\n> --- a/repository.h\n> +++ b/repository.h\n> @@ -4,6 +4,7 @@\n> struct config_set;\n> struct index_state;\n> struct submodule_cache;\n> +struct worktree;\n> \n> struct repository {\n> \t/* Environment */\n> @@ -87,6 +88,7 @@ extern struct repository *the_repository;\n> extern void repo_set_gitdir(struct repository *repo, const char *path);\n> extern void repo_set_worktree(struct repository *repo, const char *path);\n> extern int repo_init(struct repository *repo, const char *gitdir, const char *worktree);\n> +extern int repo_worktree_init(struct repository *repo, struct worktree *worktree);\n> extern int repo_submodule_init(struct repository *submodule,\n> \t\t\t struct repository *superproject,\n> \t\t\t const char *path);\n> diff --git a/revision.c b/revision.c\n> index e2e691dd5a..34e1e4b799 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -22,6 +22,7 @@\n> #include \"packfile.h\"\n> #include \"worktree.h\"\n> #include \"argv-array.h\"\n> +#include \"repository.h\"\n> \n> volatile show_early_output_fn_t show_early_output;\n> \n> @@ -1346,15 +1347,18 @@ void add_index_objects_to_pending(struct rev_info *revs, unsigned int flags)\n> \tworktrees = get_worktrees(0);\n> \tfor (p = worktrees; *p; p++) {\n> \t\tstruct worktree *wt = *p;\n> -\t\tstruct index_state istate = { NULL };\n> +\t\tstruct repository *repo;\n> \n> +\t\trepo = xmalloc(sizeof(struct repository));\n> \t\tif (wt->is_current)\n> \t\t\tcontinue; /* current index already taken care of */\n> +\t\tif (repo_worktree_init(repo, wt))\n> +\t\t\tBUG(\"couldn't initialize repository object from worktree\");\n> \n> -\t\tif (read_index_from(&istate,\n> -\t\t\t\t worktree_git_path(wt, \"index\")) > 0)\n\nOk, after thinking this through a bit more I think a better approach may\nbe to restructure the call to read_index_from to take in both an index\nfile as well as the explicit gitdir to use when constructing a path to\nthe sharedindex file. That way you can fix this for worktrees and\nsubmodules without having to pass in a repository object to the logic\nwhich is reading an index file as well as avoiding needing to init a\nrepository object for every worktree.\n\nSo you could rework the first patch to do something like:\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 2eb81a66b..3a04b5567 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1863,20 +1863,19 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n * This way, shared index can be removed if they have not been used\n * for some time.\n */\n-static void freshen_shared_index(char *base_sha1_hex, int warn)\n+static void freshen_shared_index(const char *shared_index, int warn)\n {\n-\tchar *shared_index = git_pathdup(\"sharedindex.%s\", base_sha1_hex);\n \tif (!check_and_freshen_file(shared_index, 1) && warn)\n \t\twarning(\"could not freshen shared index '%s'\", shared_index);\n-\tfree(shared_index);\n }\n \n-int read_index_from(struct index_state *istate, const char *path)\n+int read_index_from(struct index_state *istate, const char *path,\n+\t\t const char *gitdir)\n {\n \tstruct split_index *split_index;\n \tint ret;\n \tchar *base_sha1_hex;\n-\tconst char *base_path;\n+\tchar *base_path;\n \n \t/* istate->initialized covers both .git/index and .git/sharedindex.xxx */\n \tif (istate->initialized)\n@@ -1896,14 +1895,14 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n \n \tbase_sha1_hex = sha1_to_hex(split_index->base_sha1);\n-\tbase_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n+\tbase_path = xstrfmt(\"%s/sharedindex.%s\", gitdir, base_sha1_hex);\n \tret = do_read_index(split_index->base, base_path, 1);\n \tif (hashcmp(split_index->base_sha1, split_index->base->sha1))\n \t\tdie(\"broken index, expect %s in %s, got %s\",\n \t\t base_sha1_hex, base_path,\n \t\t sha1_to_hex(split_index->base->sha1));\n \n-\tfreshen_shared_index(base_sha1_hex, 0);\n+\tfreshen_shared_index(base_path, 0);\n \tmerge_base_index(istate);\n \tpost_read_index_from(istate);\n+ free(base_path);\n \treturn ret;\n\n\n-- \nBrandon Williams\n"},{"id":"335002","messageId":"20171218230511.GB2641@hank","threadId":"47421","inReplyTo":"20171218180126.GA147973@google.com","subject":"Re: [PATCH v2 1/3] repository: fix repo_read_index with submodules","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2017-12-18T23:05:11Z","receivedAt":"2017-12-18T23:03:29Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 12/18, Brandon Williams wrote:\n> On 12/17, Thomas Gummerer wrote:\n> > repo_read_index calls read_index_from, which takes an path argument for\n> > the location of the index file. For the split index however it relies\n> \n> > on the current working directory to construct the path using git_path.\n> \n> This line isn't actually true and should probably be fixed. git_path\n> doesn't rely on the CWD but rather it relies on the gitdir of the main\n> repository (the_repository).\n\nRight, let me fix that. Thanks!\n\n> > \n> > repo_read_index calls read_index_from with the full path for the index\n> > file, however it doesn't change the cwd, so when split index mode is\n> > turned on, read_index_from can't find the file for the split index.\n> > \n> > For example t3007-ls-files-recurse-submodules.sh was broken with\n> > GIT_TEST_SPLIT_INDEX set in 188dce131f (\"ls-files: use repository\n> > object\", 2017-06-22), and t7814-grep-recurse-submodules.sh was also\n> > broken in a similar manner, probably by introducing struct repository\n> > there, although I didn't track down the exact commit for that.\n> > \n> > Fix this by introducing a new read_index_for_repo function, which knows\n> > about the correct paths for the submodules.\n> > \n> > The alternative would have been to make the callers pass in the base\n> > path for the split index, however that ended up being more complicated,\n> > and I think we want to converge towards using struct repository for\n> > things like these anyway.\n> > \n> > Signed-off-by: Thomas Gummerer \n> > ---\n> > cache.h | 1 +\n> > read-cache.c | 16 ++++++++++++++--\n> > repository.c | 2 +-\n> > 3 files changed, 16 insertions(+), 3 deletions(-)\n> > \n> > diff --git a/cache.h b/cache.h\n> > index cb5db7bf83..d42bea1ef7 100644\n> > --- a/cache.h\n> > +++ b/cache.h\n> > @@ -614,6 +614,7 @@ extern int read_index_preload(struct index_state *, const struct pathspec *paths\n> > extern int do_read_index(struct index_state *istate, const char *path,\n> > \t\t\t int must_exist); /* for testting only! */\n> > extern int read_index_from(struct index_state *, const char *path);\n> > +extern int read_index_for_repo(const struct repository *);\n> > extern int is_index_unborn(struct index_state *);\n> > extern int read_index_unmerged(struct index_state *);\n> > \n> > diff --git a/read-cache.c b/read-cache.c\n> > index 2eb81a66b9..70357febdc 100644\n> > --- a/read-cache.c\n> > +++ b/read-cache.c\n> > @@ -20,6 +20,7 @@\n> > #include \"split-index.h\"\n> > #include \"utf8.h\"\n> > #include \"fsmonitor.h\"\n> > +#include \"repository.h\"\n> > \n> > /* Mask for the name length in ce_flags in the on-disk index */\n> > \n> > @@ -1871,7 +1872,8 @@ static void freshen_shared_index(char *base_sha1_hex, int warn)\n> > \tfree(shared_index);\n> > }\n> > \n> > -int read_index_from(struct index_state *istate, const char *path)\n> > +static int do_read_index_from(struct index_state *istate, const char *path,\n> > +\t\t\t const struct repository *repo)\n> > {\n> > \tstruct split_index *split_index;\n> > \tint ret;\n> > @@ -1896,7 +1898,7 @@ int read_index_from(struct index_state *istate, const char *path)\n> > \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n> > \n> > \tbase_sha1_hex = sha1_to_hex(split_index->base_sha1);\n> > -\tbase_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n> > +\tbase_path = repo_git_path(repo, \"sharedindex.%s\", base_sha1_hex);\n> > \tret = do_read_index(split_index->base, base_path, 1);\n> > \tif (hashcmp(split_index->base_sha1, split_index->base->sha1))\n> > \t\tdie(\"broken index, expect %s in %s, got %s\",\n> > @@ -1909,6 +1911,16 @@ int read_index_from(struct index_state *istate, const char *path)\n> > \treturn ret;\n> > }\n> > \n> > +int read_index_for_repo(const struct repository *repo)\n> > +{\n> > +\treturn do_read_index_from(repo->index, repo->index_file, repo);\n> > +}\n> \n> > +\n> > +int read_index_from(struct index_state *istate, const char *path)\n> > +{\n> > +\treturn do_read_index_from(istate, path, the_repository);\n> > +}\n> \n> This looks fine, though I wonder what the point of passing in the index\n> file even was since we end just ended up reading the 'sharedindex' file based\n> on the git path. I'm just curious about how this function evolved.\n\nThere are some callsites that are using an index different form\n$gitdir/index, or even GIT_INDEX_FILE. e.g. see builtin/am.c [*1*],\nwhich uses it's own 'patch-merge-index' in the am state directory for\nits internal operations.\n\nThe split index mode was later bolted on, and the sharedindex.xxxx\nwould always go in $gitdir for the repository. Others probably know\nquite a bit more about this, while I'm always interested in index\nrelated things as that's how I got started with the git project, I\ncouldn't follow all the conversations that were going on there.\n\n*1*: https://github.com/gitster/git/blob/52015aaf9d19c97b52c47c7046058e6d029ff856/builtin/am.c#L1844\n\n> > +\n> > int is_index_unborn(struct index_state *istate)\n> > {\n> > \treturn (!istate->cache_nr && !istate->timestamp.sec);\n> > diff --git a/repository.c b/repository.c\n> > index bb2fae5446..928b1f553d 100644\n> > --- a/repository.c\n> > +++ b/repository.c\n> > @@ -229,5 +229,5 @@ int repo_read_index(struct repository *repo)\n> > \tif (!repo->index)\n> > \t\trepo->index = xcalloc(1, sizeof(*repo->index));\n> > \n> > -\treturn read_index_from(repo->index, repo->index_file);\n> > +\treturn read_index_for_repo(repo);\n> > }\n> > -- \n> > 2.15.1.620.gb9897f4670\n> > \n> \n> -- \n> Brandon Williams\n"},{"id":"335003","messageId":"20171218230541.GC147973@google.com","threadId":"47421","inReplyTo":"20171218230511.GB2641@hank","subject":"Re: [PATCH v2 1/3] repository: fix repo_read_index with submodules","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-12-18T23:05:41Z","receivedAt":"2017-12-18T23:05:51Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 12/18, Thomas Gummerer wrote:\n> On 12/18, Brandon Williams wrote:\n> > On 12/17, Thomas Gummerer wrote:\n> > > repo_read_index calls read_index_from, which takes an path argument for\n> > > the location of the index file. For the split index however it relies\n> > \n> > > on the current working directory to construct the path using git_path.\n> > \n> > This line isn't actually true and should probably be fixed. git_path\n> > doesn't rely on the CWD but rather it relies on the gitdir of the main\n> > repository (the_repository).\n> \n> Right, let me fix that. Thanks!\n> \n> > > \n> > > repo_read_index calls read_index_from with the full path for the index\n> > > file, however it doesn't change the cwd, so when split index mode is\n> > > turned on, read_index_from can't find the file for the split index.\n> > > \n> > > For example t3007-ls-files-recurse-submodules.sh was broken with\n> > > GIT_TEST_SPLIT_INDEX set in 188dce131f (\"ls-files: use repository\n> > > object\", 2017-06-22), and t7814-grep-recurse-submodules.sh was also\n> > > broken in a similar manner, probably by introducing struct repository\n> > > there, although I didn't track down the exact commit for that.\n> > > \n> > > Fix this by introducing a new read_index_for_repo function, which knows\n> > > about the correct paths for the submodules.\n> > > \n> > > The alternative would have been to make the callers pass in the base\n> > > path for the split index, however that ended up being more complicated,\n> > > and I think we want to converge towards using struct repository for\n> > > things like these anyway.\n> > > \n> > > Signed-off-by: Thomas Gummerer \n> > > ---\n> > > cache.h | 1 +\n> > > read-cache.c | 16 ++++++++++++++--\n> > > repository.c | 2 +-\n> > > 3 files changed, 16 insertions(+), 3 deletions(-)\n> > > \n> > > diff --git a/cache.h b/cache.h\n> > > index cb5db7bf83..d42bea1ef7 100644\n> > > --- a/cache.h\n> > > +++ b/cache.h\n> > > @@ -614,6 +614,7 @@ extern int read_index_preload(struct index_state *, const struct pathspec *paths\n> > > extern int do_read_index(struct index_state *istate, const char *path,\n> > > \t\t\t int must_exist); /* for testting only! */\n> > > extern int read_index_from(struct index_state *, const char *path);\n> > > +extern int read_index_for_repo(const struct repository *);\n> > > extern int is_index_unborn(struct index_state *);\n> > > extern int read_index_unmerged(struct index_state *);\n> > > \n> > > diff --git a/read-cache.c b/read-cache.c\n> > > index 2eb81a66b9..70357febdc 100644\n> > > --- a/read-cache.c\n> > > +++ b/read-cache.c\n> > > @@ -20,6 +20,7 @@\n> > > #include \"split-index.h\"\n> > > #include \"utf8.h\"\n> > > #include \"fsmonitor.h\"\n> > > +#include \"repository.h\"\n> > > \n> > > /* Mask for the name length in ce_flags in the on-disk index */\n> > > \n> > > @@ -1871,7 +1872,8 @@ static void freshen_shared_index(char *base_sha1_hex, int warn)\n> > > \tfree(shared_index);\n> > > }\n> > > \n> > > -int read_index_from(struct index_state *istate, const char *path)\n> > > +static int do_read_index_from(struct index_state *istate, const char *path,\n> > > +\t\t\t const struct repository *repo)\n> > > {\n> > > \tstruct split_index *split_index;\n> > > \tint ret;\n> > > @@ -1896,7 +1898,7 @@ int read_index_from(struct index_state *istate, const char *path)\n> > > \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n> > > \n> > > \tbase_sha1_hex = sha1_to_hex(split_index->base_sha1);\n> > > -\tbase_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n> > > +\tbase_path = repo_git_path(repo, \"sharedindex.%s\", base_sha1_hex);\n> > > \tret = do_read_index(split_index->base, base_path, 1);\n> > > \tif (hashcmp(split_index->base_sha1, split_index->base->sha1))\n> > > \t\tdie(\"broken index, expect %s in %s, got %s\",\n> > > @@ -1909,6 +1911,16 @@ int read_index_from(struct index_state *istate, const char *path)\n> > > \treturn ret;\n> > > }\n> > > \n> > > +int read_index_for_repo(const struct repository *repo)\n> > > +{\n> > > +\treturn do_read_index_from(repo->index, repo->index_file, repo);\n> > > +}\n> > \n> > > +\n> > > +int read_index_from(struct index_state *istate, const char *path)\n> > > +{\n> > > +\treturn do_read_index_from(istate, path, the_repository);\n> > > +}\n> > \n> > This looks fine, though I wonder what the point of passing in the index\n> > file even was since we end just ended up reading the 'sharedindex' file based\n> > on the git path. I'm just curious about how this function evolved.\n> \n> There are some callsites that are using an index different form\n> $gitdir/index, or even GIT_INDEX_FILE. e.g. see builtin/am.c [*1*],\n> which uses it's own 'patch-merge-index' in the am state directory for\n> its internal operations.\n> \n> The split index mode was later bolted on, and the sharedindex.xxxx\n> would always go in $gitdir for the repository. Others probably know\n> quite a bit more about this, while I'm always interested in index\n> related things as that's how I got started with the git project, I\n> couldn't follow all the conversations that were going on there.\n> \n> *1*: https://github.com/gitster/git/blob/52015aaf9d19c97b52c47c7046058e6d029ff856/builtin/am.c#L1844\n\nThanks for the explanation and pointer! :) \n\n> \n> > > +\n> > > int is_index_unborn(struct index_state *istate)\n> > > {\n> > > \treturn (!istate->cache_nr && !istate->timestamp.sec);\n> > > diff --git a/repository.c b/repository.c\n> > > index bb2fae5446..928b1f553d 100644\n> > > --- a/repository.c\n> > > +++ b/repository.c\n> > > @@ -229,5 +229,5 @@ int repo_read_index(struct repository *repo)\n> > > \tif (!repo->index)\n> > > \t\trepo->index = xcalloc(1, sizeof(*repo->index));\n> > > \n> > > -\treturn read_index_from(repo->index, repo->index_file);\n> > > +\treturn read_index_for_repo(repo);\n> > > }\n> > > -- \n> > > 2.15.1.620.gb9897f4670\n> > > \n> > \n> > -- \n> > Brandon Williams\n\n-- \nBrandon Williams\n"},{"id":"335797","messageId":"20180103221849.GC2641@hank","threadId":"47421","inReplyTo":"20171218181913.GB147973@google.com","subject":"Re: [PATCH v2 2/3] prune: fix pruning with multiple worktrees and split index","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2018-01-03T22:18:49Z","receivedAt":"2018-01-03T22:16:49Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"[sorry for the late reply. I was on Christmas holidays until today\nand am still catching up on the mailing list. It will probably take\nme untill the weekend to send a re-roll]\n\nOn 12/18, Brandon Williams wrote:\n> On 12/17, Thomas Gummerer wrote:\n> > be489d02d2 (\"revision.c: --indexed-objects add objects from all\n> > worktrees\", 2017-08-23) made sure that pruning takes objects from all\n> > worktrees into account.\n> > \n> > It did that by reading the index of every worktree and adding the\n> > necessary index objects to the set of pending objects. The index is\n> > read by read_index_from. As mentioned in the previous commit,\n> > read_index_from depends on the CWD for the location of the split index,\n> \n> As I mentioned before this doesn't actually depend on the CWD but\n> rather the per-worktree gitdir.\n\nRight, will fix.\n\n> > and add_index_objects_to_pending doesn't set that before using\n> > read_index_from.\n> > \n> > Instead of using read_index_from, use repo_read_index, which is aware of\n> > the proper paths for the worktree.\n> > \n> > This fixes t5304-prune when ran with GIT_TEST_SPLIT_INDEX set.\n> > \n> > Signed-off-by: Thomas Gummerer \n> > ---\n> > repository.c | 11 +++++++++++\n> > repository.h | 2 ++\n> > revision.c | 14 +++++++++-----\n> > 3 files changed, 22 insertions(+), 5 deletions(-)\n> > \n> > diff --git a/repository.c b/repository.c\n> > index 928b1f553d..3c9bfbd1b8 100644\n> > --- a/repository.c\n> > +++ b/repository.c\n> > @@ -2,6 +2,7 @@\n> > #include \"repository.h\"\n> > #include \"config.h\"\n> > #include \"submodule-config.h\"\n> > +#include \"worktree.h\"\n> > \n> > /* The main repository */\n> > static struct repository the_repo = {\n> > @@ -146,6 +147,16 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)\n> > \treturn -1;\n> > }\n> > \n> > +/*\n> > + * Initialize 'repo' based on the provided worktree\n> > + * Return 0 upon success and a non-zero value upon failure.\n> > + */\n> > +int repo_worktree_init(struct repository *repo, struct worktree *worktree)\n> > +{\n> > +\treturn repo_init(repo, get_worktree_git_dir(worktree),\n> > +\t\t\t worktree->path);\n> \n> I still feel very unsettled about this and don't think its a good idea.\n> get_worktree_git_dir depends implicitly on the global the_repository\n> object and I would like to avoid relying on it for an initialization\n> function like this.\n> \n> > +}\n> > +\n> > /*\n> > * Initialize 'submodule' as the submodule given by 'path' in parent repository\n> > * 'superproject'.\n> > diff --git a/repository.h b/repository.h\n> > index 7f5e24a0a2..2adeb05bf4 100644\n> > --- a/repository.h\n> > +++ b/repository.h\n> > @@ -4,6 +4,7 @@\n> > struct config_set;\n> > struct index_state;\n> > struct submodule_cache;\n> > +struct worktree;\n> > \n> > struct repository {\n> > \t/* Environment */\n> > @@ -87,6 +88,7 @@ extern struct repository *the_repository;\n> > extern void repo_set_gitdir(struct repository *repo, const char *path);\n> > extern void repo_set_worktree(struct repository *repo, const char *path);\n> > extern int repo_init(struct repository *repo, const char *gitdir, const char *worktree);\n> > +extern int repo_worktree_init(struct repository *repo, struct worktree *worktree);\n> > extern int repo_submodule_init(struct repository *submodule,\n> > \t\t\t struct repository *superproject,\n> > \t\t\t const char *path);\n> > diff --git a/revision.c b/revision.c\n> > index e2e691dd5a..34e1e4b799 100644\n> > --- a/revision.c\n> > +++ b/revision.c\n> > @@ -22,6 +22,7 @@\n> > #include \"packfile.h\"\n> > #include \"worktree.h\"\n> > #include \"argv-array.h\"\n> > +#include \"repository.h\"\n> > \n> > volatile show_early_output_fn_t show_early_output;\n> > \n> > @@ -1346,15 +1347,18 @@ void add_index_objects_to_pending(struct rev_info *revs, unsigned int flags)\n> > \tworktrees = get_worktrees(0);\n> > \tfor (p = worktrees; *p; p++) {\n> > \t\tstruct worktree *wt = *p;\n> > -\t\tstruct index_state istate = { NULL };\n> > +\t\tstruct repository *repo;\n> > \n> > +\t\trepo = xmalloc(sizeof(struct repository));\n> > \t\tif (wt->is_current)\n> > \t\t\tcontinue; /* current index already taken care of */\n> > +\t\tif (repo_worktree_init(repo, wt))\n> > +\t\t\tBUG(\"couldn't initialize repository object from worktree\");\n> > \n> > -\t\tif (read_index_from(&istate,\n> > -\t\t\t\t worktree_git_path(wt, \"index\")) > 0)\n> \n> Ok, after thinking this through a bit more I think a better approach may\n> be to restructure the call to read_index_from to take in both an index\n> file as well as the explicit gitdir to use when constructing a path to\n> the sharedindex file. That way you can fix this for worktrees and\n> submodules without having to pass in a repository object to the logic\n> which is reading an index file as well as avoiding needing to init a\n> repository object for every worktree.\n\nI still think with eventually we probably only want to read the index\nthrough a repository object instead of reading it to a separate struct\nindex_state.\n\nBut we can probably defer that for now, as this series really just\nwants to fix the regressions, and we can think a bit more about how\nthe repository struct interacts with worktrees.\n\n> So you could rework the first patch to do something like:\n\nThanks for the below, I'll try the below and see how much churn it\ncauses adjusting all the callsites, and send a re-roll later this\nweek.\n\n> diff --git a/read-cache.c b/read-cache.c\n> index 2eb81a66b..3a04b5567 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -1863,20 +1863,19 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n> * This way, shared index can be removed if they have not been used\n> * for some time.\n> */\n> -static void freshen_shared_index(char *base_sha1_hex, int warn)\n> +static void freshen_shared_index(const char *shared_index, int warn)\n> {\n> -\tchar *shared_index = git_pathdup(\"sharedindex.%s\", base_sha1_hex);\n> \tif (!check_and_freshen_file(shared_index, 1) && warn)\n> \t\twarning(\"could not freshen shared index '%s'\", shared_index);\n> -\tfree(shared_index);\n> }\n> \n> -int read_index_from(struct index_state *istate, const char *path)\n> +int read_index_from(struct index_state *istate, const char *path,\n> +\t\t const char *gitdir)\n> {\n> \tstruct split_index *split_index;\n> \tint ret;\n> \tchar *base_sha1_hex;\n> -\tconst char *base_path;\n> +\tchar *base_path;\n> \n> \t/* istate->initialized covers both .git/index and .git/sharedindex.xxx */\n> \tif (istate->initialized)\n> @@ -1896,14 +1895,14 @@ int read_index_from(struct index_state *istate, const char *path)\n> \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n> \n> \tbase_sha1_hex = sha1_to_hex(split_index->base_sha1);\n> -\tbase_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n> +\tbase_path = xstrfmt(\"%s/sharedindex.%s\", gitdir, base_sha1_hex);\n> \tret = do_read_index(split_index->base, base_path, 1);\n> \tif (hashcmp(split_index->base_sha1, split_index->base->sha1))\n> \t\tdie(\"broken index, expect %s in %s, got %s\",\n> \t\t base_sha1_hex, base_path,\n> \t\t sha1_to_hex(split_index->base->sha1));\n> \n> -\tfreshen_shared_index(base_sha1_hex, 0);\n> +\tfreshen_shared_index(base_path, 0);\n> \tmerge_base_index(istate);\n> \tpost_read_index_from(istate);\n> + free(base_path);\n> \treturn ret;\n> \n> \n> -- \n> Brandon Williams\n"},{"id":"335839","messageId":"xmqqwp0xe8ul.fsf@gitster.mtv.corp.google.com","threadId":"47421","inReplyTo":"20180103221849.GC2641@hank","subject":"Re: [PATCH v2 2/3] prune: fix pruning with multiple worktrees and split index","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-01-04T19:12:50Z","receivedAt":"2018-01-04T19:12:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Gummerer writes:\n\n> [sorry for the late reply. I was on Christmas holidays until today\n> and am still catching up on the mailing list. It will probably take\n> me untill the weekend to send a re-roll]\n>\n> On 12/18, Brandon Williams wrote:\n>> On 12/17, Thomas Gummerer wrote:\n>> > be489d02d2 (\"revision.c: --indexed-objects add objects from all\n>> > worktrees\", 2017-08-23) made sure that pruning takes objects from all\n>> > worktrees into account.\n>> > \n>> > It did that by reading the index of every worktree and adding the\n>> > necessary index objects to the set of pending objects. The index is\n>> > read by read_index_from. As mentioned in the previous commit,\n>> > read_index_from depends on the CWD for the location of the split index,\n>> \n>> As I mentioned before this doesn't actually depend on the CWD but\n>> rather the per-worktree gitdir.\n>\n> Right, will fix.\n\nThanks.\n"},{"id":"335859","messageId":"20180104201312.GD2641@hank","threadId":"47421","inReplyTo":"CA6C93AD-B24D-43A2-8AAA-DE98A4C9E719@gmail.com","subject":"Re: [PATCH v2 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2018-01-04T20:13:12Z","receivedAt":"2018-01-04T20:11:10Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 12/18, Lars Schneider wrote:\n> \n> > On 17 Dec 2017, at 23:51, Thomas Gummerer wrote:\n> > \n> > Split index mode only has a few dedicated tests, but as the index is\n> > involved in nearly every git operation, this doesn't quite cover all the\n> > ways repositories with split index can break. To use split index mode\n> > throughout the test suite a GIT_TEST_SPLIT_INDEX environment variable\n> > can be set, which makes git split the index at random and thus\n> > excercises the functionality much more thoroughly.\n> > \n> > As this is not turned on by default, it is not executed nearly as often\n> > as the test suite is run, so occationally breakages slip through. Try\n> > to counteract that by running the test suite with GIT_TEST_SPLIT_INDEX\n> > mode turned on on travis.\n> > \n> > To avoid using too many cycles on travis only run split index mode in\n> > the linux-gcc and the linux 32-bit gcc targets.\n> \n> I am surprised to see the split index mode test for the linux 32-bit\n> target. Is it likely that a split index bug appears only on 32-bit? \n> Wouldn't the linux-gcc target be sufficient to save resources/time?\n\nI'm not sure it's particularly likely for a bug to appear only on\n32-bit builds. It also doesn't seem to take too long to run, so I\nthought I'd add it just in case, but I'm happy running the tests only\nin the 64-bit builds if that's preferred.\n\n> > The Linux builds were\n> > chosen over the Mac OS builds because they tend to be much faster to\n> > complete.\n> > \n> > The linux gcc build was chosen over the linux clang build because the\n> > linux clang build is the fastest build, so it can serve as an early\n> > indicator if something is broken and we want to avoid spending the extra\n> > cycles of running the test suite twice for that.\n> > \n> > Helped-by: Lars Schneider \n> > Helped-by: Junio C Hamano \n> > Signed-off-by: Thomas Gummerer \n> > ---\n> > ci/run-linux32-build.sh | 1 +\n> > ci/run-tests.sh | 4 ++++\n> > 2 files changed, 5 insertions(+)\n> > \n> > diff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh\n> > index e30fb2cddc..f173c9cf2a 100755\n> > --- a/ci/run-linux32-build.sh\n> > +++ b/ci/run-linux32-build.sh\n> > @@ -27,4 +27,5 @@ linux32 --32bit i386 su -m -l $CI_USER -c '\n> > cd /usr/src/git &&\n> > make --jobs=2 &&\n> > make --quiet test\n> > + GIT_TEST_SPLIT_INDEX=YesPlease make --quiet test\n> > '\n> > diff --git a/ci/run-tests.sh b/ci/run-tests.sh\n> > index f0c743de94..c7aee5b9ff 100755\n> > --- a/ci/run-tests.sh\n> > +++ b/ci/run-tests.sh\n> > @@ -8,3 +8,7 @@\n> > mkdir -p $HOME/travis-cache\n> > ln -s $HOME/travis-cache/.prove t/.prove\n> > make --quiet test\n> > +if test \"$jobname\" = \"linux-gcc\"\n> > +then\n> > +\tGIT_TEST_SPLIT_INDEX=YesPlease make --quiet test\n> > +fi\n> \n> For now I think that looks good. Maybe we could define additional test \n> configurations with an environment variable. That could be an array variable\n> defined in the lib-travis.ci \"case\" statement:\n> https://github.com/git/git/blob/1229713f78cd2883798e95f33c19c81b523413fd/ci/lib-travisci.sh#L42-L65\n\nThat sounds like a good idea. I'll try to see if I can come up with\nsomething.\n\n> - Lars\n"},{"id":"335923","messageId":"65BD918A-DDE4-4940-84BE-ED28006BBE07@gmail.com","threadId":"47421","inReplyTo":"20180104201312.GD2641@hank","subject":"Re: [PATCH v2 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2018-01-05T11:03:06Z","receivedAt":"2018-01-05T11:03:15Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 04 Jan 2018, at 21:13, Thomas Gummerer wrote:\n> \n> On 12/18, Lars Schneider wrote:\n>> \n>>> On 17 Dec 2017, at 23:51, Thomas Gummerer wrote:\n>>> \n>>> Split index mode only has a few dedicated tests, but as the index is\n>>> involved in nearly every git operation, this doesn't quite cover all the\n>>> ways repositories with split index can break. To use split index mode\n>>> throughout the test suite a GIT_TEST_SPLIT_INDEX environment variable\n>>> can be set, which makes git split the index at random and thus\n>>> excercises the functionality much more thoroughly.\n>>> \n>>> As this is not turned on by default, it is not executed nearly as often\n>>> as the test suite is run, so occationally breakages slip through. Try\n>>> to counteract that by running the test suite with GIT_TEST_SPLIT_INDEX\n>>> mode turned on on travis.\n>>> \n>>> To avoid using too many cycles on travis only run split index mode in\n>>> the linux-gcc and the linux 32-bit gcc targets.\n>> \n>> I am surprised to see the split index mode test for the linux 32-bit\n>> target. Is it likely that a split index bug appears only on 32-bit? \n>> Wouldn't the linux-gcc target be sufficient to save resources/time?\n> \n> I'm not sure it's particularly likely for a bug to appear only on\n> 32-bit builds. It also doesn't seem to take too long to run, so I\n> thought I'd add it just in case, but I'm happy running the tests only\n> in the 64-bit builds if that's preferred.\n\nCI running times naturally add up over time as more and more stuff is\ntested. That's why I would argue testing GIT_TEST_SPLIT_INDEX on 64-bit\nis probably sufficient. However, I am not really familiar with \nGIT_TEST_SPLIT_INDEX. \n\nThere is no right or wrong here and either way is fine with me. \nI just wanted to express my \"keep an eye on CI time\" concern.\n\n\nThanks,\nLars\n"},{"id":"336106","messageId":"20180107200234.GB24806@hank","threadId":"47421","inReplyTo":"20180104201312.GD2641@hank","subject":"Re: [PATCH v2 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2018-01-07T20:02:34Z","receivedAt":"2018-01-07T20:00:32Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 01/04, Thomas Gummerer wrote:\n> On 12/18, Lars Schneider wrote:\n> > [snip]\n> > For now I think that looks good. Maybe we could define additional test \n> > configurations with an environment variable. That could be an array variable\n> > defined in the lib-travis.ci \"case\" statement:\n> > https://github.com/git/git/blob/1229713f78cd2883798e95f33c19c81b523413fd/ci/lib-travisci.sh#L42-L65\n> \n> That sounds like a good idea. I'll try to see if I can come up with\n> something.\n\nOn second thought I'd prefer to just leave it as is for now, and leave\ndefining additional test configurations for a future iteration.\nHaving it configurable makes it a bit uglier, and I'm not even sure\nhow many configurations we can just test at runtime vs. having to\ncompile with the flag set. So I'd like to punt on that for now, and\nintroduce more configurability once we actually need it :)\n\n> > - Lars\n"},{"id":"336109","messageId":"20180107223015.17720-1-t.gummerer@gmail.com","threadId":"47421","inReplyTo":"20171217225122.28941-1-t.gummerer@gmail.com","subject":"[PATCH v3 0/3] fixes for split index mode","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2018-01-07T22:30:12Z","receivedAt":"2018-01-07T22:28:34Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Thanks Brandon and Lars for comments on the previous round.\n\nPrevious rounds were at <20171210212202.28231-1-t.gummerer@gmail.com>\nand <20171217225122.28941-1-t.gummerer@gmail.com>.\n\nChanges since the previous round:\n\n- reworked the patches to no longer try to use struct repository for\n worktrees, but pass gitdir into read_index_from instead (Thanks\n Brandon :)). So the fixes that were in 1/3 and 2/3 previously are\n now in 1/3\n- 2/3 now fixes t7009 properly. I thought it was fixed before, but it\n probably just passed the tests because of the GIT_TEST_SPLIT_INDEX\n \"randomness\".\n- The travis job is now only running the 64-bit linux build with split\n index mode to save even more cycles.\n- As this wasn't picked up anywhere yet, I took the freedom to rebase\n this onto the current master, which includes sg/travis-fixes, which\n made it a bit easier for me to test. If this makes the life harder\n for anyone reviewing this let me know and I can base it on the same\n commit previous iterations were based on.\n\nThomas Gummerer (3):\n read-cache: fix reading the shared index for other repos\n split-index: don't write cache tree with null sha1 entries\n travis: run tests with GIT_TEST_SPLIT_INDEX\n\n cache-tree.c | 2 +-\n cache.h | 8 +++++---\n ci/run-linux32-build.sh | 1 +\n ci/run-tests.sh | 4 ++++\n read-cache.c | 25 ++++++++++++++-----------\n repository.c | 2 +-\n revision.c | 3 ++-\n split-index.c | 2 ++\n t/t1700-split-index.sh | 19 +++++++++++++++++++\n 9 files changed, 49 insertions(+), 17 deletions(-)\n\n-- \n2.16.0.rc1.238.g530d649a79\n\n"},{"id":"336110","messageId":"20180107223015.17720-2-t.gummerer@gmail.com","threadId":"47421","inReplyTo":"20180107223015.17720-1-t.gummerer@gmail.com","subject":"[PATCH v3 1/3] read-cache: fix reading the shared index for other repos","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2018-01-07T22:30:13Z","receivedAt":"2018-01-07T22:28:37Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"read_index_from() takes a path argument for the location of the index\nfile. For reading the shared index in split index mode however it just\nignores that path argument, and reads it from the gitdir of the current\nrepository.\n\nThis works as long as an index in the_repository is read. Once that\nchanges, such as when we read the index of a submodule, or of a\ndifferent working tree than the current one, the gitdir of\nthe_repository will no longer contain the appropriate shared index,\nand git will fail to read it.\n\nFor example t3007-ls-files-recurse-submodules.sh was broken with\nGIT_TEST_SPLIT_INDEX set in 188dce131f (\"ls-files: use repository\nobject\", 2017-06-22), and t7814-grep-recurse-submodules.sh was also\nbroken in a similar manner, probably by introducing struct repository\nthere, although I didn't track down the exact commit for that.\n\nbe489d02d2 (\"revision.c: --indexed-objects add objects from all\nworktrees\", 2017-08-23) breaks with split index mode in a similar\nmanner, not erroring out when it can't read the index, but instead\ncarrying on with pruning, without taking the index of the worktree into\naccount.\n\nFix this by passing an additional gitdir parameter to read_index_from,\nto indicate where it should look for and read the shared index from.\n\nread_cache_from() defaults to using the gitdir of the_repository. As it\nis mostly a convenience macro, having to pass get_git_dir() for every\ncall seems overkill, and if necessary users can have more control by\nusing read_index_from().\n\nHelped-by: Brandon Williams \nSigned-off-by: Thomas Gummerer \n---\n cache-tree.c | 2 +-\n cache.h | 5 +++--\n read-cache.c | 23 +++++++++++++----------\n repository.c | 2 +-\n revision.c | 3 ++-\n 5 files changed, 20 insertions(+), 15 deletions(-)\n\ndiff --git a/cache-tree.c b/cache-tree.c\nindex e03e72c34a..0dd6292a94 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -608,7 +608,7 @@ int write_index_as_tree(unsigned char *sha1, struct index_state *index_state, co\n \n \thold_lock_file_for_update(&lock_file, index_path, LOCK_DIE_ON_ERROR);\n \n-\tentries = read_index_from(index_state, index_path);\n+\tentries = read_index_from(index_state, index_path, get_git_dir());\n \tif (entries < 0) {\n \t\tret = WRITE_TREE_UNREADABLE_INDEX;\n \t\tgoto out;\ndiff --git a/cache.h b/cache.h\nindex d8b975a571..d5a7d1d7f1 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -371,7 +371,7 @@ extern void free_name_hash(struct index_state *istate);\n #define active_cache_tree (the_index.cache_tree)\n \n #define read_cache() read_index(&the_index)\n-#define read_cache_from(path) read_index_from(&the_index, (path))\n+#define read_cache_from(path) read_index_from(&the_index, (path), (get_git_dir()))\n #define read_cache_preload(pathspec) read_index_preload(&the_index, (pathspec))\n #define is_cache_unborn() is_index_unborn(&the_index)\n #define read_cache_unmerged() read_index_unmerged(&the_index)\n@@ -616,7 +616,8 @@ extern int read_index(struct index_state *);\n extern int read_index_preload(struct index_state *, const struct pathspec *pathspec);\n extern int do_read_index(struct index_state *istate, const char *path,\n \t\t\t int must_exist); /* for testting only! */\n-extern int read_index_from(struct index_state *, const char *path);\n+extern int read_index_from(struct index_state *, const char *path,\n+\t\t\t const char *gitdir);\n extern int is_index_unborn(struct index_state *);\n extern int read_index_unmerged(struct index_state *);\n \ndiff --git a/read-cache.c b/read-cache.c\nindex 2eb81a66b9..3f5b4afa36 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1603,7 +1603,7 @@ int hold_locked_index(struct lock_file *lk, int lock_flags)\n \n int read_index(struct index_state *istate)\n {\n-\treturn read_index_from(istate, get_index_file());\n+\treturn read_index_from(istate, get_index_file(), get_git_dir());\n }\n \n static struct cache_entry *cache_entry_from_ondisk(struct ondisk_cache_entry *ondisk,\n@@ -1863,20 +1863,19 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist)\n * This way, shared index can be removed if they have not been used\n * for some time.\n */\n-static void freshen_shared_index(char *base_sha1_hex, int warn)\n+static void freshen_shared_index(const char *shared_index, int warn)\n {\n-\tchar *shared_index = git_pathdup(\"sharedindex.%s\", base_sha1_hex);\n \tif (!check_and_freshen_file(shared_index, 1) && warn)\n \t\twarning(\"could not freshen shared index '%s'\", shared_index);\n-\tfree(shared_index);\n }\n \n-int read_index_from(struct index_state *istate, const char *path)\n+int read_index_from(struct index_state *istate, const char *path,\n+\t\t const char *gitdir)\n {\n \tstruct split_index *split_index;\n \tint ret;\n \tchar *base_sha1_hex;\n-\tconst char *base_path;\n+\tchar *base_path;\n \n \t/* istate->initialized covers both .git/index and .git/sharedindex.xxx */\n \tif (istate->initialized)\n@@ -1896,16 +1895,17 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tsplit_index->base = xcalloc(1, sizeof(*split_index->base));\n \n \tbase_sha1_hex = sha1_to_hex(split_index->base_sha1);\n-\tbase_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n+\tbase_path = xstrfmt(\"%s/sharedindex.%s\", gitdir, base_sha1_hex);\n \tret = do_read_index(split_index->base, base_path, 1);\n \tif (hashcmp(split_index->base_sha1, split_index->base->sha1))\n \t\tdie(\"broken index, expect %s in %s, got %s\",\n \t\t base_sha1_hex, base_path,\n \t\t sha1_to_hex(split_index->base->sha1));\n \n-\tfreshen_shared_index(base_sha1_hex, 0);\n+\tfreshen_shared_index(base_path, 0);\n \tmerge_base_index(istate);\n \tpost_read_index_from(istate);\n+\tfree(base_path);\n \treturn ret;\n }\n \n@@ -2573,8 +2573,11 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,\n \tret = write_split_index(istate, lock, flags);\n \n \t/* Freshen the shared index only if the split-index was written */\n-\tif (!ret && !new_shared_index)\n-\t\tfreshen_shared_index(sha1_to_hex(si->base_sha1), 1);\n+\tif (!ret && !new_shared_index) {\n+\t\tconst char *shared_index = git_path(\"sharedindex.%s\",\n+\t\t\t\t\t\t sha1_to_hex(si->base_sha1));\n+\t\tfreshen_shared_index(shared_index, 1);\n+\t}\n \n out:\n \tif (flags & COMMIT_LOCK)\ndiff --git a/repository.c b/repository.c\nindex 998413b8bb..0d715f4fdb 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -236,5 +236,5 @@ int repo_read_index(struct repository *repo)\n \tif (!repo->index)\n \t\trepo->index = xcalloc(1, sizeof(*repo->index));\n \n-\treturn read_index_from(repo->index, repo->index_file);\n+\treturn read_index_from(repo->index, repo->index_file, repo->gitdir);\n }\ndiff --git a/revision.c b/revision.c\nindex 72f2b4572e..76d6d125b3 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1352,7 +1352,8 @@ void add_index_objects_to_pending(struct rev_info *revs, unsigned int flags)\n \t\t\tcontinue; /* current index already taken care of */\n \n \t\tif (read_index_from(&istate,\n-\t\t\t\t worktree_git_path(wt, \"index\")) > 0)\n+\t\t\t\t worktree_git_path(wt, \"index\"),\n+\t\t\t\t get_worktree_git_dir(wt)) > 0)\n \t\t\tdo_add_index_objects_to_pending(revs, &istate);\n \t\tdiscard_index(&istate);\n \t}\n-- \n2.16.0.rc1.238.g530d649a79\n\n"},{"id":"336111","messageId":"20180107223015.17720-3-t.gummerer@gmail.com","threadId":"47421","inReplyTo":"20180107223015.17720-1-t.gummerer@gmail.com","subject":"[PATCH v3 2/3] split-index: don't write cache tree with null oid entries","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2018-01-07T22:30:14Z","receivedAt":"2018-01-07T22:28:39Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"In a96d3cc3f6 (\"cache-tree: reject entries with null sha1\", 2017-04-21)\nwe made sure that broken cache entries do not get propagated to new\ntrees. Part of that was making sure not to re-use an existing cache\ntree that includes a null oid.\n\nIt did so by dropping the cache tree in 'do_write_index()' if one of\nthe entries contains a null oid. In split index mode however, there\nare two invocations to 'do_write_index()', one for the shared index\nand one for the split index. The cache tree is only written once, to\nthe split index.\n\nAs we only loop through the elements that are effectively being\nwritten by the current invocation, that may not include the entry with\na null oid in the split index (when it is already written to the\nshared index), where we write the cache tree. Therefore in split\nindex mode we may still end up writing the cache tree, even though\nthere is an entry with a null oid in the index.\n\nFix this by checking for null oids in prepare_to_write_split_index,\nwhere we loop the entries of the shared index as well as the entries for\nthe split index.\n\nThis fixes t7009 with GIT_TEST_SPLIT_INDEX. Also add a new test that's\nmore specifically showing the problem.\n\nSigned-off-by: Thomas Gummerer \n---\n cache.h | 3 ++-\n read-cache.c | 2 +-\n split-index.c | 2 ++\n t/t1700-split-index.sh | 19 +++++++++++++++++++\n 4 files changed, 24 insertions(+), 2 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex d5a7d1d7f1..fd755c32cf 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -345,7 +345,8 @@ struct index_state {\n \tstruct split_index *split_index;\n \tstruct cache_time timestamp;\n \tunsigned name_hash_initialized : 1,\n-\t\t initialized : 1;\n+\t\t initialized : 1,\n+\t\t drop_cache_tree : 1;\n \tstruct hashmap name_hash;\n \tstruct hashmap dir_hash;\n \tunsigned char sha1[20];\ndiff --git a/read-cache.c b/read-cache.c\nindex 3f5b4afa36..d13ce83794 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2243,7 +2243,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,\n \tstruct stat st;\n \tstruct ondisk_cache_entry_extended ondisk;\n \tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n-\tint drop_cache_tree = 0;\n+\tint drop_cache_tree = istate->drop_cache_tree;\n \n \tfor (i = removed = extended = 0; i < entries; i++) {\n \t\tif (cache[i]->ce_flags & CE_REMOVE)\ndiff --git a/split-index.c b/split-index.c\nindex 83e39ec8d7..284d04d67f 100644\n--- a/split-index.c\n+++ b/split-index.c\n@@ -238,6 +238,8 @@ void prepare_to_write_split_index(struct index_state *istate)\n \t\t\t\tALLOC_GROW(entries, nr_entries+1, nr_alloc);\n \t\t\t\tentries[nr_entries++] = ce;\n \t\t\t}\n+\t\t\tif (is_null_oid(&ce->oid))\n+\t\t\t\tistate->drop_cache_tree = 1;\n \t\t}\n \t}\n \ndiff --git a/t/t1700-split-index.sh b/t/t1700-split-index.sh\nindex af9b847761..c087b63367 100755\n--- a/t/t1700-split-index.sh\n+++ b/t/t1700-split-index.sh\n@@ -401,4 +401,23 @@ done <<\\EOF\n 0642 -rw-r---w-\n EOF\n \n+test_expect_success 'writing split index with null sha1 does not write cache tree' '\n+\tgit config core.splitIndex true &&\n+\tgit config splitIndex.maxPercentChange 0 &&\n+\tgit commit -m \"commit\" &&\n+\t{\n+\t\tgit ls-tree HEAD &&\n+\t\tprintf \"160000 commit $_z40\\\\tbroken\\\\n\"\n+\t} >broken-tree &&\n+\techo \"add broken entry\" >msg &&\n+\n+\ttree=$(git mktree cache-tree.out || true) &&\n+\ttest_line_count = 0 cache-tree.out\n+'\n+\n test_done\n-- \n2.16.0.rc1.238.g530d649a79\n\n"},{"id":"336112","messageId":"20180107223015.17720-4-t.gummerer@gmail.com","threadId":"47421","inReplyTo":"20180107223015.17720-1-t.gummerer@gmail.com","subject":"[PATCH v3 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2018-01-07T22:30:15Z","receivedAt":"2018-01-07T22:28:40Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Split index mode only has a few dedicated tests, but as the index is\ninvolved in nearly every git operation, this doesn't quite cover all the\nways repositories with split index can break. To use split index mode\nthroughout the test suite a GIT_TEST_SPLIT_INDEX environment variable\ncan be set, which makes git split the index at random and thus\nexcercises the functionality much more thoroughly.\n\nAs this is not turned on by default, it is not executed nearly as often\nas the test suite is run, so occationally breakages slip through. Try\nto counteract that by running the test suite with GIT_TEST_SPLIT_INDEX\nmode turned on on travis.\n\nTo avoid using too many cycles on travis only run split index mode in\nthe linux-gcc target only. The Linux build was chosen over the Mac OS\nbuilds because it tends to be much faster to complete.\n\nThe linux gcc build was chosen over the linux clang build because the\nlinux clang build is the fastest build, so it can serve as an early\nindicator if something is broken and we want to avoid spending the extra\ncycles of running the test suite twice for that.\n\nHelped-by: Lars Schneider \nHelped-by: Junio C Hamano \nSigned-off-by: Thomas Gummerer \n---\n ci/run-tests.sh | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/ci/run-tests.sh b/ci/run-tests.sh\nindex f0c743de94..c7aee5b9ff 100755\n--- a/ci/run-tests.sh\n+++ b/ci/run-tests.sh\n@@ -8,3 +8,7 @@\n mkdir -p $HOME/travis-cache\n ln -s $HOME/travis-cache/.prove t/.prove\n make --quiet test\n+if test \"$jobname\" = \"linux-gcc\"\n+then\n+\tGIT_TEST_SPLIT_INDEX=YesPlease make --quiet test\n+fi\n-- \n2.16.0.rc1.238.g530d649a79\n\n"},{"id":"336131","messageId":"CACsJy8DBkAwoNWSma-+WkOUkQrenbkT5c1-Ez2jmmf-+wGRbxQ@mail.gmail.com","threadId":"47421","inReplyTo":"20180107223015.17720-2-t.gummerer@gmail.com","subject":"Re: [PATCH v3 1/3] read-cache: fix reading the shared index for other repos","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-01-08T10:41:59Z","receivedAt":"2018-01-08T10:42:35Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Jan 8, 2018 at 5:30 AM, Thomas Gummerer wrote:\n> @@ -1896,16 +1895,17 @@ int read_index_from(struct index_state *istate, const char *path)\n> split_index->base = xcalloc(1, sizeof(*split_index->base));\n>\n> base_sha1_hex = sha1_to_hex(split_index->base_sha1);\n> - base_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n> + base_path = xstrfmt(\"%s/sharedindex.%s\", gitdir, base_sha1_hex);\n\nPersonally I prefer the repo_git_path() from v2 (sorry I was away and\ncould not comment anything). The thing is, git_path() and friends\ncould do some path translation underneath to support multiple\nworktrees. Think of the given path here as a \"virtual path\" that may\nbe translated to something else, not exactly + \"/\" +\n\"sharedindex.%s\". But in practice, we're not breaking the relationship\nbetween $GIT_DIR/index and $GIT_DIR/sharedindex.* any time soon, doing\nmanual path transformation here is fine.\n\nWhat about the other git_path() in this file? With patch applied I still get\n\n> ~/w/git/temp $ git grep git_path read-cache.c\nread-cache.c: shared_index_path = git_path(\"%s\", de->d_name);\nread-cache.c: temp = mks_tempfile(git_path(\"sharedindex_XXXXXX\"));\nread-cache.c: git_path(\"sharedindex.%s\",\nsha1_to_hex(si->base->sha1)));\nread-cache.c: const char *shared_index = git_path(\"sharedindex.%s\",\n\nI suppose submodule has not triggered any of these code paths yet. Not\nsure if we should deal with them now or wait until later.\n\nPerhaps if we add a \"struct repository *\" pointer inside index_state,\nwe could retrieve back the_repository (or others) and call\nrepo_git_path() everywhere without changing index api too much. I\ndon't know. I like the 'struct repository' concept but couldn't\nfollow its development so I don't if this is what it should become.\n-- \nDuy\n"},{"id":"336213","messageId":"20180108224110.GH2641@hank","threadId":"47421","inReplyTo":"CACsJy8DBkAwoNWSma-+WkOUkQrenbkT5c1-Ez2jmmf-+wGRbxQ@mail.gmail.com","subject":"Re: [PATCH v3 1/3] read-cache: fix reading the shared index for other repos","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2018-01-08T22:41:10Z","receivedAt":"2018-01-08T22:39:06Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 01/08, Duy Nguyen wrote:\n> On Mon, Jan 8, 2018 at 5:30 AM, Thomas Gummerer wrote:\n> > @@ -1896,16 +1895,17 @@ int read_index_from(struct index_state *istate, const char *path)\n> > split_index->base = xcalloc(1, sizeof(*split_index->base));\n> >\n> > base_sha1_hex = sha1_to_hex(split_index->base_sha1);\n> > - base_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n> > + base_path = xstrfmt(\"%s/sharedindex.%s\", gitdir, base_sha1_hex);\n> \n> Personally I prefer the repo_git_path() from v2 (sorry I was away and\n> could not comment anything).\n\nIt felt slightly nicer to me as well, but it threw up a bunch of\nquestions about how worktrees will fit together with struct\nrepository. As I don't feel very strongly about this either way I\ndecided to go with what Brandon suggested as an alternative, which\nallows us to defer that decision. I'd be happy to revert this to the\nway I had it in v2, but I don't want to drag the discussion on too\nlong either, as this series does fix some real regressions.\n\n> The thing is, git_path() and friends\n> could do some path translation underneath to support multiple\n> worktrees. Think of the given path here as a \"virtual path\" that may\n> be translated to something else, not exactly + \"/\" +\n> \"sharedindex.%s\". But in practice, we're not breaking the relationship\n> between $GIT_DIR/index and $GIT_DIR/sharedindex.* any time soon, doing\n> manual path transformation here is fine.\n>\n> What about the other git_path() in this file? With patch applied I still get\n> \n> > ~/w/git/temp $ git grep git_path read-cache.c\n> read-cache.c: shared_index_path = git_path(\"%s\", de->d_name);\n> read-cache.c: temp = mks_tempfile(git_path(\"sharedindex_XXXXXX\"));\n> read-cache.c: git_path(\"sharedindex.%s\",\n> sha1_to_hex(si->base->sha1)));\n> read-cache.c: const char *shared_index = git_path(\"sharedindex.%s\",\n\nGood point, I hadn't looked at these, I only looked at the current\ntest failures. I'm going to be away for the rest of the week, but\nI'll have a look at them when I come back.\n\n> I suppose submodule has not triggered any of these code paths yet. Not\n> sure if we should deal with them now or wait until later.\n> \n> Perhaps if we add a \"struct repository *\" pointer inside index_state,\n> we could retrieve back the_repository (or others) and call\n> repo_git_path() everywhere without changing index api too much. I\n> don't know. I like the 'struct repository' concept but couldn't\n> follow its development so I don't if this is what it should become.\n\nInteresting. I didn't follow the development of struct repository\ntoo closely either, so I'm not sure. Brandon might have more of an\nopinion on that? :)\n\n> -- \n> Duy\n"},{"id":"336217","messageId":"20180108233840.GB156560@google.com","threadId":"47421","inReplyTo":"CACsJy8DBkAwoNWSma-+WkOUkQrenbkT5c1-Ez2jmmf-+wGRbxQ@mail.gmail.com","subject":"Re: [PATCH v3 1/3] read-cache: fix reading the shared index for other repos","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-08T23:38:40Z","receivedAt":"2018-01-08T23:38:49Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/08, Duy Nguyen wrote:\n> On Mon, Jan 8, 2018 at 5:30 AM, Thomas Gummerer wrote:\n> > @@ -1896,16 +1895,17 @@ int read_index_from(struct index_state *istate, const char *path)\n> > split_index->base = xcalloc(1, sizeof(*split_index->base));\n> >\n> > base_sha1_hex = sha1_to_hex(split_index->base_sha1);\n> > - base_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n> > + base_path = xstrfmt(\"%s/sharedindex.%s\", gitdir, base_sha1_hex);\n> \n> Personally I prefer the repo_git_path() from v2 (sorry I was away and\n> could not comment anything). The thing is, git_path() and friends\n> could do some path translation underneath to support multiple\n> worktrees. Think of the given path here as a \"virtual path\" that may\n> be translated to something else, not exactly + \"/\" +\n> \"sharedindex.%s\". But in practice, we're not breaking the relationship\n> between $GIT_DIR/index and $GIT_DIR/sharedindex.* any time soon, doing\n> manual path transformation here is fine.\n\nMy biggest complaint about v2 is that we still don't quite know the best\nway to integrate worktrees and struct repository yet so I was very\nreluctant to start having them interact in the way v2 was using them\ntogether. I'm very much in favor of this version (v3) as each worktree\ncan explicitly provide their gitdir to be used to determine where to\nread the shared index file without having to replicate a struct\nrepository for each.\n\n> \n> What about the other git_path() in this file? With patch applied I still get\n> \n> > ~/w/git/temp $ git grep git_path read-cache.c\n> read-cache.c: shared_index_path = git_path(\"%s\", de->d_name);\n> read-cache.c: temp = mks_tempfile(git_path(\"sharedindex_XXXXXX\"));\n> read-cache.c: git_path(\"sharedindex.%s\",\n> sha1_to_hex(si->base->sha1)));\n> read-cache.c: const char *shared_index = git_path(\"sharedindex.%s\",\n> \n> I suppose submodule has not triggered any of these code paths yet. Not\n> sure if we should deal with them now or wait until later.\n> \n> Perhaps if we add a \"struct repository *\" pointer inside index_state,\n> we could retrieve back the_repository (or others) and call\n> repo_git_path() everywhere without changing index api too much. I\n> don't know. I like the 'struct repository' concept but couldn't\n> follow its development so I don't if this is what it should become.\n\nI'm not too keen on having an index_state struct contain a back pointer\nto a repository struct. I do think that we may want to have worktree\nstructs contain a back pointer to the struct repository they correspond\nto. That way there is only one instance of the repository (and\nobject-store once that gets integrated) yet multiple worktrees.\n\n-- \nBrandon Williams\n"},{"id":"336218","messageId":"CACsJy8AkbMdQZq85N_LSvkM+JN4E2mkEUfaHHGTvrNJ4yCPs2Q@mail.gmail.com","threadId":"47421","inReplyTo":"20180108233840.GB156560@google.com","subject":"Re: [PATCH v3 1/3] read-cache: fix reading the shared index for other repos","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-01-09T01:24:11Z","receivedAt":"2018-01-09T01:24:51Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jan 9, 2018 at 6:38 AM, Brandon Williams wrote:\n> On 01/08, Duy Nguyen wrote:\n>> On Mon, Jan 8, 2018 at 5:30 AM, Thomas Gummerer wrote:\n>> > @@ -1896,16 +1895,17 @@ int read_index_from(struct index_state *istate, const char *path)\n>> > split_index->base = xcalloc(1, sizeof(*split_index->base));\n>> >\n>> > base_sha1_hex = sha1_to_hex(split_index->base_sha1);\n>> > - base_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n>> > + base_path = xstrfmt(\"%s/sharedindex.%s\", gitdir, base_sha1_hex);\n>>\n>> Personally I prefer the repo_git_path() from v2 (sorry I was away and\n>> could not comment anything). The thing is, git_path() and friends\n>> could do some path translation underneath to support multiple\n>> worktrees. Think of the given path here as a \"virtual path\" that may\n>> be translated to something else, not exactly + \"/\" +\n>> \"sharedindex.%s\". But in practice, we're not breaking the relationship\n>> between $GIT_DIR/index and $GIT_DIR/sharedindex.* any time soon, doing\n>> manual path transformation here is fine.\n>\n> My biggest complaint about v2 is that we still don't quite know the best\n> way to integrate worktrees and struct repository yet so I was very\n> reluctant to start having them interact in the way v2 was using them\n> together.\n\nThis will be on my todo list (after I have finished with\nnd/worktree-move which has been on 'pu' for too long). I'm ok with v3\nthen.\n\n> I'm very much in favor of this version (v3) as each worktree\n> can explicitly provide their gitdir to be used to determine where to\n> read the shared index file without having to replicate a struct\n> repository for each.\n\nIsn't that the end goal though (I vaguely recall this discussion when\n'struct repository' was an idea), that we will pass struct repository\naround [1] rather than relying on global objects.\n\n[1] or in this case struct worktree-something because the index\nbelongs more to a worktree than the object/refs database.\n\n>> What about the other git_path() in this file? With patch applied I still get\n>>\n>> > ~/w/git/temp $ git grep git_path read-cache.c\n>> read-cache.c: shared_index_path = git_path(\"%s\", de->d_name);\n>> read-cache.c: temp = mks_tempfile(git_path(\"sharedindex_XXXXXX\"));\n>> read-cache.c: git_path(\"sharedindex.%s\",\n>> sha1_to_hex(si->base->sha1)));\n>> read-cache.c: const char *shared_index = git_path(\"sharedindex.%s\",\n>>\n>> I suppose submodule has not triggered any of these code paths yet. Not\n>> sure if we should deal with them now or wait until later.\n>>\n>> Perhaps if we add a \"struct repository *\" pointer inside index_state,\n>> we could retrieve back the_repository (or others) and call\n>> repo_git_path() everywhere without changing index api too much. I\n>> don't know. I like the 'struct repository' concept but couldn't\n>> follow its development so I don't if this is what it should become.\n>\n> I'm not too keen on having an index_state struct contain a back pointer\n> to a repository struct. I do think that we may want to have worktree\n> structs contain a back pointer to the struct repository they correspond\n> to. That way there is only one instance of the repository (and\n> object-store once that gets integrated) yet multiple worktrees.\n\nYeah back pointer from worktree struct makes sense.\n-- \nDuy\n"},{"id":"336570","messageId":"20180113223303.GI2641@hank","threadId":"47421","inReplyTo":"20180108224110.GH2641@hank","subject":"Re: [PATCH v3 1/3] read-cache: fix reading the shared index for other repos","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2018-01-13T22:33:03Z","receivedAt":"2018-01-13T22:30:54Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 01/08, Thomas Gummerer wrote:\n> On 01/08, Duy Nguyen wrote:\n> > On Mon, Jan 8, 2018 at 5:30 AM, Thomas Gummerer wrote:\n> > > @@ -1896,16 +1895,17 @@ int read_index_from(struct index_state *istate, const char *path)\n> > > split_index->base = xcalloc(1, sizeof(*split_index->base));\n> > >\n> > > base_sha1_hex = sha1_to_hex(split_index->base_sha1);\n> > > - base_path = git_path(\"sharedindex.%s\", base_sha1_hex);\n> > > + base_path = xstrfmt(\"%s/sharedindex.%s\", gitdir, base_sha1_hex);\n> > \n> > Personally I prefer the repo_git_path() from v2 (sorry I was away and\n> > could not comment anything).\n> \n> It felt slightly nicer to me as well, but it threw up a bunch of\n> questions about how worktrees will fit together with struct\n> repository. As I don't feel very strongly about this either way I\n> decided to go with what Brandon suggested as an alternative, which\n> allows us to defer that decision. I'd be happy to revert this to the\n> way I had it in v2, but I don't want to drag the discussion on too\n> long either, as this series does fix some real regressions.\n> \n> > The thing is, git_path() and friends\n> > could do some path translation underneath to support multiple\n> > worktrees. Think of the given path here as a \"virtual path\" that may\n> > be translated to something else, not exactly + \"/\" +\n> > \"sharedindex.%s\". But in practice, we're not breaking the relationship\n> > between $GIT_DIR/index and $GIT_DIR/sharedindex.* any time soon, doing\n> > manual path transformation here is fine.\n> >\n> > What about the other git_path() in this file? With patch applied I still get\n> > \n> > > ~/w/git/temp $ git grep git_path read-cache.c\n> > read-cache.c: shared_index_path = git_path(\"%s\", de->d_name);\n> > read-cache.c: temp = mks_tempfile(git_path(\"sharedindex_XXXXXX\"));\n> > read-cache.c: git_path(\"sharedindex.%s\",\n> > sha1_to_hex(si->base->sha1)));\n> > read-cache.c: const char *shared_index = git_path(\"sharedindex.%s\",\n> \n> Good point, I hadn't looked at these, I only looked at the current\n> test failures. I'm going to be away for the rest of the week, but\n> I'll have a look at them when I come back.\n> \n> > I suppose submodule has not triggered any of these code paths yet. Not\n> > sure if we should deal with them now or wait until later.\n\nHaving had a look at these now, they are all in the write_locked_index\ncodepath. We should probably have something like\n'repo_write_locked_index()' for this. But that probably requires a\nbit more work/discussion to see what this should look like. I'd\nrather keep this patch series focused on the current breakages, and\ndeal with that in a separate patch series.\n\nWhile looking at this, I did find another breakage in the split index\ncode, which I'll send as 4/3.\n\n> > Perhaps if we add a \"struct repository *\" pointer inside index_state,\n> > we could retrieve back the_repository (or others) and call\n> > repo_git_path() everywhere without changing index api too much. I\n> > don't know. I like the 'struct repository' concept but couldn't\n> > follow its development so I don't if this is what it should become.\n> \n> Interesting. I didn't follow the development of struct repository\n> too closely either, so I'm not sure. Brandon might have more of an\n> opinion on that? :)\n> \n> > -- \n> > Duy\n"},{"id":"336571","messageId":"20180113223715.GJ2641@hank","threadId":"47421","inReplyTo":"20180107223015.17720-1-t.gummerer@gmail.com","subject":"[PATCH v3 4/3] read-cache: don't try to write index if we can't write shared index","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2018-01-13T22:37:15Z","receivedAt":"2018-01-13T22:35:05Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"In a0a967568e (\"update-index --split-index: do not split if $GIT_DIR is\nread only\", 2014-06-13), we tried to make sure we can still write an\nindex, even if the shared index can not be written.\n\nWe did so by just calling 'do_write_locked_index()' from\n'write_shared_index()'. 'do_write_locked_index()' always at least\ncloses the tempfile nowadays, and used to close or commit the lockfile\nif COMMIT_LOCK or CLOSE_LOCK were given at the time this feature was\nintroduced. COMMIT_LOCK or CLOSE_LOCK is passed in by most callers of\n'write_locked_index()'.\n\nAfter calling 'write_shared_index()', we call 'write_split_index()',\nwhich calls 'do_write_locked_index()' again, which then tries to use the\nclosed lockfile again, but in fact fails to do so as it's already\nclosed.\n\nIn the current version, git will in fact segfault if it can't create a\nnew file in $gitdir, and this feature seems to never have worked in the\nfirst place.\n\nEver since introducing the split index feature, nobody has complained\nabout this failing, and it really just papers over repositories that\nwill sooner or later need fixing anyway.\n\nTherefore just make being unable to write the split index a proper\nerror, and have users fix their repositories instead of trying (but\nfailing) to paper over the error.\n\nSigned-off-by: Thomas Gummerer \n---\n read-cache.c | 11 ++++-------\n 1 file changed, 4 insertions(+), 7 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex d13ce83794..a9c8facdfd 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2471,18 +2471,15 @@ static int clean_shared_index_files(const char *current_hex)\n \treturn 0;\n }\n \n-static int write_shared_index(struct index_state *istate,\n-\t\t\t struct lock_file *lock, unsigned flags)\n+static int write_shared_index(struct index_state *istate)\n {\n \tstruct tempfile *temp;\n \tstruct split_index *si = istate->split_index;\n \tint ret;\n \n \ttemp = mks_tempfile(git_path(\"sharedindex_XXXXXX\"));\n-\tif (!temp) {\n-\t\thashclr(si->base_sha1);\n-\t\treturn do_write_locked_index(istate, lock, flags);\n-\t}\n+\tif (!temp)\n+\t\treturn error(\"cannot create new shared index\");\n \tmove_cache_to_base_index(istate);\n \tret = do_write_index(si->base, temp, 1);\n \tif (ret) {\n@@ -2565,7 +2562,7 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,\n \tnew_shared_index = istate->cache_changed & SPLIT_INDEX_ORDERED;\n \n \tif (new_shared_index) {\n-\t\tret = write_shared_index(istate, lock, flags);\n+\t\tret = write_shared_index(istate);\n \t\tif (ret)\n \t\t\tgoto out;\n \t}\n-- \n2.16.0.rc2.280.g09355b536d\n\n"},{"id":"336574","messageId":"CACsJy8A_moFProjfPAJFn2aP52w5qdYdOu4Ygox1qMMitNUHLg@mail.gmail.com","threadId":"47421","inReplyTo":"20180113223715.GJ2641@hank","subject":"Re: [PATCH v3 4/3] read-cache: don't try to write index if we can't write shared index","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-01-14T09:36:19Z","receivedAt":"2018-01-14T09:36:56Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Jan 14, 2018 at 5:37 AM, Thomas Gummerer wrote:\n> In a0a967568e (\"update-index --split-index: do not split if $GIT_DIR is\n> read only\", 2014-06-13), we tried to make sure we can still write an\n> index, even if the shared index can not be written.\n>\n> We did so by just calling 'do_write_locked_index()' from\n> 'write_shared_index()'. 'do_write_locked_index()' always at least\n> closes the tempfile nowadays, and used to close or commit the lockfile\n> if COMMIT_LOCK or CLOSE_LOCK were given at the time this feature was\n> introduced. COMMIT_LOCK or CLOSE_LOCK is passed in by most callers of\n> 'write_locked_index()'.\n>\n> After calling 'write_shared_index()', we call 'write_split_index()',\n> which calls 'do_write_locked_index()' again, which then tries to use the\n> closed lockfile again, but in fact fails to do so as it's already\n> closed.\n>\n> In the current version, git will in fact segfault if it can't create a\n> new file in $gitdir, and this feature seems to never have worked in the\n> first place.\n>\n> Ever since introducing the split index feature, nobody has complained\n> about this failing, and it really just papers over repositories that\n> will sooner or later need fixing anyway.\n\nActually there's one valid case for this: you're accessing a read-only\n$GIT_DIR (.e.g maybe from a web server cgi script which may be run by\nuser nobody or something) and creating a temporary index _outside_\n$GIT_DIR. I used to do this when I wanted to do \"git grep\" on some\nSHA-1 a couple times. Doing \"git grep \" directly (a couple\ntimes) pays full cost for walking trees. If you prepare an index\nfirst, you pay it only once.\n\n> Therefore just make being unable to write the split index a proper\n> error, and have users fix their repositories instead of trying (but\n> failing) to paper over the error.\n>\n> Signed-off-by: Thomas Gummerer \n> ---\n> read-cache.c | 11 ++++-------\n> 1 file changed, 4 insertions(+), 7 deletions(-)\n>\n> diff --git a/read-cache.c b/read-cache.c\n> index d13ce83794..a9c8facdfd 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -2471,18 +2471,15 @@ static int clean_shared_index_files(const char *current_hex)\n> return 0;\n> }\n>\n> -static int write_shared_index(struct index_state *istate,\n> - struct lock_file *lock, unsigned flags)\n> +static int write_shared_index(struct index_state *istate)\n> {\n> struct tempfile *temp;\n> struct split_index *si = istate->split_index;\n> int ret;\n>\n> temp = mks_tempfile(git_path(\"sharedindex_XXXXXX\"));\n> - if (!temp) {\n> - hashclr(si->base_sha1);\n> - return do_write_locked_index(istate, lock, flags);\n\nI think this code tries to do what's done near the beginning of\nwrite_locked_index() where we also bail out early:\n\n-- 8< --\n if (!si || alternate_index_output ||\n (istate->cache_changed & ~EXTMASK)) {\n if (si)\n hashclr(si->base_sha1);\n ret = do_write_locked_index(istate, lock, flags);\n goto out;\n }\n-- 8< --\n\nthe only difference is it does not realize that it can't do \"goto\nout;\" like that code unless something goes wrong. I'll try to prepare\na patch that move tempfile creation out of write_shared_index()\ninstead. Patches coming in a bit..\n-- \nDuy\n"},{"id":"336576","messageId":"20180114101820.4273-1-pclouds@gmail.com","threadId":"47421","inReplyTo":"CACsJy8A_moFProjfPAJFn2aP52w5qdYdOu4Ygox1qMMitNUHLg@mail.gmail.com","subject":"[PATCH 1/3] read-cache.c: change type of \"temp\" in write_shared_index()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-01-14T10:18:18Z","receivedAt":"2018-01-14T10:18:47Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This local variable 'temp' will be passed in from the caller in the next\npatch. To reduce patch noise, let's change its type now while it's still\na local variable and get all the trival conversion out of the next patch.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n read-cache.c | 20 +++++++++++---------\n 1 file changed, 11 insertions(+), 9 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 2eb81a66b9..536086e1fe 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2474,30 +2474,32 @@ static int clean_shared_index_files(const char *current_hex)\n static int write_shared_index(struct index_state *istate,\n \t\t\t struct lock_file *lock, unsigned flags)\n {\n-\tstruct tempfile *temp;\n+\tstruct tempfile *real_temp;\n+\tstruct tempfile **temp = &real_temp;\n \tstruct split_index *si = istate->split_index;\n \tint ret;\n \n-\ttemp = mks_tempfile(git_path(\"sharedindex_XXXXXX\"));\n-\tif (!temp) {\n+\treal_temp = mks_tempfile(git_path(\"sharedindex_XXXXXX\"));\n+\tif (!real_temp) {\n \t\thashclr(si->base_sha1);\n \t\treturn do_write_locked_index(istate, lock, flags);\n \t}\n+\ttemp = &real_temp;\n \tmove_cache_to_base_index(istate);\n-\tret = do_write_index(si->base, temp, 1);\n+\tret = do_write_index(si->base, *temp, 1);\n \tif (ret) {\n-\t\tdelete_tempfile(&temp);\n+\t\tdelete_tempfile(temp);\n \t\treturn ret;\n \t}\n-\tret = adjust_shared_perm(get_tempfile_path(temp));\n+\tret = adjust_shared_perm(get_tempfile_path(*temp));\n \tif (ret) {\n \t\tint save_errno = errno;\n-\t\terror(\"cannot fix permission bits on %s\", get_tempfile_path(temp));\n-\t\tdelete_tempfile(&temp);\n+\t\terror(\"cannot fix permission bits on %s\", get_tempfile_path(*temp));\n+\t\tdelete_tempfile(temp);\n \t\terrno = save_errno;\n \t\treturn ret;\n \t}\n-\tret = rename_tempfile(&temp,\n+\tret = rename_tempfile(temp,\n \t\t\t git_path(\"sharedindex.%s\", sha1_to_hex(si->base->sha1)));\n \tif (!ret) {\n \t\thashcpy(si->base_sha1, si->base->sha1);\n-- \n2.15.1.600.g899a5f85c6\n\n"},{"id":"336577","messageId":"20180114101820.4273-2-pclouds@gmail.com","threadId":"47421","inReplyTo":"20180114101820.4273-1-pclouds@gmail.com","subject":"[PATCH 2/3] read-cache.c: move tempfile creation/cleanup out of write_shared_index","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-01-14T10:18:19Z","receivedAt":"2018-01-14T10:18:54Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"For one thing, we have more consistent cleanup procedure now and always\nkeep errno intact.\n\nThe real purpose is the ability to break out of write_locked_index()\nearly when mks_tempfile() fails in the next patch. It's more awkward to\ndo it if this mks_tempfile() is still inside write_shared_index().\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n read-cache.c | 33 +++++++++++++++++----------------\n 1 file changed, 17 insertions(+), 16 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 536086e1fe..c568643f55 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2472,31 +2472,18 @@ static int clean_shared_index_files(const char *current_hex)\n }\n \n static int write_shared_index(struct index_state *istate,\n-\t\t\t struct lock_file *lock, unsigned flags)\n+\t\t\t struct tempfile **temp)\n {\n-\tstruct tempfile *real_temp;\n-\tstruct tempfile **temp = &real_temp;\n \tstruct split_index *si = istate->split_index;\n \tint ret;\n \n-\treal_temp = mks_tempfile(git_path(\"sharedindex_XXXXXX\"));\n-\tif (!real_temp) {\n-\t\thashclr(si->base_sha1);\n-\t\treturn do_write_locked_index(istate, lock, flags);\n-\t}\n-\ttemp = &real_temp;\n \tmove_cache_to_base_index(istate);\n \tret = do_write_index(si->base, *temp, 1);\n-\tif (ret) {\n-\t\tdelete_tempfile(temp);\n+\tif (ret)\n \t\treturn ret;\n-\t}\n \tret = adjust_shared_perm(get_tempfile_path(*temp));\n \tif (ret) {\n-\t\tint save_errno = errno;\n \t\terror(\"cannot fix permission bits on %s\", get_tempfile_path(*temp));\n-\t\tdelete_tempfile(temp);\n-\t\terrno = save_errno;\n \t\treturn ret;\n \t}\n \tret = rename_tempfile(temp,\n@@ -2567,7 +2554,21 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,\n \tnew_shared_index = istate->cache_changed & SPLIT_INDEX_ORDERED;\n \n \tif (new_shared_index) {\n-\t\tret = write_shared_index(istate, lock, flags);\n+\t\tstruct tempfile *temp;\n+\t\tint saved_errno;\n+\n+\t\ttemp = mks_tempfile(git_path(\"sharedindex_XXXXXX\"));\n+\t\tif (!temp) {\n+\t\t\thashclr(si->base_sha1);\n+\t\t\tret = do_write_locked_index(istate, lock, flags);\n+\t\t} else\n+\t\t\tret = write_shared_index(istate, &temp);\n+\n+\t\tsaved_errno = errno;\n+\t\tif (is_tempfile_active(temp))\n+\t\t\tdelete_tempfile(&temp);\n+\t\terrno = saved_errno;\n+\n \t\tif (ret)\n \t\t\tgoto out;\n \t}\n-- \n2.15.1.600.g899a5f85c6\n\n"},{"id":"336578","messageId":"20180114101820.4273-3-pclouds@gmail.com","threadId":"47421","inReplyTo":"20180114101820.4273-1-pclouds@gmail.com","subject":"[PATCH 3/3] read-cache: don't write index twice if we can't write shared index","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2018-01-14T10:18:20Z","receivedAt":"2018-01-14T10:19:01Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"In a0a967568e (\"update-index --split-index: do not split if $GIT_DIR is\nread only\", 2014-06-13), we tried to make sure we can still write an\nindex, even if the shared index can not be written.\n\nWe did so by just calling 'do_write_locked_index()' just before\n'write_shared_index()'. 'do_write_locked_index()' always at least\ncloses the tempfile nowadays, and used to close or commit the lockfile\nif COMMIT_LOCK or CLOSE_LOCK were given at the time this feature was\nintroduced. COMMIT_LOCK or CLOSE_LOCK is passed in by most callers of\n'write_locked_index()'.\n\nAfter calling 'write_shared_index()', we call 'write_split_index()',\nwhich calls 'do_write_locked_index()' again, which then tries to use the\nclosed lockfile again, but in fact fails to do so as it's already\nclosed. This eventually leads to a segfault.\n\nMake sure to write the main index only once.\n\n[nd: most of the commit message and investigation done by Thomas, I only\ntweaked the solution a bit]\n\nHelped-by: Thomas Gummerer \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n read-cache.c | 5 +++--\n t/t1700-split-index.sh | 19 +++++++++++++++++++\n 2 files changed, 22 insertions(+), 2 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex c568643f55..c58c0a978a 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2561,8 +2561,9 @@ int write_locked_index(struct index_state *istate, struct lock_file *lock,\n \t\tif (!temp) {\n \t\t\thashclr(si->base_sha1);\n \t\t\tret = do_write_locked_index(istate, lock, flags);\n-\t\t} else\n-\t\t\tret = write_shared_index(istate, &temp);\n+\t\t\tgoto out;\n+\t\t}\n+\t\tret = write_shared_index(istate, &temp);\n \n \t\tsaved_errno = errno;\n \t\tif (is_tempfile_active(temp))\ndiff --git a/t/t1700-split-index.sh b/t/t1700-split-index.sh\nindex af9b847761..5494389dbb 100755\n--- a/t/t1700-split-index.sh\n+++ b/t/t1700-split-index.sh\n@@ -401,4 +401,23 @@ done <<\\EOF\n 0642 -rw-r---w-\n EOF\n \n+test_expect_success POSIXPERM 'graceful handling splitting index is not allowed' '\n+\ttest_create_repo ro &&\n+\t(\n+\t\tcd ro &&\n+\t\ttest_commit initial &&\n+\t\tgit update-index --split-index &&\n+\t\ttest -f .git/sharedindex.*\n+\t) &&\n+\ttest_when_finished \"chmod -R u+w ro\" &&\n+\tchmod -R u-w ro &&\n+\tcp ro/.git/index new-index &&\n+\tGIT_INDEX_FILE=\"$(pwd)/new-index\" git -C ro update-index --split-index &&\n+\tchmod -R u+w ro &&\n+\trm ro/.git/sharedindex.* &&\n+\tGIT_INDEX_FILE=new-index git ls-files >actual &&\n+\techo initial.t >expected &&\n+\ttest_cmp expected actual\n+'\n+\n test_done\n-- \n2.15.1.600.g899a5f85c6\n\n"},{"id":"336584","messageId":"20180114142914.GK2641@hank","threadId":"47421","inReplyTo":"CACsJy8A_moFProjfPAJFn2aP52w5qdYdOu4Ygox1qMMitNUHLg@mail.gmail.com","subject":"Re: [PATCH v3 4/3] read-cache: don't try to write index if we can't write shared index","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2018-01-14T14:29:14Z","receivedAt":"2018-01-14T14:27:05Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 01/14, Duy Nguyen wrote:\n> On Sun, Jan 14, 2018 at 5:37 AM, Thomas Gummerer wrote:\n> > In a0a967568e (\"update-index --split-index: do not split if $GIT_DIR is\n> > read only\", 2014-06-13), we tried to make sure we can still write an\n> > index, even if the shared index can not be written.\n> >\n> > We did so by just calling 'do_write_locked_index()' from\n> > 'write_shared_index()'. 'do_write_locked_index()' always at least\n> > closes the tempfile nowadays, and used to close or commit the lockfile\n> > if COMMIT_LOCK or CLOSE_LOCK were given at the time this feature was\n> > introduced. COMMIT_LOCK or CLOSE_LOCK is passed in by most callers of\n> > 'write_locked_index()'.\n> >\n> > After calling 'write_shared_index()', we call 'write_split_index()',\n> > which calls 'do_write_locked_index()' again, which then tries to use the\n> > closed lockfile again, but in fact fails to do so as it's already\n> > closed.\n> >\n> > In the current version, git will in fact segfault if it can't create a\n> > new file in $gitdir, and this feature seems to never have worked in the\n> > first place.\n> >\n> > Ever since introducing the split index feature, nobody has complained\n> > about this failing, and it really just papers over repositories that\n> > will sooner or later need fixing anyway.\n> \n> Actually there's one valid case for this: you're accessing a read-only\n> $GIT_DIR (.e.g maybe from a web server cgi script which may be run by\n> user nobody or something) and creating a temporary index _outside_\n> $GIT_DIR. I used to do this when I wanted to do \"git grep\" on some\n> SHA-1 a couple times. Doing \"git grep \" directly (a couple\n> times) pays full cost for walking trees. If you prepare an index\n> first, you pay it only once.\n\nMakes sense, I didn't realize that usecase, thanks!\n\n> > Therefore just make being unable to write the split index a proper\n> > error, and have users fix their repositories instead of trying (but\n> > failing) to paper over the error.\n> >\n> > Signed-off-by: Thomas Gummerer \n> > ---\n> > read-cache.c | 11 ++++-------\n> > 1 file changed, 4 insertions(+), 7 deletions(-)\n> >\n> > diff --git a/read-cache.c b/read-cache.c\n> > index d13ce83794..a9c8facdfd 100644\n> > --- a/read-cache.c\n> > +++ b/read-cache.c\n> > @@ -2471,18 +2471,15 @@ static int clean_shared_index_files(const char *current_hex)\n> > return 0;\n> > }\n> >\n> > -static int write_shared_index(struct index_state *istate,\n> > - struct lock_file *lock, unsigned flags)\n> > +static int write_shared_index(struct index_state *istate)\n> > {\n> > struct tempfile *temp;\n> > struct split_index *si = istate->split_index;\n> > int ret;\n> >\n> > temp = mks_tempfile(git_path(\"sharedindex_XXXXXX\"));\n> > - if (!temp) {\n> > - hashclr(si->base_sha1);\n> > - return do_write_locked_index(istate, lock, flags);\n> \n> I think this code tries to do what's done near the beginning of\n> write_locked_index() where we also bail out early:\n> \n> -- 8< --\n> if (!si || alternate_index_output ||\n> (istate->cache_changed & ~EXTMASK)) {\n> if (si)\n> hashclr(si->base_sha1);\n> ret = do_write_locked_index(istate, lock, flags);\n> goto out;\n> }\n> -- 8< --\n> \n> the only difference is it does not realize that it can't do \"goto\n> out;\" like that code unless something goes wrong. I'll try to prepare\n> a patch that move tempfile creation out of write_shared_index()\n> instead. Patches coming in a bit..\n\nThanks for fixing this in a nicer way :)\n\n> -- \n> Duy\n"},{"id":"336676","messageId":"20180116214239.GA3622@google.com","threadId":"47421","inReplyTo":"20180107223015.17720-2-t.gummerer@gmail.com","subject":"Re: [PATCH v3 1/3] read-cache: fix reading the shared index for other repos","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-16T21:42:39Z","receivedAt":"2018-01-16T21:42:47Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/07, Thomas Gummerer wrote:\n> read_index_from() takes a path argument for the location of the index\n> file. For reading the shared index in split index mode however it just\n> ignores that path argument, and reads it from the gitdir of the current\n> repository.\n> \n> This works as long as an index in the_repository is read. Once that\n> changes, such as when we read the index of a submodule, or of a\n> different working tree than the current one, the gitdir of\n> the_repository will no longer contain the appropriate shared index,\n> and git will fail to read it.\n> \n> For example t3007-ls-files-recurse-submodules.sh was broken with\n> GIT_TEST_SPLIT_INDEX set in 188dce131f (\"ls-files: use repository\n> object\", 2017-06-22), and t7814-grep-recurse-submodules.sh was also\n> broken in a similar manner, probably by introducing struct repository\n> there, although I didn't track down the exact commit for that.\n> \n> be489d02d2 (\"revision.c: --indexed-objects add objects from all\n> worktrees\", 2017-08-23) breaks with split index mode in a similar\n> manner, not erroring out when it can't read the index, but instead\n> carrying on with pruning, without taking the index of the worktree into\n> account.\n> \n> Fix this by passing an additional gitdir parameter to read_index_from,\n> to indicate where it should look for and read the shared index from.\n> \n> read_cache_from() defaults to using the gitdir of the_repository. As it\n> is mostly a convenience macro, having to pass get_git_dir() for every\n> call seems overkill, and if necessary users can have more control by\n> using read_index_from().\n\nI'm not saying we need to change this again but I got to thinking about\nwhat the root cause for this bug is and I think it's a design flaw in\nhow split index is implemented. IIUC Split index is an index extension\nthat can be enabled to limit the size of the index file that is written\nwhen making changes to the index. It breaks the index into two pieces,\nindex (which contains only changes) and sharedindex.XXXXX (which\ncontains unchanged information) where 'XXXXX' is a value found in the\nindex file. If we don't do anything fancy then these two files live\nnext to one another in a repository's git directory at $GIT_DIR/index\nand $GIT_DIR/sharedindex.XXXXX. This seems to work all well and fine\nexcept that this isn't always the case and the read_index_from function\ntakes this into account by enabling a caller to specify a path to where\nthe index file is located. We can do this by specifying the index file\nwe want to use by setting GIT_INDEX_FILE.\n\nBeing able to specify an index file via the environment is a feature\nthat has existed for a very long time (one that I personally think makes\nthings difficult because of things like additions like the sharedindex)\nand I don't think was taken into account when introducing the split\nindex extension. In this case if i were to specify a location of an\nindex file in my home directory '~/index' and be using the split index\nfeature then the corresponding sharedindex file would live in my\nrepository's git directory '~/project/.git/sharedindex.XXXXX'. So the\nsharedindex file is always located relative to the project's git\ndirectory and not the index file itself, which is kind of confusing.\nMaybe a better design would be to have the sharedindex file located\nrelative to the index file.\n\nAnyway, some food for thought.\n\n-- \nBrandon Williams\n"},{"id":"336681","messageId":"CACsJy8A9AheuBK-mKy9d9m5yfSFFZ9N8=wg8FZD_C6mHHTdqXg@mail.gmail.com","threadId":"47421","inReplyTo":"20180116214239.GA3622@google.com","subject":"Re: [PATCH v3 1/3] read-cache: fix reading the shared index for other repos","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-01-17T00:16:38Z","receivedAt":"2018-01-17T00:17:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Jan 17, 2018 at 4:42 AM, Brandon Williams wrote:\n> On 01/07, Thomas Gummerer wrote:\n>> read_index_from() takes a path argument for the location of the index\n>> file. For reading the shared index in split index mode however it just\n>> ignores that path argument, and reads it from the gitdir of the current\n>> repository.\n>>\n>> This works as long as an index in the_repository is read. Once that\n>> changes, such as when we read the index of a submodule, or of a\n>> different working tree than the current one, the gitdir of\n>> the_repository will no longer contain the appropriate shared index,\n>> and git will fail to read it.\n>>\n>> For example t3007-ls-files-recurse-submodules.sh was broken with\n>> GIT_TEST_SPLIT_INDEX set in 188dce131f (\"ls-files: use repository\n>> object\", 2017-06-22), and t7814-grep-recurse-submodules.sh was also\n>> broken in a similar manner, probably by introducing struct repository\n>> there, although I didn't track down the exact commit for that.\n>>\n>> be489d02d2 (\"revision.c: --indexed-objects add objects from all\n>> worktrees\", 2017-08-23) breaks with split index mode in a similar\n>> manner, not erroring out when it can't read the index, but instead\n>> carrying on with pruning, without taking the index of the worktree into\n>> account.\n>>\n>> Fix this by passing an additional gitdir parameter to read_index_from,\n>> to indicate where it should look for and read the shared index from.\n>>\n>> read_cache_from() defaults to using the gitdir of the_repository. As it\n>> is mostly a convenience macro, having to pass get_git_dir() for every\n>> call seems overkill, and if necessary users can have more control by\n>> using read_index_from().\n>\n> I'm not saying we need to change this again but I got to thinking about\n> what the root cause for this bug is and I think it's a design flaw in\n> how split index is implemented. IIUC Split index is an index extension\n> that can be enabled to limit the size of the index file that is written\n> when making changes to the index. It breaks the index into two pieces,\n> index (which contains only changes) and sharedindex.XXXXX (which\n> contains unchanged information) where 'XXXXX' is a value found in the\n> index file. If we don't do anything fancy then these two files live\n> next to one another in a repository's git directory at $GIT_DIR/index\n> and $GIT_DIR/sharedindex.XXXXX. This seems to work all well and fine\n> except that this isn't always the case and the read_index_from function\n> takes this into account by enabling a caller to specify a path to where\n> the index file is located. We can do this by specifying the index file\n> we want to use by setting GIT_INDEX_FILE.\n>\n> Being able to specify an index file via the environment is a feature\n> that has existed for a very long time (one that I personally think makes\n> things difficult because of things like additions like the sharedindex)\n> and I don't think was taken into account when introducing the split\n> index extension.\n\nIt was (partly because I did use GIT_INDEX_FILE on occasions).\n\n> In this case if i were to specify a location of an\n> index file in my home directory '~/index' and be using the split index\n> feature then the corresponding sharedindex file would live in my\n> repository's git directory '~/project/.git/sharedindex.XXXXX'. So the\n> sharedindex file is always located relative to the project's git\n> directory and not the index file itself, which is kind of confusing.\n> Maybe a better design would be to have the sharedindex file located\n> relative to the index file.\n\nThat adds more problems. Now when you move the index file around you\nhave to move the shared index file too (think about atomic rename\nwhich we use in plenty of places, we can't achieve that by moving two\nfiles). A new dependency to $GIT_DIR is not that confusing to me, the\nindex file is useless anyway if you don't have access to\n$GIT_DIR/objects. There was always the option to _not_ split the index\nwhen $GIT_INDEX_FILE is specified, I think I did consider that but I\ndropped it because we'd lose the performance gain by splitting.\n\n> Anyway, some food for thought.\n>\n> --\n> Brandon Williams\n\n\n\n-- \nDuy\n"},{"id":"336684","messageId":"20180117003208.qkjtwijmb6o3n6xn@google.com","threadId":"47421","inReplyTo":"CACsJy8A9AheuBK-mKy9d9m5yfSFFZ9N8=wg8FZD_C6mHHTdqXg@mail.gmail.com","subject":"Re: [PATCH v3 1/3] read-cache: fix reading the shared index for other repos","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2018-01-17T00:32:08Z","receivedAt":"2018-01-17T00:32:15Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 01/17, Duy Nguyen wrote:\n> On Wed, Jan 17, 2018 at 4:42 AM, Brandon Williams wrote:\n> > On 01/07, Thomas Gummerer wrote:\n> >> read_index_from() takes a path argument for the location of the index\n> >> file. For reading the shared index in split index mode however it just\n> >> ignores that path argument, and reads it from the gitdir of the current\n> >> repository.\n> >>\n> >> This works as long as an index in the_repository is read. Once that\n> >> changes, such as when we read the index of a submodule, or of a\n> >> different working tree than the current one, the gitdir of\n> >> the_repository will no longer contain the appropriate shared index,\n> >> and git will fail to read it.\n> >>\n> >> For example t3007-ls-files-recurse-submodules.sh was broken with\n> >> GIT_TEST_SPLIT_INDEX set in 188dce131f (\"ls-files: use repository\n> >> object\", 2017-06-22), and t7814-grep-recurse-submodules.sh was also\n> >> broken in a similar manner, probably by introducing struct repository\n> >> there, although I didn't track down the exact commit for that.\n> >>\n> >> be489d02d2 (\"revision.c: --indexed-objects add objects from all\n> >> worktrees\", 2017-08-23) breaks with split index mode in a similar\n> >> manner, not erroring out when it can't read the index, but instead\n> >> carrying on with pruning, without taking the index of the worktree into\n> >> account.\n> >>\n> >> Fix this by passing an additional gitdir parameter to read_index_from,\n> >> to indicate where it should look for and read the shared index from.\n> >>\n> >> read_cache_from() defaults to using the gitdir of the_repository. As it\n> >> is mostly a convenience macro, having to pass get_git_dir() for every\n> >> call seems overkill, and if necessary users can have more control by\n> >> using read_index_from().\n> >\n> > I'm not saying we need to change this again but I got to thinking about\n> > what the root cause for this bug is and I think it's a design flaw in\n> > how split index is implemented. IIUC Split index is an index extension\n> > that can be enabled to limit the size of the index file that is written\n> > when making changes to the index. It breaks the index into two pieces,\n> > index (which contains only changes) and sharedindex.XXXXX (which\n> > contains unchanged information) where 'XXXXX' is a value found in the\n> > index file. If we don't do anything fancy then these two files live\n> > next to one another in a repository's git directory at $GIT_DIR/index\n> > and $GIT_DIR/sharedindex.XXXXX. This seems to work all well and fine\n> > except that this isn't always the case and the read_index_from function\n> > takes this into account by enabling a caller to specify a path to where\n> > the index file is located. We can do this by specifying the index file\n> > we want to use by setting GIT_INDEX_FILE.\n> >\n> > Being able to specify an index file via the environment is a feature\n> > that has existed for a very long time (one that I personally think makes\n> > things difficult because of things like additions like the sharedindex)\n> > and I don't think was taken into account when introducing the split\n> > index extension.\n> \n> It was (partly because I did use GIT_INDEX_FILE on occasions).\n> \n> > In this case if i were to specify a location of an\n> > index file in my home directory '~/index' and be using the split index\n> > feature then the corresponding sharedindex file would live in my\n> > repository's git directory '~/project/.git/sharedindex.XXXXX'. So the\n> > sharedindex file is always located relative to the project's git\n> > directory and not the index file itself, which is kind of confusing.\n> > Maybe a better design would be to have the sharedindex file located\n> > relative to the index file.\n> \n> That adds more problems. Now when you move the index file around you\n> have to move the shared index file too (think about atomic rename\n> which we use in plenty of places, we can't achieve that by moving two\n> files). A new dependency to $GIT_DIR is not that confusing to me, the\n> index file is useless anyway if you don't have access to\n> $GIT_DIR/objects. There was always the option to _not_ split the index\n> when $GIT_INDEX_FILE is specified, I think I did consider that but I\n> dropped it because we'd lose the performance gain by splitting.\n> \n> > Anyway, some food for thought.\n> >\n> > --\n> > Brandon Williams\n> \n> \n> \n> -- \n> Duy\n\nThanks for giving me some more context :) makes me feel more confident\nin the solution that's already been proposed.\n\n-- \nBrandon Williams\n"},{"id":"336706","messageId":"20180117181618.GA112319@aiede.svl.corp.google.com","threadId":"47421","inReplyTo":"CACsJy8A9AheuBK-mKy9d9m5yfSFFZ9N8=wg8FZD_C6mHHTdqXg@mail.gmail.com","subject":"Re: [PATCH v3 1/3] read-cache: fix reading the shared index for other repos","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-01-17T18:16:18Z","receivedAt":"2018-01-17T18:16:53Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nDuy Nguyen wrote:\n> On Wed, Jan 17, 2018 at 4:42 AM, Brandon Williams wrote:\n\n>> IIUC Split index is an index extension\n>> that can be enabled to limit the size of the index file that is written\n>> when making changes to the index. It breaks the index into two pieces,\n>> index (which contains only changes) and sharedindex.XXXXX (which\n>> contains unchanged information) where 'XXXXX' is a value found in the\n>> index file. If we don't do anything fancy then these two files live\n>> next to one another in a repository's git directory at $GIT_DIR/index\n>> and $GIT_DIR/sharedindex.XXXXX. This seems to work all well and fine\n>> except that this isn't always the case and the read_index_from function\n>> takes this into account by enabling a caller to specify a path to where\n>> the index file is located. We can do this by specifying the index file\n>> we want to use by setting GIT_INDEX_FILE.\n[...]\n>> In this case if i were to specify a location of an\n>> index file in my home directory '~/index' and be using the split index\n>> feature then the corresponding sharedindex file would live in my\n>> repository's git directory '~/project/.git/sharedindex.XXXXX'. So the\n>> sharedindex file is always located relative to the project's git\n>> directory and not the index file itself, which is kind of confusing.\n>> Maybe a better design would be to have the sharedindex file located\n>> relative to the index file.\n>\n> That adds more problems. Now when you move the index file around you\n> have to move the shared index file too (think about atomic rename\n> which we use in plenty of places, we can't achieve that by moving two\n> files). A new dependency to $GIT_DIR is not that confusing to me, the\n> index file is useless anyway if you don't have access to\n> $GIT_DIR/objects. There was always the option to _not_ split the index\n> when $GIT_INDEX_FILE is specified, I think I did consider that but I\n> dropped it because we'd lose the performance gain by splitting.\n\nCan you elaborate a little more on this?\n\nAt first glance, it seems simpler to say \"paths in index extensions\nnamed in the index file are relative to the location of the index\nfile\" and to make moving the index file also require moving the shared\nindex file, exactly as you say. So at least from a \"principle of\nleast surprise\" perspective I would be tempted to go that way.\n\nIt's true that we rely on atomic rename in plenty of places, but only\nwithin a directory. (Filesystem boundaries, NFS, etc mean that atomic\nrenames across directories are a lost cause.)\n\nFortunately index files (including temp index files used by scripts)\ntend to only be in $GIT_DIR, for exactly that reason. So I am\nwondering if switching to index-file-relative semantics would be an\ninvasive move and what the pros and cons of such a move are.\n\nThanks,\nJonathan\n"},{"id":"336766","messageId":"CACsJy8D+SjWsTtwBSPbw5cVVSuVFYP6GVRSWaP7AWSgLC3e=TQ@mail.gmail.com","threadId":"47421","inReplyTo":"20180117181618.GA112319@aiede.svl.corp.google.com","subject":"Re: [PATCH v3 1/3] read-cache: fix reading the shared index for other repos","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-01-18T10:19:22Z","receivedAt":"2018-01-18T10:20:00Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jan 18, 2018 at 1:16 AM, Jonathan Nieder wrote:\n> Hi,\n>\n> Duy Nguyen wrote:\n>> On Wed, Jan 17, 2018 at 4:42 AM, Brandon Williams wrote:\n>\n>>> IIUC Split index is an index extension\n>>> that can be enabled to limit the size of the index file that is written\n>>> when making changes to the index. It breaks the index into two pieces,\n>>> index (which contains only changes) and sharedindex.XXXXX (which\n>>> contains unchanged information) where 'XXXXX' is a value found in the\n>>> index file. If we don't do anything fancy then these two files live\n>>> next to one another in a repository's git directory at $GIT_DIR/index\n>>> and $GIT_DIR/sharedindex.XXXXX. This seems to work all well and fine\n>>> except that this isn't always the case and the read_index_from function\n>>> takes this into account by enabling a caller to specify a path to where\n>>> the index file is located. We can do this by specifying the index file\n>>> we want to use by setting GIT_INDEX_FILE.\n> [...]\n>>> In this case if i were to specify a location of an\n>>> index file in my home directory '~/index' and be using the split index\n>>> feature then the corresponding sharedindex file would live in my\n>>> repository's git directory '~/project/.git/sharedindex.XXXXX'. So the\n>>> sharedindex file is always located relative to the project's git\n>>> directory and not the index file itself, which is kind of confusing.\n>>> Maybe a better design would be to have the sharedindex file located\n>>> relative to the index file.\n>>\n>> That adds more problems. Now when you move the index file around you\n>> have to move the shared index file too (think about atomic rename\n>> which we use in plenty of places, we can't achieve that by moving two\n>> files). A new dependency to $GIT_DIR is not that confusing to me, the\n>> index file is useless anyway if you don't have access to\n>> $GIT_DIR/objects. There was always the option to _not_ split the index\n>> when $GIT_INDEX_FILE is specified, I think I did consider that but I\n>> dropped it because we'd lose the performance gain by splitting.\n>\n> Can you elaborate a little more on this?\n>\n> At first glance, it seems simpler to say \"paths in index extensions\n> named in the index file are relative to the location of the index\n> file\" and to make moving the index file also require moving the shared\n> index file, exactly as you say. So at least from a \"principle of\n> least surprise\" perspective I would be tempted to go that way.\n>\n> It's true that we rely on atomic rename in plenty of places, but only\n> within a directory. (Filesystem boundaries, NFS, etc mean that atomic\n> renames across directories are a lost cause.)\n>\n> Fortunately index files (including temp index files used by scripts)\n> tend to only be in $GIT_DIR, for exactly that reason. So I am\n> wondering if switching to index-file-relative semantics would be an\n> invasive move and what the pros and cons of such a move are.\n\nI think it gets messier. Now you have to move two files. If the first\nmove succeeds but the second one fails, recovery may involve un-move\nthe first file, but its old content is already gone. We probably can\nget around that. But since the shared index is assumed big and heavy,\nI just went with \"store it in the place it's going to be and never\nmove it anywhere ever (until nobody uses it then it's deleted)\"\n-- \nDuy\n"},{"id":"336769","messageId":"CAM0VKjmGo5gjUBpTQp9M+pvZzBMDAmYYCu-5VfG-8LYRxU7i0Q@mail.gmail.com","threadId":"47421","inReplyTo":"20180114101820.4273-3-pclouds@gmail.com","subject":"Re: [PATCH 3/3] read-cache: don't write index twice if we can't write shared index","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-01-18T11:36:06Z","receivedAt":"2018-01-18T11:36:12Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"This series, queued as 'nd/shared-index-fix', makes the 32 bit Linux\nbuild job fail on Travis CI. Unfortunately, all it can tell us about\nthe failure is this:\n\n Test Summary Report\n -------------------\n t1700-split-index.sh (Wstat: 256 Tests: 23\n Failed: 1)\n Failed test: 23\n Non-zero exit status: 1\n Files=809, Tests=18491, 401 wallclock secs ( 7.22 usr 1.60 sys + 263.16\n cusr 49.58 csys = 321.56 CPU)\n Result: FAIL\n\nbecause it can't access the test's verbose log due to lack of\npermissions.\n\n\n https://travis-ci.org/git/git/jobs/329681826#L2074\n"},{"id":"336771","messageId":"CACsJy8BBXQ9KErfiuf2ty_4szE2fiHLDiKvMig1LbSefzf-o7w@mail.gmail.com","threadId":"47421","inReplyTo":"CAM0VKjmGo5gjUBpTQp9M+pvZzBMDAmYYCu-5VfG-8LYRxU7i0Q@mail.gmail.com","subject":"Re: [PATCH 3/3] read-cache: don't write index twice if we can't write shared index","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-01-18T12:47:50Z","receivedAt":"2018-01-18T12:48:26Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jan 18, 2018 at 6:36 PM, SZEDER Gábor wrote:\n> This series, queued as 'nd/shared-index-fix', makes the 32 bit Linux\n> build job fail on Travis CI. Unfortunately, all it can tell us about\n> the failure is this:\n>\n> Test Summary Report\n> -------------------\n> t1700-split-index.sh (Wstat: 256 Tests: 23\n> Failed: 1)\n> Failed test: 23\n> Non-zero exit status: 1\n> Files=809, Tests=18491, 401 wallclock secs ( 7.22 usr 1.60 sys + 263.16\n> cusr 49.58 csys = 321.56 CPU)\n> Result: FAIL\n>\n> because it can't access the test's verbose log due to lack of\n> permissions.\n>\n>\n> https://travis-ci.org/git/git/jobs/329681826#L2074\n\nI may need help getting that log (or even better the trash directory\nof t1700). I tried -m32 build, then valgrind on amd64 (because it\ndidn't work with my 32 build), running the tests with dash and even\nthe linux32 docker version that comes with \"ci\" directory. Everything\nworked for me.\n-- \nDuy\n"},{"id":"336772","messageId":"20180118132931.GA21914@sigill.intra.peff.net","threadId":"47421","inReplyTo":"CACsJy8BBXQ9KErfiuf2ty_4szE2fiHLDiKvMig1LbSefzf-o7w@mail.gmail.com","subject":"Re: [PATCH 3/3] read-cache: don't write index twice if we can't write shared index","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-01-18T13:29:31Z","receivedAt":"2018-01-18T13:29:39Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 18, 2018 at 07:47:50PM +0700, Duy Nguyen wrote:\n\n> I may need help getting that log (or even better the trash directory\n> of t1700). I tried -m32 build, then valgrind on amd64 (because it\n> didn't work with my 32 build), running the tests with dash and even\n> the linux32 docker version that comes with \"ci\" directory. Everything\n> worked for me.\n\nIt fails for me locally if I run it in the linux32 docker environment.\nHere's the end of the \"-v -x\" output:\n\n + GIT_INDEX_FILE=/usr/src/git/t/trash directory.t1700-split-index/new-index git -C ro update-index --split-index\n + chmod -R u+w ro\n + rm ro/.git/sharedindex.bbdf63f5a51242904dba2ea929ea4f56fcc340b6\n + GIT_INDEX_FILE=new-index git ls-files\n fatal: .git/sharedindex.bbdf63f5a51242904dba2ea929ea4f56fcc340b6: index file open failed: No such file or directory\n error: last command exited with $?=128\n not ok 23 - graceful handling splitting index is not allowed\n\nI don't know if the trash state will be helpful, but it's attached.\n\n-Peff\n"},{"id":"336773","messageId":"CACsJy8BkJXeaG3mYNEsfVK4N_-WxwU2Pk2pa1FThH1sMEW2yxA@mail.gmail.com","threadId":"47421","inReplyTo":"20180118132931.GA21914@sigill.intra.peff.net","subject":"Re: [PATCH 3/3] read-cache: don't write index twice if we can't write shared index","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-01-18T13:36:32Z","receivedAt":"2018-01-18T13:37:07Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jan 18, 2018 at 8:29 PM, Jeff King wrote:\n> On Thu, Jan 18, 2018 at 07:47:50PM +0700, Duy Nguyen wrote:\n>\n>> I may need help getting that log (or even better the trash directory\n>> of t1700). I tried -m32 build, then valgrind on amd64 (because it\n>> didn't work with my 32 build), running the tests with dash and even\n>> the linux32 docker version that comes with \"ci\" directory. Everything\n>> worked for me.\n>\n> It fails for me locally if I run it in the linux32 docker environment.\n> Here's the end of the \"-v -x\" output:\n>\n> + GIT_INDEX_FILE=/usr/src/git/t/trash directory.t1700-split-index/new-index git -C ro update-index --split-index\n> + chmod -R u+w ro\n> + rm ro/.git/sharedindex.bbdf63f5a51242904dba2ea929ea4f56fcc340b6\n> + GIT_INDEX_FILE=new-index git ls-files\n> fatal: .git/sharedindex.bbdf63f5a51242904dba2ea929ea4f56fcc340b6: index file open failed: No such file or directory\n> error: last command exited with $?=128\n> not ok 23 - graceful handling splitting index is not allowed\n>\n> I don't know if the trash state will be helpful, but it's attached.\n\nThanks. Somehow my way of forcing mks_tempfile() to fail, well..\nfailed in this environment. I see the same output if I remove \"chmod\n-R u-w ro\". I think I have enough to continue from here.\n-- \nDuy\n"},{"id":"336791","messageId":"20180118150014.GA2097@ash","threadId":"47421","inReplyTo":"CACsJy8BkJXeaG3mYNEsfVK4N_-WxwU2Pk2pa1FThH1sMEW2yxA@mail.gmail.com","subject":"Re: [PATCH 3/3] read-cache: don't write index twice if we can't write shared index","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-01-18T15:00:14Z","receivedAt":"2018-01-18T15:00:38Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jan 18, 2018 at 08:36:32PM +0700, Duy Nguyen wrote:\n> On Thu, Jan 18, 2018 at 8:29 PM, Jeff King wrote:\n> > On Thu, Jan 18, 2018 at 07:47:50PM +0700, Duy Nguyen wrote:\n> >\n> >> I may need help getting that log (or even better the trash directory\n> >> of t1700). I tried -m32 build, then valgrind on amd64 (because it\n> >> didn't work with my 32 build), running the tests with dash and even\n> >> the linux32 docker version that comes with \"ci\" directory. Everything\n> >> worked for me.\n> >\n> > It fails for me locally if I run it in the linux32 docker environment.\n> > Here's the end of the \"-v -x\" output:\n> >\n> > + GIT_INDEX_FILE=/usr/src/git/t/trash directory.t1700-split-index/new-index git -C ro update-index --split-index\n> > + chmod -R u+w ro\n> > + rm ro/.git/sharedindex.bbdf63f5a51242904dba2ea929ea4f56fcc340b6\n> > + GIT_INDEX_FILE=new-index git ls-files\n> > fatal: .git/sharedindex.bbdf63f5a51242904dba2ea929ea4f56fcc340b6: index file open failed: No such file or directory\n> > error: last command exited with $?=128\n> > not ok 23 - graceful handling splitting index is not allowed\n> >\n> > I don't know if the trash state will be helpful, but it's attached.\n> \n> Thanks. Somehow my way of forcing mks_tempfile() to fail, well..\n> failed in this environment. I see the same output if I remove \"chmod\n> -R u-w ro\". I think I have enough to continue from here.\n\nThe test suite was run as root, no wonder why my removing write access\nhas no effect. I got the test to pass with this, but then it fails\nwith\n\n Can't write .prove (Permission denied) at /usr/share/perl/5.22/App/Prove.pm line 542.\n\nSome more chown'ing or chmod'ing is required....\n\n-- 8< --\nSubject: [PATCH] ci: don't accidentally run the test suite as root\n\nThis script assigns and adds a user named \"ci\" in a subshell so the\nouter CI_USER is not affected. For some reason, CI_USER is actually\nempty on Travis linux32 builds. This makes the following \"su\" useless\nand the test suite is run as root.\n\nSome tests may expect file/dir permissions to be respected. But root\nrules all and ignores all. That could make tests fail.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n ci/run-linux32-build.sh | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh\nindex c19c50c1c9..92b488ff27 100755\n--- a/ci/run-linux32-build.sh\n+++ b/ci/run-linux32-build.sh\n@@ -21,8 +21,8 @@ linux32 --32bit i386 sh -c '\n # If a host user id is given, then create a user \"ci\" with the host user\n # id to make everything accessible to the host user.\n HOST_UID=$1 &&\n-CI_USER=$USER &&\n-test -z $HOST_UID || (CI_USER=\"ci\" && useradd -u $HOST_UID $CI_USER) &&\n+CI_USER=${USER:-ci} &&\n+test -z \"$HOST_UID\" || useradd -u $HOST_UID $CI_USER &&\n \n # Build and test\n linux32 --32bit i386 su -m -l $CI_USER -c '\n-- \n2.16.0.47.g5ff498d35b\n\n-- 8< --\n"},{"id":"336830","messageId":"20180118213732.GA7403@sigill.intra.peff.net","threadId":"47421","inReplyTo":"20180118150014.GA2097@ash","subject":"Re: [PATCH 3/3] read-cache: don't write index twice if we can't write shared index","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-01-18T21:37:32Z","receivedAt":"2018-01-18T21:37:43Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 18, 2018 at 10:00:14PM +0700, Duy Nguyen wrote:\n\n> The test suite was run as root, no wonder why my removing write access\n> has no effect. I got the test to pass with this, but then it fails\n> with\n> \n> Can't write .prove (Permission denied) at /usr/share/perl/5.22/App/Prove.pm line 542.\n> \n> Some more chown'ing or chmod'ing is required....\n\nAh, right. I agree that we probably ought to run the ci as non-root.\nHowever, if the test requires non-root, it probably needs to be marked\nwith the SANITY prereq.\n\nI also ran into one funny thing: if you run the script with \"-i\", then\nwe do not run the test_when_finished block. And therefore the \"ro\"\ndirectory is left without its write bit, and the next test run fails, as\nit cannot \"rm -rf\" the old trash directory out of the way.\n\nI'm not sure there's a good solution, though. Skipping the\ntest_when_finished block on a \"-i\" run is intentional, to let you\ninspect the broken state.\n\n> Subject: [PATCH] ci: don't accidentally run the test suite as root\n> \n> This script assigns and adds a user named \"ci\" in a subshell so the\n> outer CI_USER is not affected. For some reason, CI_USER is actually\n> empty on Travis linux32 builds. This makes the following \"su\" useless\n> and the test suite is run as root.\n\nAre we sure this was the problem on Travis, and it wasn't just an issue\nwith how I reproduced via docker?\n\n-Peff\n"},{"id":"336833","messageId":"20180118215331.GN2641@hank","threadId":"47421","inReplyTo":"20180107223015.17720-1-t.gummerer@gmail.com","subject":"Re: [PATCH v3 0/3] fixes for split index mode","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2018-01-18T21:53:31Z","receivedAt":"2018-01-18T21:51:17Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Friendly ping on this series now that 2.16 is out :) Is there anything\nin this series (up to 3/3, 4/3 can be dropped now that Duy fixed it in\na nicer way) that still needs updating? It fixes a few bugs in split\nindex mode with submodules/worktrees, so it would be nice to get this\nreviewed/merged.\n\nOn 01/07, Thomas Gummerer wrote:\n> Thanks Brandon and Lars for comments on the previous round.\n> \n> Previous rounds were at <20171210212202.28231-1-t.gummerer@gmail.com>\n> and <20171217225122.28941-1-t.gummerer@gmail.com>.\n> \n> Changes since the previous round:\n> \n> - reworked the patches to no longer try to use struct repository for\n> worktrees, but pass gitdir into read_index_from instead (Thanks\n> Brandon :)). So the fixes that were in 1/3 and 2/3 previously are\n> now in 1/3\n> - 2/3 now fixes t7009 properly. I thought it was fixed before, but it\n> probably just passed the tests because of the GIT_TEST_SPLIT_INDEX\n> \"randomness\".\n> - The travis job is now only running the 64-bit linux build with split\n> index mode to save even more cycles.\n> - As this wasn't picked up anywhere yet, I took the freedom to rebase\n> this onto the current master, which includes sg/travis-fixes, which\n> made it a bit easier for me to test. If this makes the life harder\n> for anyone reviewing this let me know and I can base it on the same\n> commit previous iterations were based on.\n> \n> Thomas Gummerer (3):\n> read-cache: fix reading the shared index for other repos\n> split-index: don't write cache tree with null sha1 entries\n> travis: run tests with GIT_TEST_SPLIT_INDEX\n> \n> cache-tree.c | 2 +-\n> cache.h | 8 +++++---\n> ci/run-linux32-build.sh | 1 +\n> ci/run-tests.sh | 4 ++++\n> read-cache.c | 25 ++++++++++++++-----------\n> repository.c | 2 +-\n> revision.c | 3 ++-\n> split-index.c | 2 ++\n> t/t1700-split-index.sh | 19 +++++++++++++++++++\n> 9 files changed, 49 insertions(+), 17 deletions(-)\n> \n> -- \n> 2.16.0.rc1.238.g530d649a79\n> \n"},{"id":"336842","messageId":"CAM0VKjmAjw0a2JzsY1BaAwZcCp8ge+jQaGN6L_nKTLugHeWyvQ@mail.gmail.com","threadId":"47421","inReplyTo":"20180118213732.GA7403@sigill.intra.peff.net","subject":"Re: [PATCH 3/3] read-cache: don't write index twice if we can't write shared index","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-01-18T22:32:40Z","receivedAt":"2018-01-18T22:32:45Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Jan 18, 2018 at 10:37 PM, Jeff King wrote:\n> On Thu, Jan 18, 2018 at 10:00:14PM +0700, Duy Nguyen wrote:\n>\n>> The test suite was run as root, no wonder why my removing write access\n>> has no effect. I got the test to pass with this, but then it fails\n>> with\n>>\n>> Can't write .prove (Permission denied) at /usr/share/perl/5.22/App/Prove.pm line 542.\n>>\n>> Some more chown'ing or chmod'ing is required....\n\nThis is the fallout of running the tests as root in the past. With your\npatch 'prove' is run as a non-root user, but the prove state is loaded\nfrom Travis CI's cache, where it has been written as root the last time\naround, so now we don't have permissions to (over)write it.\n\nI have patches in the works to address this as well.\n\n>> Subject: [PATCH] ci: don't accidentally run the test suite as root\n>>\n>> This script assigns and adds a user named \"ci\" in a subshell so the\n>> outer CI_USER is not affected. For some reason, CI_USER is actually\n>> empty on Travis linux32 builds. This makes the following \"su\" useless\n>> and the test suite is run as root.\n>\n> Are we sure this was the problem on Travis, and it wasn't just an issue\n> with how I reproduced via docker?\n\nYes, we are.\n\n\nGábor\n"},{"id":"336870","messageId":"CACsJy8AgwKY8xXYp3w39EfoGgoWjN3oNiBju4fV2du4PWaWsHg@mail.gmail.com","threadId":"47421","inReplyTo":"CAM0VKjmAjw0a2JzsY1BaAwZcCp8ge+jQaGN6L_nKTLugHeWyvQ@mail.gmail.com","subject":"Re: [PATCH 3/3] read-cache: don't write index twice if we can't write shared index","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-01-19T00:30:02Z","receivedAt":"2018-01-19T00:30:38Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Jan 19, 2018 at 5:32 AM, SZEDER Gábor wrote:\n> On Thu, Jan 18, 2018 at 10:37 PM, Jeff King wrote:\n>> On Thu, Jan 18, 2018 at 10:00:14PM +0700, Duy Nguyen wrote:\n>>\n>>> The test suite was run as root, no wonder why my removing write access\n>>> has no effect. I got the test to pass with this, but then it fails\n>>> with\n>>>\n>>> Can't write .prove (Permission denied) at /usr/share/perl/5.22/App/Prove.pm line 542.\n>>>\n>>> Some more chown'ing or chmod'ing is required....\n>\n> This is the fallout of running the tests as root in the past. With your\n> patch 'prove' is run as a non-root user, but the prove state is loaded\n> from Travis CI's cache, where it has been written as root the last time\n> around, so now we don't have permissions to (over)write it.\n>\n> I have patches in the works to address this as well.\n\nGreat. I'll leave it to you then. I will update the test with SANITY\n(and a small fix in the test name) and think more about the \"-i\" Jeff\nmentioned.\n-- \nDuy\n"},{"id":"336918","messageId":"xmqqbmhpr92m.fsf@gitster.mtv.corp.google.com","threadId":"47421","inReplyTo":"20180118215331.GN2641@hank","subject":"Re: [PATCH v3 0/3] fixes for split index mode","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-01-19T18:34:57Z","receivedAt":"2018-01-19T18:35:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Gummerer writes:\n\n> Friendly ping on this series now that 2.16 is out :) Is there anything\n> in this series (up to 3/3, 4/3 can be dropped now that Duy fixed it in\n> a nicer way) that still needs updating? It fixes a few bugs in split\n> index mode with submodules/worktrees, so it would be nice to get this\n> reviewed/merged.\n\nI was wondering about the same thing. Especially it wasn't very\nclear to me what Duy's replacement was meant to replace and how well\nit was supposed to work with the rest of your series.\n\nLet's drop 4/3 and queue 1-3/3 for now.\n"},{"id":"336936","messageId":"20180119211149.GO2641@hank","threadId":"47421","inReplyTo":"xmqqbmhpr92m.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 0/3] fixes for split index mode","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2018-01-19T21:11:49Z","receivedAt":"2018-01-19T21:09:33Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 01/19, Junio C Hamano wrote:\n> Thomas Gummerer writes:\n> \n> > Friendly ping on this series now that 2.16 is out :) Is there anything\n> > in this series (up to 3/3, 4/3 can be dropped now that Duy fixed it in\n> > a nicer way) that still needs updating? It fixes a few bugs in split\n> > index mode with submodules/worktrees, so it would be nice to get this\n> > reviewed/merged.\n> \n> I was wondering about the same thing. Especially it wasn't very\n> clear to me what Duy's replacement was meant to replace and how well\n> it was supposed to work with the rest of your series.\n\nSorry about the confusion. 4/3 was replaced by Duy's series, but the\nrest of this series is independent of those patches.\n\n> Let's drop 4/3 and queue 1-3/3 for now.\n\nThanks!\n"},{"id":"336945","messageId":"xmqq1silpl4g.fsf@gitster.mtv.corp.google.com","threadId":"47421","inReplyTo":"20180107223015.17720-2-t.gummerer@gmail.com","subject":"Re: [PATCH v3 1/3] read-cache: fix reading the shared index for other repos","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-01-19T21:57:35Z","receivedAt":"2018-01-19T21:57:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Gummerer writes:\n\n> read_cache_from() defaults to using the gitdir of the_repository. As it\n> is mostly a convenience macro, having to pass get_git_dir() for every\n> call seems overkill, and if necessary users can have more control by\n> using read_index_from().\n\nThis was a bit painful change, given that some changes in flight do\nadd new callsites to read_index_from() and they got the function\nchanged under their feet.\n\nPlease double check if I made the right git-dir to be passed to them\nwhen I push out 'pu' in a few hours.\n\nThanks.\n"},{"id":"336978","messageId":"20180120115819.GR2641@hank","threadId":"47421","inReplyTo":"xmqq1silpl4g.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 1/3] read-cache: fix reading the shared index for other repos","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2018-01-20T11:58:19Z","receivedAt":"2018-01-20T11:57:18Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 01/19, Junio C Hamano wrote:\n> Thomas Gummerer writes:\n> \n> > read_cache_from() defaults to using the gitdir of the_repository. As it\n> > is mostly a convenience macro, having to pass get_git_dir() for every\n> > call seems overkill, and if necessary users can have more control by\n> > using read_index_from().\n> \n> This was a bit painful change, given that some changes in flight do\n> add new callsites to read_index_from() and they got the function\n> changed under their feet.\n\nSorry about that. Is there any way to make such a change less painful\nin the future?\n\n> Please double check if I made the right git-dir to be passed to them\n> when I push out 'pu' in a few hours.\n\nI think one conversion was not quite correct, even though all tests\nstill pass with GIT_TEST_SPLIT_INDEX set.\n\nThe following diff fixes that conversation and has a test showing the\nbreakage:\n\ndiff --git a/builtin/worktree.c b/builtin/worktree.c\nindex 6a49f9e628..4d86a3574f 100644\n--- a/builtin/worktree.c\n+++ b/builtin/worktree.c\n@@ -612,7 +612,8 @@ static void validate_no_submodules(const struct worktree *wt)\n struct index_state istate = {0};\n int i, found_submodules = 0;\n \n- if (read_index_from(&istate, worktree_git_path(wt, \"index\"), get_git_dir()) > 0) {\n+ if (read_index_from(&istate, worktree_git_path(wt, \"index\"),\n+ get_worktree_git_dir(wt)) > 0) {\n for (i = 0; i < istate.cache_nr; i++) {\n struct cache_entry *ce = istate.cache[i];\n \ndiff --git a/t/t2028-worktree-move.sh b/t/t2028-worktree-move.sh\nindex b3105eaaed..8faf61bbf5 100755\n--- a/t/t2028-worktree-move.sh\n+++ b/t/t2028-worktree-move.sh\n@@ -90,6 +90,16 @@ test_expect_success 'move main worktree' '\n test_must_fail git worktree move . def\n '\n \n+test_expect_success 'move worktree with split index' '\n+ git worktree add test &&\n+ (\n+ cd test &&\n+ test_commit file &&\n+ git update-index --split-index\n+ ) &&\n+ git worktree move test test-destination\n+'\n+\n test_expect_success 'remove main worktree' '\n test_must_fail git worktree remove .\n '\n\nWith this applied what you have in pu looks good to me. Does the\nabove help, or should I send this in another for for you to apply?\n\n> Thanks.\n"},{"id":"337050","messageId":"xmqqpo62mncc.fsf@gitster.mtv.corp.google.com","threadId":"47421","inReplyTo":"20180120115819.GR2641@hank","subject":"Re: [PATCH v3 1/3] read-cache: fix reading the shared index for other repos","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-01-22T06:14:43Z","receivedAt":"2018-01-22T06:14:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Gummerer writes:\n\n> On 01/19, Junio C Hamano wrote:\n>> Thomas Gummerer writes:\n>> \n>> > read_cache_from() defaults to using the gitdir of the_repository. As it\n>> > is mostly a convenience macro, having to pass get_git_dir() for every\n>> > call seems overkill, and if necessary users can have more control by\n>> > using read_index_from().\n>> \n>> This was a bit painful change, given that some changes in flight do\n>> add new callsites to read_index_from() and they got the function\n>> changed under their feet.\n>\n> Sorry about that. Is there any way to make such a change less painful\n> in the future?\n\nOne way is to do for read_index_from() what you did for the existing\nusers of read_cache_from(). Introduce a _new_ helper that will not\nbe known for any existing topics in flight, and use that to make the\nexisting API a thin wrapper around it.\n\nI _think_ I got it right with evil merge, so unless this fix needs\nto be delayed for extended period of time for whatever reason while\nany more new callers of the function appears (which is unlikely), we\nshould be OK ;-)\n\nThanks.\n"},{"id":"337085","messageId":"20180122133220.18587-1-szeder.dev@gmail.com","threadId":"47421","inReplyTo":"CAM0VKjmAjw0a2JzsY1BaAwZcCp8ge+jQaGN6L_nKTLugHeWyvQ@mail.gmail.com","subject":"[PATCH 0/5] Travis CI: don't run the test suite as root in the 32 bit Linux build","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-01-22T13:32:15Z","receivedAt":"2018-01-22T13:32:47Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"\nThe important one is patch 4, which fixes the issue of running the test\nsuite as root, with a bit of sugar on top to make sure that a future\n\"non-root\" build job can cope with a root-written cache directory from\nthe past.\n\nThe rest is a collection of small cleanups and improvements to make the\ndebugging this Docked-based build more convenient.\n\nSZEDER Gábor (5):\n travis-ci: use 'set -x' for the commands under 'su' in the 32 bit\n Linux build\n travis-ci: use 'set -e' in the 32 bit Linux build job\n travis-ci: don't repeat the path of the cache directory\n travis-ci: don't run the test suite as root in the 32 bit Linux build\n travis-ci: don't fail if user already exists on 32 bit Linux build job\n\n ci/lib-travisci.sh | 7 +++---\n ci/run-build-and-tests.sh | 2 +-\n ci/run-linux32-build.sh | 55 +++++++++++++++++++++++++++++++++++------------\n ci/run-linux32-docker.sh | 7 ++++--\n 4 files changed, 51 insertions(+), 20 deletions(-)\n\n-- \n2.16.1.80.gc0eec9753d\n\n"},{"id":"337086","messageId":"20180122133220.18587-3-szeder.dev@gmail.com","threadId":"47421","inReplyTo":"20180122133220.18587-1-szeder.dev@gmail.com","subject":"[PATCH 2/5] travis-ci: use 'set -e' in the 32 bit Linux build job","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-01-22T13:32:17Z","receivedAt":"2018-01-22T13:32:50Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"All 'ci/*' scripts use 'set -e' to break the build job if a command\nfails, except 'ci/run-linux32-build.sh' which relies on the && chain\nto do the same. This inconsistency among the 'ci/*' scripts is asking\nfor trouble: I forgot about the && chain more than once while working\non this patch series.\n\nEnable 'set -e' for the whole script and for the commands executed\nunder 'su' as well.\n\nWhile touching every line in the 'su' command block anyway, change\ntheir indentation to use a tab instead of spaces.\n\nSigned-off-by: SZEDER Gábor \n---\n ci/run-linux32-build.sh | 20 ++++++++++----------\n 1 file changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh\nindex 5a36a8d7c0..248183982b 100755\n--- a/ci/run-linux32-build.sh\n+++ b/ci/run-linux32-build.sh\n@@ -6,29 +6,29 @@\n # run-linux32-build.sh [host-user-id]\n #\n \n-set -x\n+set -ex\n \n # Update packages to the latest available versions\n linux32 --32bit i386 sh -c '\n apt update >/dev/null &&\n apt install -y build-essential libcurl4-openssl-dev libssl-dev \\\n \tlibexpat-dev gettext python >/dev/null\n-' &&\n+'\n \n # If this script runs inside a docker container, then all commands are\n # usually executed as root. Consequently, the host user might not be\n # able to access the test output files.\n # If a host user id is given, then create a user \"ci\" with the host user\n # id to make everything accessible to the host user.\n-HOST_UID=$1 &&\n-CI_USER=$USER &&\n-test -z $HOST_UID || (CI_USER=\"ci\" && useradd -u $HOST_UID $CI_USER) &&\n+HOST_UID=$1\n+CI_USER=$USER\n+test -z $HOST_UID || (CI_USER=\"ci\" && useradd -u $HOST_UID $CI_USER)\n \n # Build and test\n linux32 --32bit i386 su -m -l $CI_USER -c '\n- set -x &&\n- cd /usr/src/git &&\n- ln -s /tmp/travis-cache/.prove t/.prove &&\n- make --jobs=2 &&\n- make --quiet test\n+\tset -ex\n+\tcd /usr/src/git\n+\tln -s /tmp/travis-cache/.prove t/.prove\n+\tmake --jobs=2\n+\tmake --quiet test\n '\n-- \n2.16.1.80.gc0eec9753d\n\n"},{"id":"337087","messageId":"20180122133220.18587-6-szeder.dev@gmail.com","threadId":"47421","inReplyTo":"20180122133220.18587-1-szeder.dev@gmail.com","subject":"[PATCH 5/5] travis-ci: don't fail if user already exists on 32 bit Linux build job","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-01-22T13:32:20Z","receivedAt":"2018-01-22T13:32:52Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"The 32 bit Linux build job runs in a Docker container, which lends\nitself to running and debugging locally, too. Especially during\ndebugging one usually doesn't want to start with a fresh container\nevery time, to save time spent on installing a bunch of dependencies.\nHowever, that doesn't work quite smootly, because the script running\nin the container always creates a new user, which then must be removed\nevery time before subsequent executions, or the build script fails.\n\nMake this process more convenient and don't try to create that user if\nit already exists and has the right user ID in the container, so\ndevelopers don't have to bother with running a 'userdel' each time\nbefore they run the build script.\n\nThe build job on Travis CI always starts with a fresh Docker\ncontainer, so this change doesn't make a difference there.\n\nSigned-off-by: SZEDER Gábor \n---\n ci/run-linux32-build.sh | 8 +++++++-\n 1 file changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh\nindex e37e1d2d5f..13047adde3 100755\n--- a/ci/run-linux32-build.sh\n+++ b/ci/run-linux32-build.sh\n@@ -33,7 +33,13 @@ then\n \tCI_USER=root\n else\n \tCI_USER=ci\n-\tuseradd -u $HOST_UID $CI_USER\n+\tif test \"$(id -u $CI_USER)\" = $HOST_UID\n+\tthen\n+\t\t: # user already exists with the right ID\n+\telse\n+\t\tuseradd -u $HOST_UID $CI_USER\n+\tfi\n+\n \t# Due to a bug the test suite was run as root in the past, so\n \t# a prove state file created back then is only accessible by\n \t# root. Now that bug is fixed, the test suite is run as a\n-- \n2.16.1.80.gc0eec9753d\n\n"},{"id":"337088","messageId":"20180122133220.18587-2-szeder.dev@gmail.com","threadId":"47421","inReplyTo":"20180122133220.18587-1-szeder.dev@gmail.com","subject":"[PATCH 1/5] travis-ci: use 'set -x' for the commands under 'su' in the 32 bit Linux build","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-01-22T13:32:16Z","receivedAt":"2018-01-22T13:32:56Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Signed-off-by: SZEDER Gábor \n---\n ci/run-linux32-build.sh | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh\nindex c19c50c1c9..5a36a8d7c0 100755\n--- a/ci/run-linux32-build.sh\n+++ b/ci/run-linux32-build.sh\n@@ -26,6 +26,7 @@ test -z $HOST_UID || (CI_USER=\"ci\" && useradd -u $HOST_UID $CI_USER) &&\n \n # Build and test\n linux32 --32bit i386 su -m -l $CI_USER -c '\n+ set -x &&\n cd /usr/src/git &&\n ln -s /tmp/travis-cache/.prove t/.prove &&\n make --jobs=2 &&\n-- \n2.16.1.80.gc0eec9753d\n\n"},{"id":"337089","messageId":"20180122133220.18587-4-szeder.dev@gmail.com","threadId":"47421","inReplyTo":"20180122133220.18587-1-szeder.dev@gmail.com","subject":"[PATCH 3/5] travis-ci: don't repeat the path of the cache directory","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-01-22T13:32:18Z","receivedAt":"2018-01-22T13:32:57Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Some of our 'ci/*' scripts repeat the name or full path of the Travis\nCI cache directory, and the following patches will add new places\nusing that path.\n\nUse a variable to refer to the path of the cache directory instead, so\nit's hard-coded only in a single place.\n\nPay extra attention to the 32 bit Linux build: it runs in a Docker\ncontainer, so pass the path of the cache directory from the host to\nthe container in an environment variable. Furthermore, use the\nvariable in the container only if it's set, to prevent errors when\nsomeone is running or debugging the Docker build locally, because in\nthat case the variable won't be set as there won't be any Travis CI\ncache.\n\nSigned-off-by: SZEDER Gábor \n---\n ci/lib-travisci.sh | 7 ++++---\n ci/run-build-and-tests.sh | 2 +-\n ci/run-linux32-build.sh | 6 +++---\n ci/run-linux32-docker.sh | 5 ++++-\n 4 files changed, 12 insertions(+), 8 deletions(-)\n\ndiff --git a/ci/lib-travisci.sh b/ci/lib-travisci.sh\nindex 07f27c7270..1efee55ef7 100755\n--- a/ci/lib-travisci.sh\n+++ b/ci/lib-travisci.sh\n@@ -21,8 +21,6 @@ skip_branch_tip_with_tag () {\n \tfi\n }\n \n-good_trees_file=\"$HOME/travis-cache/good-trees\"\n-\n # Save some info about the current commit's tree, so we can skip the build\n # job if we encounter the same tree again and can provide a useful info\n # message.\n@@ -83,7 +81,10 @@ check_unignored_build_artifacts ()\n # and installing dependencies.\n set -ex\n \n-mkdir -p \"$HOME/travis-cache\"\n+cache_dir=\"$HOME/travis-cache\"\n+good_trees_file=\"$cache_dir/good-trees\"\n+\n+mkdir -p \"$cache_dir\"\n \n skip_branch_tip_with_tag\n skip_good_tree\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex d3a094603f..30790e258d 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -5,7 +5,7 @@\n \n . ${0%/*}/lib-travisci.sh\n \n-ln -s $HOME/travis-cache/.prove t/.prove\n+ln -s \"$cache_dir/.prove\" t/.prove\n \n make --jobs=2\n make --quiet test\ndiff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh\nindex 248183982b..c9476d6598 100755\n--- a/ci/run-linux32-build.sh\n+++ b/ci/run-linux32-build.sh\n@@ -25,10 +25,10 @@ CI_USER=$USER\n test -z $HOST_UID || (CI_USER=\"ci\" && useradd -u $HOST_UID $CI_USER)\n \n # Build and test\n-linux32 --32bit i386 su -m -l $CI_USER -c '\n+linux32 --32bit i386 su -m -l $CI_USER -c \"\n \tset -ex\n \tcd /usr/src/git\n-\tln -s /tmp/travis-cache/.prove t/.prove\n+\ttest -n '$cache_dir' && ln -s '$cache_dir/.prove' t/.prove\n \tmake --jobs=2\n \tmake --quiet test\n-'\n+\"\ndiff --git a/ci/run-linux32-docker.sh b/ci/run-linux32-docker.sh\nindex 4f191c5bb1..15288ea2cf 100755\n--- a/ci/run-linux32-docker.sh\n+++ b/ci/run-linux32-docker.sh\n@@ -11,6 +11,8 @@ docker pull daald/ubuntu32:xenial\n # $ docker run -itv \"${PWD}:/usr/src/git\" --entrypoint /bin/bash daald/ubuntu32:xenial\n # root@container:/# /usr/src/git/ci/run-linux32-build.sh\n \n+container_cache_dir=/tmp/travis-cache\n+\n docker run \\\n \t--interactive \\\n \t--env DEVELOPER \\\n@@ -18,8 +20,9 @@ docker run \\\n \t--env GIT_PROVE_OPTS \\\n \t--env GIT_TEST_OPTS \\\n \t--env GIT_TEST_CLONE_2GB \\\n+\t--env cache_dir=\"$container_cache_dir\" \\\n \t--volume \"${PWD}:/usr/src/git\" \\\n-\t--volume \"${HOME}/travis-cache:/tmp/travis-cache\" \\\n+\t--volume \"$cache_dir:$container_cache_dir\" \\\n \tdaald/ubuntu32:xenial \\\n \t/usr/src/git/ci/run-linux32-build.sh $(id -u $USER)\n \n-- \n2.16.1.80.gc0eec9753d\n\n"},{"id":"337090","messageId":"20180122133220.18587-5-szeder.dev@gmail.com","threadId":"47421","inReplyTo":"20180122133220.18587-1-szeder.dev@gmail.com","subject":"[PATCH 4/5] travis-ci: don't run the test suite as root in the 32 bit Linux build","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-01-22T13:32:19Z","receivedAt":"2018-01-22T13:33:00Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"Travis CI runs the 32 bit Linux build job in a Docker container, where\nall commands are executed as root by default. Therefore, ever since\nwe added this build job in 88dedd5e7 (Travis: also test on 32-bit\nLinux, 2017-03-05), we have a bit of code to create a user in the\ncontainer matching the ID of the host user and then to run the test\nsuite as this user. Matching the host user ID is important, because\notherwise the host user would have no access to any files written by\nprocesses running in the container, notably the logs of failed tests\ncouldn't be included in the build job's trace log.\n\nAlas, this piece of code never worked, because it sets the variable\nholding the user name ($CI_USER) in a subshell, meaning it doesn't\nhave any effect by the time we get to the point to actually use the\nvariable to switch users with 'su'. So all this time we were running\nthe test suite as root.\n\nReorganize that piece of code in 'ci/run-linux32-build.sh' a bit to\navoid that problematic subshell and to ensure that we switch to the\nright user. Furthermore, make the script's optional host user ID\noption mandatory, so running the build accidentally as root will\nbecome harder when debugging locally. If someone really wants to run\nthe test suite as root, whatever the reasons might be, it'll still be\npossible to do so by explicitly passing '0' as host user ID.\n\nFinally, one last catch: since commit 7e72cfcee (travis-ci: save prove\nstate for the 32 bit Linux build, 2017-12-27) the 'prove' test harness\nhas been writing its state to the Travis CI cache directory from\nwithin the Docker container while running as root. After this patch\n'prove' will run as a regular user, so in future build jobs it won't\nbe able overwrite a previously written, still root-owned state file,\nresulting in build job failures. To resolve this we should manually\ndelete caches containing such root-owned files, but that would be a\nhassle. Instead, work this around by changing the owner of the whole\ncontents of the cache directory to the host user ID.\n\nSigned-off-by: SZEDER Gábor \n---\n ci/run-linux32-build.sh | 30 +++++++++++++++++++++++++-----\n ci/run-linux32-docker.sh | 2 +-\n 2 files changed, 26 insertions(+), 6 deletions(-)\n\ndiff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh\nindex c9476d6598..e37e1d2d5f 100755\n--- a/ci/run-linux32-build.sh\n+++ b/ci/run-linux32-build.sh\n@@ -3,11 +3,17 @@\n # Build and test Git in a 32-bit environment\n #\n # Usage:\n-# run-linux32-build.sh [host-user-id]\n+# run-linux32-build.sh \n #\n \n set -ex\n \n+if test $# -ne 1 || test -z \"$1\"\n+then\n+\techo >&2 \"usage: run-linux32-build.sh \"\n+\texit 1\n+fi\n+\n # Update packages to the latest available versions\n linux32 --32bit i386 sh -c '\n apt update >/dev/null &&\n@@ -18,11 +24,25 @@ linux32 --32bit i386 sh -c '\n # If this script runs inside a docker container, then all commands are\n # usually executed as root. Consequently, the host user might not be\n # able to access the test output files.\n-# If a host user id is given, then create a user \"ci\" with the host user\n-# id to make everything accessible to the host user.\n+# If a non 0 host user id is given, then create a user \"ci\" with that\n+# user id to make everything accessible to the host user.\n HOST_UID=$1\n-CI_USER=$USER\n-test -z $HOST_UID || (CI_USER=\"ci\" && useradd -u $HOST_UID $CI_USER)\n+if test $HOST_UID -eq 0\n+then\n+\t# Just in case someone does want to run the test suite as root.\n+\tCI_USER=root\n+else\n+\tCI_USER=ci\n+\tuseradd -u $HOST_UID $CI_USER\n+\t# Due to a bug the test suite was run as root in the past, so\n+\t# a prove state file created back then is only accessible by\n+\t# root. Now that bug is fixed, the test suite is run as a\n+\t# regular user, but the prove state file coming from Travis\n+\t# CI's cache might still be owned by root.\n+\t# Make sure that this user has rights to any cached files,\n+\t# including an existing prove state file.\n+\ttest -n \"$cache_dir\" && chown -R $HOST_UID:$HOST_UID \"$cache_dir\"\n+fi\n \n # Build and test\n linux32 --32bit i386 su -m -l $CI_USER -c \"\ndiff --git a/ci/run-linux32-docker.sh b/ci/run-linux32-docker.sh\nindex 15288ea2cf..21637903ce 100755\n--- a/ci/run-linux32-docker.sh\n+++ b/ci/run-linux32-docker.sh\n@@ -9,7 +9,7 @@ docker pull daald/ubuntu32:xenial\n \n # Use the following command to debug the docker build locally:\n # $ docker run -itv \"${PWD}:/usr/src/git\" --entrypoint /bin/bash daald/ubuntu32:xenial\n-# root@container:/# /usr/src/git/ci/run-linux32-build.sh\n+# root@container:/# /usr/src/git/ci/run-linux32-build.sh \n \n container_cache_dir=/tmp/travis-cache\n \n-- \n2.16.1.80.gc0eec9753d\n\n"},{"id":"337098","messageId":"20180122182717.21539-1-szeder.dev@gmail.com","threadId":"47421","inReplyTo":"CACsJy8BBXQ9KErfiuf2ty_4szE2fiHLDiKvMig1LbSefzf-o7w@mail.gmail.com","subject":"Re: [PATCH 3/3] read-cache: don't write index twice if we can't write shared index","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-01-22T18:27:17Z","receivedAt":"2018-01-22T18:27:37Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"\nOn Thu, Jan 18, 2018 at 1:47 PM, Duy Nguyen wrote:\n> On Thu, Jan 18, 2018 at 6:36 PM, SZEDER Gábor wrote:\n>> This series, queued as 'nd/shared-index-fix', makes the 32 bit Linux\n>> build job fail on Travis CI. Unfortunately, all it can tell us about\n>> the failure is this:\n>>\n>> Test Summary Report\n>> -------------------\n>> t1700-split-index.sh (Wstat: 256 Tests: 23\n>> Failed: 1)\n>> Failed test: 23\n>> Non-zero exit status: 1\n>> Files=809, Tests=18491, 401 wallclock secs ( 7.22 usr 1.60 sys + 263.16\n>> cusr 49.58 csys = 321.56 CPU)\n>> Result: FAIL\n>>\n>> because it can't access the test's verbose log due to lack of\n>> permissions.\n>>\n>>\n>> https://travis-ci.org/git/git/jobs/329681826#L2074\n>\n> I may need help getting that log (or even better the trash directory\n> of t1700). \n\nWell, shower thoughts gave me an idea, see the included PoC patch below.\nI can't really decide whether it's too clever or too ugly :)\n\nIt produces output like this (a previous WIP version without\n'--immediate'):\n\n https://travis-ci.org/szeder/git/jobs/331601009#L2486\n\n\n -- >8 --\n\nSubject: [PATCH] travis-ci: include the trash directories of failed tests in\n the trace log\n\nThe trash directory of a failed test might contain valuable\ninformation about the cause of the failure, but we have no access to\nthe trash directories of Travis CI build jobs. The only feedback we\nget from there is the trace log, so...\n\nModify 'ci/print-test-failures.sh' to create a tar.gz archive of the\ntest directory of each failed test and encode it with base64, so the\nresult is a block of ASCII text that can be safely included in the\ntrace log, along with a hint about how to restore it. Furthermore,\nrun tests with '--immediate' to faithfully preserve the failed state.\n\nA few of our tests create a sizeable trash directory, so limit the\nsize of each included base64-encoded block, let's say, to 1MB.\n\nNote:\n\n - The logs of Linux build jobs coming from Travis CI have mostly\n CRLF line endings, which makes 'base64 -d' from 'coreutils'\n complain about \"invalid input\"; it has to be converted to LF\n first. 'openssl base64 -d' can grok it just fine, even without\n conversion.\n\n - The logs of OSX build jobs have CRCRLF line endings. However, the\n 'base64' util of OSX doesn't wrap its output at 76 columns, i.e.\n prints one single albeit very long line. This means that while\n 'base64' from 'coreutils' still complains, by the time it gets to\n the invalid input it already decoded everything and produced a\n valid .tar.gz. OTOH, 'openssl base64 -d' doesn't grok it, but\n exits without any error message or even an error code, even after\n converting to CRLF or LF line endings.\n\n Go figure.\n\nSigned-off-by: SZEDER Gábor \n---\n ci/lib-travisci.sh | 2 +-\n ci/print-test-failures.sh | 20 ++++++++++++++++++++\n 2 files changed, 21 insertions(+), 1 deletion(-)\n\ndiff --git a/ci/lib-travisci.sh b/ci/lib-travisci.sh\nindex 1efee55ef7..981c6e9504 100755\n--- a/ci/lib-travisci.sh\n+++ b/ci/lib-travisci.sh\n@@ -97,7 +97,7 @@ fi\n export DEVELOPER=1\n export DEFAULT_TEST_TARGET=prove\n export GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n-export GIT_TEST_OPTS=\"--verbose-log\"\n+export GIT_TEST_OPTS=\"--verbose-log --immediate\"\n export GIT_TEST_CLONE_2GB=YesPlease\n \n case \"$jobname\" in\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex 4f261ddc01..cc6a1247a4 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -23,5 +23,25 @@ do\n \t\techo \"$(tput setaf 1)${TEST_OUT}...$(tput sgr0)\"\n \t\techo \"------------------------------------------------------------------------\"\n \t\tcat \"${TEST_OUT}\"\n+\n+\t\tTEST_NAME=\"${TEST_EXIT%.exit}\"\n+\t\tTEST_NAME=\"${TEST_NAME##*/}\"\n+\t\tTRASH_DIR=\"t/trash directory.$TEST_NAME\"\n+\t\tTRASH_TGZ_B64=\"$TEST_NAME.trash.base64\"\n+\t\tif [ -d \"$TRASH_DIR\" ]\n+\t\tthen\n+\t\t\ttar czp \"$TRASH_DIR\" |base64 >\"$TRASH_TGZ_B64\"\n+\n+\t\t\tif [ 1048576 -lt $(wc -c <\"$TRASH_TGZ_B64\") ]\n+\t\t\tthen\n+\t\t\t\t# larger than 1MB\n+\t\t\t\techo \"$(tput setaf 1)Trash directory of '$TEST_NAME' is too big to be included in the trace log$(tput sgr0)\"\n+\t\t\telse\n+\t\t\t\techo \"$(tput setaf 1)Trash directory of '$TEST_NAME':$(tput sgr0)\"\n+\t\t\t\techo \"(Extract it to a file from the raw log, make sure it has Unix line endings, then run 'base64 -d |tar vxzp' to restore it)\"\n+\t\t\t\tcat \"$TRASH_TGZ_B64\"\n+\t\t\t\techo \"$(tput setaf 1)End of trash directory of '$TEST_NAME'$(tput sgr0)\"\n+\t\t\tfi\n+\t\tfi\n \tfi\n done\n-- \n2.16.1.80.gc0eec9753d\n\n"},{"id":"337101","messageId":"CAPig+cQnVfFcN_kH-pQmjJWa_3SeEb4_vdL7-nDCO-tV-U7Pvg@mail.gmail.com","threadId":"47421","inReplyTo":"20180122182717.21539-1-szeder.dev@gmail.com","subject":"Re: [PATCH 3/3] read-cache: don't write index twice if we can't write shared index","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-01-22T19:46:37Z","receivedAt":"2018-01-22T19:46:42Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Jan 22, 2018 at 1:27 PM, SZEDER Gábor wrote:\n> Subject: [PATCH] travis-ci: include the trash directories of failed tests in\n> the trace log\n>\n> The trash directory of a failed test might contain valuable\n> information about the cause of the failure, but we have no access to\n> the trash directories of Travis CI build jobs. The only feedback we\n> get from there is the trace log, so...\n>\n> Modify 'ci/print-test-failures.sh' to create a tar.gz archive of the\n> test directory of each failed test and encode it with base64, so the\n> result is a block of ASCII text that can be safely included in the\n> trace log, along with a hint about how to restore it. Furthermore,\n> run tests with '--immediate' to faithfully preserve the failed state.\n>\n> A few of our tests create a sizeable trash directory, so limit the\n> size of each included base64-encoded block, let's say, to 1MB.\n>\n> Note:\n>\n> - The logs of Linux build jobs coming from Travis CI have mostly\n> CRLF line endings, which makes 'base64 -d' from 'coreutils'\n> complain about \"invalid input\"; it has to be converted to LF\n> first. 'openssl base64 -d' can grok it just fine, even without\n> conversion.\n>\n> - The logs of OSX build jobs have CRCRLF line endings. However, the\n> 'base64' util of OSX doesn't wrap its output at 76 columns, i.e.\n> prints one single albeit very long line. This means that while\n\nPerhaps you could pipe the 'base64' output through 'fold' or 'fmt'?\n\n> 'base64' from 'coreutils' still complains, by the time it gets to\n> the invalid input it already decoded everything and produced a\n> valid .tar.gz. OTOH, 'openssl base64 -d' doesn't grok it, but\n> exits without any error message or even an error code, even after\n> converting to CRLF or LF line endings.\n>\n> Go figure.\n>\n> Signed-off-by: SZEDER Gábor \n"},{"id":"337107","messageId":"CAM0VKj=EK7ZQkxvhvzyDgKJXxzd19Mtj1vCRULqthVmPj4W2Tg@mail.gmail.com","threadId":"47421","inReplyTo":"CAPig+cQnVfFcN_kH-pQmjJWa_3SeEb4_vdL7-nDCO-tV-U7Pvg@mail.gmail.com","subject":"Re: [PATCH 3/3] read-cache: don't write index twice if we can't write shared index","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-01-22T22:10:30Z","receivedAt":"2018-01-22T22:10:36Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Jan 22, 2018 at 8:46 PM, Eric Sunshine wrote:\n> On Mon, Jan 22, 2018 at 1:27 PM, SZEDER Gábor wrote:\n\n>> - The logs of OSX build jobs have CRCRLF line endings. However, the\n>> 'base64' util of OSX doesn't wrap its output at 76 columns, i.e.\n>> prints one single albeit very long line. This means that while\n>\n> Perhaps you could pipe the 'base64' output through 'fold' or 'fmt'?\n\nNo need to, according to its manpage[1], OSX's base64 has an option to\nwrap lines after given number of characters. Of course it uses a\ndifferent letter for the option than the coreutils base64... :)\n\n(While long lines are ugly, in this particular case it comes handy: when\nusing vim to extract the base64-encoded section from the log to a\nseparate file, it's less keystrokes to yank a single line than to search\nfor the end of the encoded section.)\n\n\n[1] - https://developer.apple.com/legacy/library/documentation/Darwin/Reference/ManPages/man1/base64.1.html\n"},{"id":"337158","messageId":"20180123162633.GE13068@sigill.intra.peff.net","threadId":"47421","inReplyTo":"20180122133220.18587-3-szeder.dev@gmail.com","subject":"Re: [PATCH 2/5] travis-ci: use 'set -e' in the 32 bit Linux build job","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-01-23T16:26:33Z","receivedAt":"2018-01-23T16:26:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 22, 2018 at 02:32:17PM +0100, SZEDER Gábor wrote:\n\n> All 'ci/*' scripts use 'set -e' to break the build job if a command\n> fails, except 'ci/run-linux32-build.sh' which relies on the && chain\n> to do the same. This inconsistency among the 'ci/*' scripts is asking\n> for trouble: I forgot about the && chain more than once while working\n> on this patch series.\n\nI think this actually fixes a bug:\n\n> # Update packages to the latest available versions\n> linux32 --32bit i386 sh -c '\n> apt update >/dev/null &&\n> apt install -y build-essential libcurl4-openssl-dev libssl-dev \\\n> \tlibexpat-dev gettext python >/dev/null\n> -' &&\n> +'\n\nIf this step failed, then...\n\n> # If this script runs inside a docker container, then all commands are\n> # usually executed as root. Consequently, the host user might not be\n> # able to access the test output files.\n> # If a host user id is given, then create a user \"ci\" with the host user\n> # id to make everything accessible to the host user.\n> -HOST_UID=$1 &&\n> -CI_USER=$USER &&\n> -test -z $HOST_UID || (CI_USER=\"ci\" && useradd -u $HOST_UID $CI_USER) &&\n\nWe'd have triggered the right-hand side of this \"||\", and run the rest\nof the script. The whole \"||\" block should have been inside {}.\n\nBut after your patch, it should be fine with \"set -e\". Although...\n\n> +HOST_UID=$1\n> +CI_USER=$USER\n> +test -z $HOST_UID || (CI_USER=\"ci\" && useradd -u $HOST_UID $CI_USER)\n\nIf this \"useradd\" step fails, we wouldn't abort the script, because it's\npart of a conditional. You'd need a manual \"|| exit 1\" at the end of\nthis line. Or to use a real \"if\" block.\n\nReading this line, I'm also slightly confused by setting CI_USER in the\nsubshell, and then only using it once. Should it be respecting the outer\nCI_USER setting? If not, should it just hard-code \"ci\" on the useradd\ncommand-line? But that has nothing to do with your patch here.\n\n-Peff\n"},{"id":"337159","messageId":"20180123163038.GF13068@sigill.intra.peff.net","threadId":"47421","inReplyTo":"20180122133220.18587-4-szeder.dev@gmail.com","subject":"Re: [PATCH 3/5] travis-ci: don't repeat the path of the cache directory","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-01-23T16:30:38Z","receivedAt":"2018-01-23T16:30:44Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 22, 2018 at 02:32:18PM +0100, SZEDER Gábor wrote:\n\n> Some of our 'ci/*' scripts repeat the name or full path of the Travis\n> CI cache directory, and the following patches will add new places\n> using that path.\n> \n> Use a variable to refer to the path of the cache directory instead, so\n> it's hard-coded only in a single place.\n> \n> Pay extra attention to the 32 bit Linux build: it runs in a Docker\n> container, so pass the path of the cache directory from the host to\n> the container in an environment variable. Furthermore, use the\n> variable in the container only if it's set, to prevent errors when\n> someone is running or debugging the Docker build locally, because in\n> that case the variable won't be set as there won't be any Travis CI\n> cache.\n\nMakes sense that we need to pass it down, but...\n\n> diff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh\n> index 248183982b..c9476d6598 100755\n> --- a/ci/run-linux32-build.sh\n> +++ b/ci/run-linux32-build.sh\n> @@ -25,10 +25,10 @@ CI_USER=$USER\n> test -z $HOST_UID || (CI_USER=\"ci\" && useradd -u $HOST_UID $CI_USER)\n> \n> # Build and test\n> -linux32 --32bit i386 su -m -l $CI_USER -c '\n> +linux32 --32bit i386 su -m -l $CI_USER -c \"\n> \tset -ex\n> \tcd /usr/src/git\n> -\tln -s /tmp/travis-cache/.prove t/.prove\n> +\ttest -n '$cache_dir' && ln -s '$cache_dir/.prove' t/.prove\n> \tmake --jobs=2\n> \tmake --quiet test\n> -'\n> +\"\n\nThis interpolates $cache_dir while generating the snippet. You've stuck\nit in single-quotes, which prevents most quoting problems, but obviously\nit's an issue if the variable itself contains a single-quote. Is there\na reason not to just export $cache_dir in the environment and access it\ndirectly from the snippet?\n\nProbably not a _big_ deal, since we control the contents of the\nvariable, but it just seems like a fragile practice to avoid.\n\n-Peff\n"},{"id":"337161","messageId":"20180123163205.GA14626@sigill.intra.peff.net","threadId":"47421","inReplyTo":"20180123162633.GE13068@sigill.intra.peff.net","subject":"Re: [PATCH 2/5] travis-ci: use 'set -e' in the 32 bit Linux build job","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-01-23T16:32:05Z","receivedAt":"2018-01-23T16:32:11Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 23, 2018 at 11:26:33AM -0500, Jeff King wrote:\n\n> > +HOST_UID=$1\n> > +CI_USER=$USER\n> > +test -z $HOST_UID || (CI_USER=\"ci\" && useradd -u $HOST_UID $CI_USER)\n> \n> If this \"useradd\" step fails, we wouldn't abort the script, because it's\n> part of a conditional. You'd need a manual \"|| exit 1\" at the end of\n> this line. Or to use a real \"if\" block.\n> \n> Reading this line, I'm also slightly confused by setting CI_USER in the\n> subshell, and then only using it once. Should it be respecting the outer\n> CI_USER setting? If not, should it just hard-code \"ci\" on the useradd\n> command-line? But that has nothing to do with your patch here.\n\nOK, nevermind on all this. I just read patch 4. :)\n\n-Peff\n"},{"id":"337162","messageId":"20180123164340.GG13068@sigill.intra.peff.net","threadId":"47421","inReplyTo":"20180122133220.18587-5-szeder.dev@gmail.com","subject":"Re: [PATCH 4/5] travis-ci: don't run the test suite as root in the 32 bit Linux build","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-01-23T16:43:40Z","receivedAt":"2018-01-23T16:43:47Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 22, 2018 at 02:32:19PM +0100, SZEDER Gábor wrote:\n\n> Travis CI runs the 32 bit Linux build job in a Docker container, where\n> all commands are executed as root by default. Therefore, ever since\n> we added this build job in 88dedd5e7 (Travis: also test on 32-bit\n> Linux, 2017-03-05), we have a bit of code to create a user in the\n> container matching the ID of the host user and then to run the test\n> suite as this user. Matching the host user ID is important, because\n> otherwise the host user would have no access to any files written by\n> processes running in the container, notably the logs of failed tests\n> couldn't be included in the build job's trace log.\n> \n> Alas, this piece of code never worked, because it sets the variable\n> holding the user name ($CI_USER) in a subshell, meaning it doesn't\n> have any effect by the time we get to the point to actually use the\n> variable to switch users with 'su'. So all this time we were running\n> the test suite as root.\n\nThis all makes sense to me, and the patch looks sane.\n\n> Reorganize that piece of code in 'ci/run-linux32-build.sh' a bit to\n> avoid that problematic subshell and to ensure that we switch to the\n> right user. Furthermore, make the script's optional host user ID\n> option mandatory, so running the build accidentally as root will\n> become harder when debugging locally. If someone really wants to run\n> the test suite as root, whatever the reasons might be, it'll still be\n> possible to do so by explicitly passing '0' as host user ID.\n\nCoincidentally, we recently set up a docker test build for our local fork\nof Git. We found the whole \"mount the existing git source tree\" strategy\nslightly annoying, exactly because of these mismatches between the host\nand docker uids.\n\nInstead, we ended up with something more like:\n\n git archive HEAD | docker run ...\n\nand the in-container script starts with:\n\n tar -x\n\nto extract the to-be-tested source, and ends with a dump of the failures\nto stdout.\n\nI don't know if it's worth switching strategies now, but just some food\nfor thought. It may be less interesting with Travis, too, because you're\nalso trying to carry the prove cache across runs, which does require\nsome filesystem interaction.\n\n(As an aside, I'm not sure the prove cache is doing much. Running in\nslow-to-fast order helps if you are trying to run massively in parallel,\nbut we only use -j3 for our Travis builds).\n\n-Peff\n"},{"id":"337163","messageId":"20180123164605.GH13068@sigill.intra.peff.net","threadId":"47421","inReplyTo":"20180122133220.18587-6-szeder.dev@gmail.com","subject":"Re: [PATCH 5/5] travis-ci: don't fail if user already exists on 32 bit Linux build job","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-01-23T16:46:05Z","receivedAt":"2018-01-23T16:46:12Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jan 22, 2018 at 02:32:20PM +0100, SZEDER Gábor wrote:\n\n> The 32 bit Linux build job runs in a Docker container, which lends\n> itself to running and debugging locally, too. Especially during\n> debugging one usually doesn't want to start with a fresh container\n> every time, to save time spent on installing a bunch of dependencies.\n> However, that doesn't work quite smootly, because the script running\n> in the container always creates a new user, which then must be removed\n> every time before subsequent executions, or the build script fails.\n> \n> Make this process more convenient and don't try to create that user if\n> it already exists and has the right user ID in the container, so\n> developers don't have to bother with running a 'userdel' each time\n> before they run the build script.\n\nMakes sense.\n\n> The build job on Travis CI always starts with a fresh Docker\n> container, so this change doesn't make a difference there.\n\nI wonder if that is fixable. Installing dependencies into the container\ntakes quite a lot of time, and is just wasted effort.\n\n> diff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh\n> index e37e1d2d5f..13047adde3 100755\n> --- a/ci/run-linux32-build.sh\n> +++ b/ci/run-linux32-build.sh\n> @@ -33,7 +33,13 @@ then\n> \tCI_USER=root\n> else\n> \tCI_USER=ci\n> -\tuseradd -u $HOST_UID $CI_USER\n> +\tif test \"$(id -u $CI_USER)\" = $HOST_UID\n> +\tthen\n> +\t\t: # user already exists with the right ID\n> +\telse\n> +\t\tuseradd -u $HOST_UID $CI_USER\n> +\tfi\n\nIs it worth redirecting the stderr of \"id\" to avoid noise when $CI_USER\ndoes not yet exist at all? Or is that a useful log message? :)\n\n-Peff\n"},{"id":"337200","messageId":"CACsJy8AuY4_2i28Z4P3Y7P+N8vb+NhJLGBa=Gzi=rMxToAPrww@mail.gmail.com","threadId":"47421","inReplyTo":"20180123164605.GH13068@sigill.intra.peff.net","subject":"Re: [PATCH 5/5] travis-ci: don't fail if user already exists on 32 bit Linux build job","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-01-24T00:32:53Z","receivedAt":"2018-01-24T00:33:29Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jan 23, 2018 at 11:46 PM, Jeff King wrote:\n>> The build job on Travis CI always starts with a fresh Docker\n>> container, so this change doesn't make a difference there.\n>\n> I wonder if that is fixable. Installing dependencies into the container\n> takes quite a lot of time, and is just wasted effort.\n\nI agree (because apt-get install phase took forever on my laptop).\nI've seen people just include a Dockerfile that builds everything in\n(except things like creating user matching host). First try to fetch\nit from docker hub (or a local registry but I don't think it applies\nto us). If not found, rebuild the image with Dockerfile and run a new\ncontainer with it. Every time somebody changes dockerfile, they are\nsupposed to step the image version up.\n\nOf course this is still wasted effort unless somebody pushes images to\ndocker registry and the script has to rebuild the image every single\ntime [1]. Somebody could do that manually. I see docker hub has some\nautomated rebuild feature, perhaps that'll work for us.\n\n[1] It still benefits users who run ci scripts manually because the\ncreated image will stay in their local machine.\n-- \nDuy\n"},{"id":"337206","messageId":"CACsJy8AVL4HzPxRjwT9qSc8g_5FwAVgpbEPQ5TfDuxr9C91wqA@mail.gmail.com","threadId":"47421","inReplyTo":"20180122182717.21539-1-szeder.dev@gmail.com","subject":"Re: [PATCH 3/3] read-cache: don't write index twice if we can't write shared index","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-01-24T09:11:11Z","receivedAt":"2018-01-24T09:11:50Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jan 23, 2018 at 1:27 AM, SZEDER Gábor wrote:\n>\n> On Thu, Jan 18, 2018 at 1:47 PM, Duy Nguyen wrote:\n>> On Thu, Jan 18, 2018 at 6:36 PM, SZEDER Gábor wrote:\n>>> This series, queued as 'nd/shared-index-fix', makes the 32 bit Linux\n>>> build job fail on Travis CI. Unfortunately, all it can tell us about\n>>> the failure is this:\n>>>\n>>> Test Summary Report\n>>> -------------------\n>>> t1700-split-index.sh (Wstat: 256 Tests: 23\n>>> Failed: 1)\n>>> Failed test: 23\n>>> Non-zero exit status: 1\n>>> Files=809, Tests=18491, 401 wallclock secs ( 7.22 usr 1.60 sys + 263.16\n>>> cusr 49.58 csys = 321.56 CPU)\n>>> Result: FAIL\n>>>\n>>> because it can't access the test's verbose log due to lack of\n>>> permissions.\n>>>\n>>>\n>>> https://travis-ci.org/git/git/jobs/329681826#L2074\n>>\n>> I may need help getting that log (or even better the trash directory\n>> of t1700).\n>\n> Well, shower thoughts gave me an idea, see the included PoC patch below.\n> I can't really decide whether it's too clever or too ugly :)\n\nI can't comment on the patch. But there is one thing I think I should\nmention. As someone new to Travis, I didn't know that I could set up\nmy own Travis jobs and get the logs myself. Maybe you should point\nthat out when you point people to travis test failures (which I\nappreciate).\n-- \nDuy\n"},{"id":"337251","messageId":"CAM0VKjkGEfL2j2uzRa9=N5jZFbMMKEHn=UX=thxTAMM_g8TB4g@mail.gmail.com","threadId":"47421","inReplyTo":"20180123162633.GE13068@sigill.intra.peff.net","subject":"Re: [PATCH 2/5] travis-ci: use 'set -e' in the 32 bit Linux build job","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-01-24T12:12:48Z","receivedAt":"2018-01-24T12:12:55Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Jan 23, 2018 at 5:26 PM, Jeff King wrote:\n> On Mon, Jan 22, 2018 at 02:32:17PM +0100, SZEDER Gábor wrote:\n>\n>> All 'ci/*' scripts use 'set -e' to break the build job if a command\n>> fails, except 'ci/run-linux32-build.sh' which relies on the && chain\n>> to do the same. This inconsistency among the 'ci/*' scripts is asking\n>> for trouble: I forgot about the && chain more than once while working\n>> on this patch series.\n>\n> I think this actually fixes a bug:\n>\n>> # Update packages to the latest available versions\n>> linux32 --32bit i386 sh -c '\n>> apt update >/dev/null &&\n>> apt install -y build-essential libcurl4-openssl-dev libssl-dev \\\n>> libexpat-dev gettext python >/dev/null\n>> -' &&\n>> +'\n>\n> If this step failed, then...\n>\n>> # If this script runs inside a docker container, then all commands are\n>> # usually executed as root. Consequently, the host user might not be\n>> # able to access the test output files.\n>> # If a host user id is given, then create a user \"ci\" with the host user\n>> # id to make everything accessible to the host user.\n>> -HOST_UID=$1 &&\n>> -CI_USER=$USER &&\n>> -test -z $HOST_UID || (CI_USER=\"ci\" && useradd -u $HOST_UID $CI_USER) &&\n>\n> We'd have triggered the right-hand side of this \"||\", and run the rest\n> of the script. The whole \"||\" block should have been inside {}.\n\nIndeed, the && chain is broken, I didn't noticed that.\n\nLuckily it was broken in a way that it didn't lead to false successes:\nif installing dependencies fails, then the first && chain\nensures that HOST_UID is not set, and then useradd will error out with\n\"invalid user ID 'ci'\", causing the second && chain to abort the script\nand thus breaking the build.\n\nWill update the commit message accordingly.\n\n> But after your patch, it should be fine with \"set -e\". Although...\n>\n>> +HOST_UID=$1\n>> +CI_USER=$USER\n>> +test -z $HOST_UID || (CI_USER=\"ci\" && useradd -u $HOST_UID $CI_USER)\n>\n> If this \"useradd\" step fails, we wouldn't abort the script, because it's\n> part of a conditional. You'd need a manual \"|| exit 1\" at the end of\n> this line. Or to use a real \"if\" block.\n\nNo. It does abort the script, because it isn't part of a conditional.\nTry to run the script twice in the same container instance: in the\nsecond run the user already exists, useradd fails and the whole script\naborts.\n\n\n> Reading this line, I'm also slightly confused by setting CI_USER in the\n> subshell, and then only using it once. Should it be respecting the outer\n> CI_USER setting? If not, should it just hard-code \"ci\" on the useradd\n> command-line? But that has nothing to do with your patch here.\n\nI see you've already made it to patch 4, good :)\n"},{"id":"337259","messageId":"CAM0VKjmS1QMj5=rSnsPBh+49JqChYTZb=LPXhFCH57KVSApS6A@mail.gmail.com","threadId":"47421","inReplyTo":"20180123163038.GF13068@sigill.intra.peff.net","subject":"Re: [PATCH 3/5] travis-ci: don't repeat the path of the cache directory","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-01-24T13:14:11Z","receivedAt":"2018-01-24T13:14:20Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Jan 23, 2018 at 5:30 PM, Jeff King wrote:\n> On Mon, Jan 22, 2018 at 02:32:18PM +0100, SZEDER Gábor wrote:\n\n>> diff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh\n>> index 248183982b..c9476d6598 100755\n>> --- a/ci/run-linux32-build.sh\n>> +++ b/ci/run-linux32-build.sh\n>> @@ -25,10 +25,10 @@ CI_USER=$USER\n>> test -z $HOST_UID || (CI_USER=\"ci\" && useradd -u $HOST_UID $CI_USER)\n>>\n>> # Build and test\n>> -linux32 --32bit i386 su -m -l $CI_USER -c '\n>> +linux32 --32bit i386 su -m -l $CI_USER -c \"\n>> set -ex\n>> cd /usr/src/git\n>> - ln -s /tmp/travis-cache/.prove t/.prove\n>> + test -n '$cache_dir' && ln -s '$cache_dir/.prove' t/.prove\n>> make --jobs=2\n>> make --quiet test\n>> -'\n>> +\"\n>\n> This interpolates $cache_dir while generating the snippet. You've stuck\n> it in single-quotes, which prevents most quoting problems, but obviously\n> it's an issue if the variable itself contains a single-quote. Is there\n> a reason not to just export $cache_dir in the environment and access it\n> directly from the snippet?\n\nRight, that's my conditioned response to single quotes kicking in :)\n\nWe don't even need to export the variable, because Docker has already\ndone it (that's what 'docker run --env VAR' does).\n\n\n> Probably not a _big_ deal, since we control the contents of the\n> variable, but it just seems like a fragile practice to avoid.\n"},{"id":"337262","messageId":"CAM0VKjkRGxAjjibHbJPmC6KZDFL67WEhN1VFqEjbsQAZbgCB8A@mail.gmail.com","threadId":"47421","inReplyTo":"20180123164340.GG13068@sigill.intra.peff.net","subject":"Re: [PATCH 4/5] travis-ci: don't run the test suite as root in the 32 bit Linux build","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-01-24T13:45:18Z","receivedAt":"2018-01-24T13:45:29Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Jan 23, 2018 at 5:43 PM, Jeff King wrote:\n> On Mon, Jan 22, 2018 at 02:32:19PM +0100, SZEDER Gábor wrote:\n>\n>> Travis CI runs the 32 bit Linux build job in a Docker container, where\n>> all commands are executed as root by default. Therefore, ever since\n>> we added this build job in 88dedd5e7 (Travis: also test on 32-bit\n>> Linux, 2017-03-05), we have a bit of code to create a user in the\n>> container matching the ID of the host user and then to run the test\n>> suite as this user. Matching the host user ID is important, because\n>> otherwise the host user would have no access to any files written by\n>> processes running in the container, notably the logs of failed tests\n>> couldn't be included in the build job's trace log.\n>>\n>> Alas, this piece of code never worked, because it sets the variable\n>> holding the user name ($CI_USER) in a subshell, meaning it doesn't\n>> have any effect by the time we get to the point to actually use the\n>> variable to switch users with 'su'. So all this time we were running\n>> the test suite as root.\n>\n> This all makes sense to me, and the patch looks sane.\n>\n>> Reorganize that piece of code in 'ci/run-linux32-build.sh' a bit to\n>> avoid that problematic subshell and to ensure that we switch to the\n>> right user. Furthermore, make the script's optional host user ID\n>> option mandatory, so running the build accidentally as root will\n>> become harder when debugging locally. If someone really wants to run\n>> the test suite as root, whatever the reasons might be, it'll still be\n>> possible to do so by explicitly passing '0' as host user ID.\n>\n> Coincidentally, we recently set up a docker test build for our local fork\n> of Git. We found the whole \"mount the existing git source tree\" strategy\n> slightly annoying, exactly because of these mismatches between the host\n> and docker uids.\n>\n> Instead, we ended up with something more like:\n>\n> git archive HEAD | docker run ...\n>\n> and the in-container script starts with:\n>\n> tar -x\n>\n> to extract the to-be-tested source, and ends with a dump of the failures\n> to stdout.\n>\n> I don't know if it's worth switching strategies now, but just some food\n> for thought. It may be less interesting with Travis, too, because you're\n> also trying to carry the prove cache across runs, which does require\n> some filesystem interaction.\n\nThe prove state in itself doesn't need matching user IDs in the host and\nthe container, because it's only accessed in the container[1].\n\nI think the key is the handling of verbose logs of failed test(s). The\noriginal motivation for matching the user IDs was, I suppose, that we\nwanted to dump the verbose log of the failed test(s) to the trace log on\nthe host, because this way it's fairly consistent with how other build\njob do the same: it uses the same 'after_failure' script to dump the\nverbose logs, it relies on Travis CI to automatically run that script if\nsomething goes wrong, and those logs end up in the 'after_failure'\nfold.\n\n[1] - I ignored saving and restoring the cache here: while it's done on\n the host, but it's done by Travis CI, presumably as root, so the\n owner of the file doesn't matter.\n\n> (As an aside, I'm not sure the prove cache is doing much. Running in\n> slow-to-fast order helps if you are trying to run massively in parallel,\n> but we only use -j3 for our Travis builds).\n\nIt saves about a minute / 10% of runtime; it's mentioned in 7e72cfcee\n(travis-ci: save prove state for the 32 bit Linux build, 2017-12-27).\n"},{"id":"337264","messageId":"20180124154925.GA15803@sigill.intra.peff.net","threadId":"47421","inReplyTo":"CAM0VKjkGEfL2j2uzRa9=N5jZFbMMKEHn=UX=thxTAMM_g8TB4g@mail.gmail.com","subject":"Re: [PATCH 2/5] travis-ci: use 'set -e' in the 32 bit Linux build job","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-01-24T15:49:26Z","receivedAt":"2018-01-24T15:49:33Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 24, 2018 at 01:12:48PM +0100, SZEDER Gábor wrote:\n\n> >> -test -z $HOST_UID || (CI_USER=\"ci\" && useradd -u $HOST_UID $CI_USER) &&\n> [...]\n> Indeed, the && chain is broken, I didn't noticed that.\n> \n> Luckily it was broken in a way that it didn't lead to false successes:\n> if installing dependencies fails, then the first && chain\n> ensures that HOST_UID is not set, and then useradd will error out with\n> \"invalid user ID 'ci'\", causing the second && chain to abort the script\n> and thus breaking the build.\n\nTrue. :) I didn't even look closely at whether the failures could be\ncorrelated.\n\n> >> +HOST_UID=$1\n> >> +CI_USER=$USER\n> >> +test -z $HOST_UID || (CI_USER=\"ci\" && useradd -u $HOST_UID $CI_USER)\n> >\n> > If this \"useradd\" step fails, we wouldn't abort the script, because it's\n> > part of a conditional. You'd need a manual \"|| exit 1\" at the end of\n> > this line. Or to use a real \"if\" block.\n> \n> No. It does abort the script, because it isn't part of a conditional.\n> Try to run the script twice in the same container instance: in the\n> second run the user already exists, useradd fails and the whole script\n> aborts.\n\nYou're right. I was confusing it with this case:\n\n (one; two) || three\n\nin which we continue with \"two\" even if \"one\" fails. But it's only the\nleft-hand side of the || that does that.\n\n-Peff\n"},{"id":"337265","messageId":"20180124155622.GB15803@sigill.intra.peff.net","threadId":"47421","inReplyTo":"CAM0VKjkRGxAjjibHbJPmC6KZDFL67WEhN1VFqEjbsQAZbgCB8A@mail.gmail.com","subject":"Re: [PATCH 4/5] travis-ci: don't run the test suite as root in the 32 bit Linux build","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-01-24T15:56:23Z","receivedAt":"2018-01-24T15:56:33Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 24, 2018 at 02:45:18PM +0100, SZEDER Gábor wrote:\n\n> I think the key is the handling of verbose logs of failed test(s). The\n> original motivation for matching the user IDs was, I suppose, that we\n> wanted to dump the verbose log of the failed test(s) to the trace log on\n> the host, because this way it's fairly consistent with how other build\n> job do the same: it uses the same 'after_failure' script to dump the\n> verbose logs, it relies on Travis CI to automatically run that script if\n> something goes wrong, and those logs end up in the 'after_failure'\n> fold.\n\nThat makes sense, I guess. In the CI system I'm using there's no such\ndistinction, and I just do:\n\n tar -x && make -j64 test && exit 0\n echo \"Failing tests:\"\n echo \"--------------\"\n grep -l '[^0]' t/test-results/*.exit |\n while read failed; do\n base=${failed%.exit}\n name=${base#t/test-results/}\n echo \"==> $name\"\n cat \"$base.out\"\n done\n exit 1\n\nThere's no folds there at all, but we could of course do our own. It may\nnot be worth messing with, though, if you've shaken all the problems out\nof the existing setup.\n\n> > (As an aside, I'm not sure the prove cache is doing much. Running in\n> > slow-to-fast order helps if you are trying to run massively in parallel,\n> > but we only use -j3 for our Travis builds).\n> \n> It saves about a minute / 10% of runtime; it's mentioned in 7e72cfcee\n> (travis-ci: save prove state for the 32 bit Linux build, 2017-12-27).\n\nI'm surprised we get that much benefit out of a 3-way parallel run, but\nI'll believe you if you measured it. I guess it's because a lot of the\nreally long tests are right at the end, numerically (especially if svn\ntests are enabled). I wonder if \"--shuffle\" would yield similar\nbenefits.\n\n-Peff\n"},{"id":"337271","messageId":"20180124180142.GA26290@sigill.intra.peff.net","threadId":"47421","inReplyTo":"20180124155622.GB15803@sigill.intra.peff.net","subject":"Re: [PATCH 4/5] travis-ci: don't run the test suite as root in the 32 bit Linux build","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-01-24T18:01:42Z","receivedAt":"2018-01-24T18:01:57Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 24, 2018 at 10:56:22AM -0500, Jeff King wrote:\n\n> > > (As an aside, I'm not sure the prove cache is doing much. Running in\n> > > slow-to-fast order helps if you are trying to run massively in parallel,\n> > > but we only use -j3 for our Travis builds).\n> > \n> > It saves about a minute / 10% of runtime; it's mentioned in 7e72cfcee\n> > (travis-ci: save prove state for the 32 bit Linux build, 2017-12-27).\n> \n> I'm surprised we get that much benefit out of a 3-way parallel run, but\n> I'll believe you if you measured it. I guess it's because a lot of the\n> really long tests are right at the end, numerically (especially if svn\n> tests are enabled). I wonder if \"--shuffle\" would yield similar\n> benefits.\n\nJust for fun, I tried running:\n\n cd t\n best-of-five make GIT_PROVE_OPTS='-j3'\n best-of-five make GIT_PROVE_OPTS='-j3 --state=slow,save'\n best-of-five make GIT_PROVE_OPTS='-j3 --shuffle'\n\nand got:\n\n stock\n Attempt 1: 137.057\n Attempt 2: 137.635\n Attempt 3: 138.925\n Attempt 4: 134.693\n Attempt 5: 139.581\n\n slow,save\n Attempt 1: 133.157\n Attempt 2: 135.602\n Attempt 3: 133.225\n Attempt 4: 136.278\n Attempt 5: 133.382\n\n shuffle\n Attempt 1: 136.717\n Attempt 2: 138.805\n Attempt 3: 145.734\n Attempt 4: 145.226\n Attempt 5: 145.889\n\nI had expected the shuffle to be sometimes-fast and sometimes-slow, but\nit seems like it is just-slow. So that's probably not a big win. It also\ndoesn't look like state-saving gets us much.\n\nThose runs don't have cvs/svn installed. I repeated with those\ninstalled. The whole run is much slower then (about 230s), but the\nrelative timings are the same.\n\nI wonder what is different between my setup and Travis. I guess one\nis that I use a tmpfs for the test-root. I wonder if that could throw\noff the relative timings, or the importance of parallelization.\n\n-Peff\n"},{"id":"337287","messageId":"CAM0VKjms5jRJj8=yD25cREkjn309tnnje4XhUAY1V1+OKWmgfg@mail.gmail.com","threadId":"47421","inReplyTo":"20180123164605.GH13068@sigill.intra.peff.net","subject":"Re: [PATCH 5/5] travis-ci: don't fail if user already exists on 32 bit Linux build job","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-01-24T19:39:12Z","receivedAt":"2018-01-24T19:39:20Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Jan 23, 2018 at 5:46 PM, Jeff King wrote:\n> On Mon, Jan 22, 2018 at 02:32:20PM +0100, SZEDER Gábor wrote:\n\n>> diff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh\n>> index e37e1d2d5f..13047adde3 100755\n>> --- a/ci/run-linux32-build.sh\n>> +++ b/ci/run-linux32-build.sh\n>> @@ -33,7 +33,13 @@ then\n>> CI_USER=root\n>> else\n>> CI_USER=ci\n>> - useradd -u $HOST_UID $CI_USER\n>> + if test \"$(id -u $CI_USER)\" = $HOST_UID\n>> + then\n>> + : # user already exists with the right ID\n>> + else\n>> + useradd -u $HOST_UID $CI_USER\n>> + fi\n>\n> Is it worth redirecting the stderr of \"id\" to avoid noise when $CI_USER\n> does not yet exist at all? Or is that a useful log message? :)\n\nI think it's worth silencing that error. I'm not even sure it was\nuseful while working on this patch series, but an error message in the\nmiddle of the log of a successful build job is surely distracting and\nmight raise some eyebrows (though I suspect that barely anyone looks at\nlogs of successful build jobs).\n\nOTOH, I might turn that comment into an echo msg...\n"},{"id":"337290","messageId":"20180124195122.GA7278@sigill.intra.peff.net","threadId":"47421","inReplyTo":"20180124180142.GA26290@sigill.intra.peff.net","subject":"Re: [PATCH 4/5] travis-ci: don't run the test suite as root in the 32 bit Linux build","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-01-24T19:51:22Z","receivedAt":"2018-01-24T19:51:29Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 24, 2018 at 01:01:42PM -0500, Jeff King wrote:\n\n> Just for fun, I tried running:\n> \n> cd t\n> best-of-five make GIT_PROVE_OPTS='-j3'\n> best-of-five make GIT_PROVE_OPTS='-j3 --state=slow,save'\n> best-of-five make GIT_PROVE_OPTS='-j3 --shuffle'\n> [...]\n> I wonder what is different between my setup and Travis. I guess one\n> is that I use a tmpfs for the test-root. I wonder if that could throw\n> off the relative timings, or the importance of parallelization.\n\nI just tried again without a tmpfs (actually, with\nGIT_TEST_OPTS=--verbose-log, which matches Travis) and still got the\n\"prove\" version as only about 10s faster. So I dunno. My disk is an SSD,\nso that probably _is_ faster than Travis's disks, and maybe that would\nmatter.\n\nI'm giving up on finding it. If you happen to have a recipe for\nreproducing a larger speedup with \"--state=slow,save\", I'd be curious to\nsee it. But don't spend much time on it on my account. The existing\nprove-caching setup is there and working, so even in the worst case if\nit's not helping, it's not actively hurting much.\n\n-Peff\n"},{"id":"337558","messageId":"5C0E7737-7FE0-48F0-8667-E3891D892249@gmail.com","threadId":"47421","inReplyTo":"20180122182717.21539-1-szeder.dev@gmail.com","subject":"Re: [PATCH 3/3] read-cache: don't write index twice if we can't write shared index","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2018-01-26T22:44:56Z","receivedAt":"2018-01-26T22:45:05Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 22 Jan 2018, at 19:27, SZEDER Gábor wrote:\n> \n> \n> On Thu, Jan 18, 2018 at 1:47 PM, Duy Nguyen wrote:\n>> On Thu, Jan 18, 2018 at 6:36 PM, SZEDER Gábor wrote:\n>>> This series, queued as 'nd/shared-index-fix', makes the 32 bit Linux\n>>> build job fail on Travis CI. Unfortunately, all it can tell us about\n>>> the failure is this:\n>>> \n>>> Test Summary Report\n>>> -------------------\n>>> t1700-split-index.sh (Wstat: 256 Tests: 23\n>>> Failed: 1)\n>>> Failed test: 23\n>>> Non-zero exit status: 1\n>>> Files=809, Tests=18491, 401 wallclock secs ( 7.22 usr 1.60 sys + 263.16\n>>> cusr 49.58 csys = 321.56 CPU)\n>>> Result: FAIL\n>>> \n>>> because it can't access the test's verbose log due to lack of\n>>> permissions.\n>>> \n>>> \n>>> https://travis-ci.org/git/git/jobs/329681826#L2074\n>> \n>> I may need help getting that log (or even better the trash directory\n>> of t1700). \n> \n> Well, shower thoughts gave me an idea, see the included PoC patch below.\n> I can't really decide whether it's too clever or too ugly :)\n\nCreative :-)\n\nAlternatively, we could store write access credentials [1] of a public\nrepo in Travis and upload the trash directory to it. This could\nwork \"out-of-the-box\" for git/git. Personal Travis environments (e.g.\nlarsxschneider/git) would need to setup that individually.\n\n- Lars\n\n[1] https://docs.travis-ci.com/user/environment-variables/#Defining-Variables-in-Repository-Settings\n ^^^ that's the mechanism we use for the Windows build credentials"},{"id":"337569","messageId":"20180127121801.GC2130@hank","threadId":"47421","inReplyTo":"xmqqpo62mncc.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 1/3] read-cache: fix reading the shared index for other repos","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2018-01-27T12:18:01Z","receivedAt":"2018-01-27T12:15:40Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 01/21, Junio C Hamano wrote:\n> Thomas Gummerer writes:\n> \n> > On 01/19, Junio C Hamano wrote:\n> >> Thomas Gummerer writes:\n> >> \n> >> > read_cache_from() defaults to using the gitdir of the_repository. As it\n> >> > is mostly a convenience macro, having to pass get_git_dir() for every\n> >> > call seems overkill, and if necessary users can have more control by\n> >> > using read_index_from().\n> >> \n> >> This was a bit painful change, given that some changes in flight do\n> >> add new callsites to read_index_from() and they got the function\n> >> changed under their feet.\n> >\n> > Sorry about that. Is there any way to make such a change less painful\n> > in the future?\n> \n> One way is to do for read_index_from() what you did for the existing\n> users of read_cache_from(). Introduce a _new_ helper that will not\n> be known for any existing topics in flight, and use that to make the\n> existing API a thin wrapper around it.\n\nI'll do that next time. My worries were just that the\n'read_index_from()' API is broken with split index, which is what this\nseries fixes. It would be easy to introduce new breakages if we're\nnot careful.\n\n> I _think_ I got it right with evil merge, so unless this fix needs\n> to be delayed for extended period of time for whatever reason while\n> any more new callers of the function appears (which is unlikely), we\n> should be OK ;-)\n\nThanks! As mentioned there's one call that was added that the evil\nmerge didn't get quite right, for which I sent the diff below in my\nprevious email. But I'm happy to fix that on top once this series\ngoes to master or next if that's preferred.\n\ndiff --git a/builtin/worktree.c b/builtin/worktree.c\nindex 6a49f9e628..4d86a3574f 100644\n--- a/builtin/worktree.c\n+++ b/builtin/worktree.c\n@@ -612,7 +612,8 @@ static void validate_no_submodules(const struct worktree *wt)\n struct index_state istate = {0};\n int i, found_submodules = 0;\n \n- if (read_index_from(&istate, worktree_git_path(wt, \"index\"), get_git_dir()) > 0) {\n+ if (read_index_from(&istate, worktree_git_path(wt, \"index\"),\n+ get_worktree_git_dir(wt)) > 0) {\n for (i = 0; i < istate.cache_nr; i++) {\n struct cache_entry *ce = istate.cache[i];\n \ndiff --git a/t/t2028-worktree-move.sh b/t/t2028-worktree-move.sh\nindex b3105eaaed..8faf61bbf5 100755\n--- a/t/t2028-worktree-move.sh\n+++ b/t/t2028-worktree-move.sh\n@@ -90,6 +90,16 @@ test_expect_success 'move main worktree' '\n test_must_fail git worktree move . def\n '\n \n+test_expect_success 'move worktree with split index' '\n+ git worktree add test &&\n+ (\n+ cd test &&\n+ test_commit file &&\n+ git update-index --split-index\n+ ) &&\n+ git worktree move test test-destination\n+'\n+\n test_expect_success 'remove main worktree' '\n test_must_fail git worktree remove .\n '\n\n> Thanks.\n"},{"id":"338675","messageId":"xmqqy3k45sob.fsf@gitster-ct.c.googlers.com","threadId":"47421","inReplyTo":"xmqqpo62mncc.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 1/3] read-cache: fix reading the shared index for other repos","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-02-07T22:41:24Z","receivedAt":"2018-02-07T22:41:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>>> This was a bit painful change, given that some changes in flight do\n>>> add new callsites to read_index_from() and they got the function\n>>> changed under their feet.\n>>\n>> Sorry about that. Is there any way to make such a change less painful\n>> in the future?\n>\n> One way is to do for read_index_from() what you did for the existing\n> users of read_cache_from(). Introduce a _new_ helper that will not\n> be known for any existing topics in flight, and use that to make the\n> existing API a thin wrapper around it.\n\nThis continues to cause pain simply because read_index_from() is\nsomething new topics want to stay stable X-<. \n\nI'll be merging this to 'next' hopefully as part of today's\nintegration run, so will need to endure the pain only until it (and\nother topics that conflict with it) all graduate to 'master'.\n\nNext time a patch with an internal API change like this appears,\nplease remind me to push it back a lot stronger. I was too lenient\nand ended up slowing down the progress of other topics this time.\nThanks.\n\n\n"}]}