{"thread":{"id":"37720","subject":"[PATCH 1/4] checkout: do not fail if target is an empty directory","startedAt":"2014-10-12T05:13:07Z","lastAt":"2014-11-03T22:07:02Z","messageCount":24,"participants":["Max Kirillov","Duy Nguyen","Jens Lehmann","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"250525","messageId":"1413090791-14428-1-git-send-email-max@max630.net","threadId":"37720","inReplyTo":null,"subject":"[PATCH 0/4] Multiple worktrees vs. submodules fixes","fromName":"Max Kirillov","fromEmail":"max@max630.net","sentAt":"2014-10-12T05:13:07Z","receivedAt":"2014-10-12T05:13:07Z","isPatch":true,"sender":{"key":"max@max630.net","avatar":"https://avatars.githubusercontent.com/u/381560?v=4"},"body":"Hi.\n\nThese are fixes of issues with submodules with use of multiple working\ntrees.\n\nTo be applied on top of the $gmane/257559, (6b4ce012cb in current pu).\n\nMax Kirillov (4):\n  checkout: do not fail if target is an empty directory\n  submodule refactor: use git_path_submodule() in add_submodule_odb()\n  git-common-dir: make \"modules/\" per-working-directory directory\n  path: implement common_dir handling in git_path_submodule()\n\n Documentation/gitrepository-layout.txt |  4 +--\n builtin/checkout.c                     |  2 +-\n cache.h                                |  1 +\n path.c                                 | 26 +++++++++++---\n setup.c                                | 17 +++++++---\n submodule.c                            | 28 ++++++---------\n t/t2025-checkout-to.sh                 |  7 +++-\n t/t7410-submodule-checkout-to.sh       | 62 ++++++++++++++++++++++++++++++++++\n 8 files changed, 114 insertions(+), 33 deletions(-)\n create mode 100755 t/t7410-submodule-checkout-to.sh\n\n-- \n2.0.1.1697.g73c6810\n"},{"id":"250522","messageId":"1413090791-14428-2-git-send-email-max@max630.net","threadId":"37720","inReplyTo":"1413090791-14428-1-git-send-email-max@max630.net","subject":"[PATCH 1/4] checkout: do not fail if target is an empty directory","fromName":"Max Kirillov","fromEmail":"max@max630.net","sentAt":"2014-10-12T05:13:08Z","receivedAt":"2014-10-12T05:13:08Z","isPatch":true,"sender":{"key":"max@max630.net","avatar":"https://avatars.githubusercontent.com/u/381560?v=4"},"body":"Non-recursive checkout creates empty directpries in place of submodules.\nIf then I try to \"checkout --to\" submodules there, it refuses to do so,\nbecause directory already exists.\n\nFix by allowing checking out to empty directory. Add test and modify the\nexisting one so that it uses non-empty directory.\n\nSigned-off-by: Max Kirillov <max@max630.net>\n---\n builtin/checkout.c     | 2 +-\n t/t2025-checkout-to.sh | 7 ++++++-\n 2 files changed, 7 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/checkout.c b/builtin/checkout.c\nindex 01d0f2f..74eabe7 100644\n--- a/builtin/checkout.c\n+++ b/builtin/checkout.c\n@@ -865,7 +865,7 @@ static int prepare_linked_checkout(const struct checkout_opts *opts,\n \n \tif (!new->commit)\n \t\tdie(_(\"no branch specified\"));\n-\tif (file_exists(path))\n+\tif (file_exists(path) && !is_empty_dir(path))\n \t\tdie(_(\"'%s' already exists\"), path);\n \n \tlen = strlen(path);\ndiff --git a/t/t2025-checkout-to.sh b/t/t2025-checkout-to.sh\nindex eddd325..915b506 100755\n--- a/t/t2025-checkout-to.sh\n+++ b/t/t2025-checkout-to.sh\n@@ -13,10 +13,15 @@ test_expect_success 'checkout --to not updating paths' '\n '\n \n test_expect_success 'checkout --to an existing worktree' '\n-\tmkdir existing &&\n+\tmkdir -p existing/subtree &&\n \ttest_must_fail git checkout --detach --to existing master\n '\n \n+test_expect_success 'checkout --to an existing empty worktree' '\n+\tmkdir existing_empty &&\n+\tgit checkout --detach --to existing_empty master\n+'\n+\n test_expect_success 'checkout --to refuses to checkout locked branch' '\n \ttest_must_fail git checkout --to zere master &&\n \t! test -d zere &&\n-- \n2.0.1.1697.g73c6810\n"},{"id":"250523","messageId":"1413090791-14428-3-git-send-email-max@max630.net","threadId":"37720","inReplyTo":"1413090791-14428-1-git-send-email-max@max630.net","subject":"[PATCH 2/4] submodule refactor: use git_path_submodule() in add_submodule_odb()","fromName":"Max Kirillov","fromEmail":"max@max630.net","sentAt":"2014-10-12T05:13:09Z","receivedAt":"2014-10-12T05:13:09Z","isPatch":true,"sender":{"key":"max@max630.net","avatar":"https://avatars.githubusercontent.com/u/381560?v=4"},"body":"Signed-off-by: Max Kirillov <max@max630.net>\n---\n submodule.c | 28 ++++++++++------------------\n 1 file changed, 10 insertions(+), 18 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 34094f5..4aad3d4 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -122,43 +122,35 @@ void stage_updated_gitmodules(void)\n \n static int add_submodule_odb(const char *path)\n {\n-\tstruct strbuf objects_directory = STRBUF_INIT;\n \tstruct alternate_object_database *alt_odb;\n+\tconst char* objects_directory;\n \tint ret = 0;\n-\tconst char *git_dir;\n \n-\tstrbuf_addf(&objects_directory, \"%s/.git\", path);\n-\tgit_dir = read_gitfile(objects_directory.buf);\n-\tif (git_dir) {\n-\t\tstrbuf_reset(&objects_directory);\n-\t\tstrbuf_addstr(&objects_directory, git_dir);\n-\t}\n-\tstrbuf_addstr(&objects_directory, \"/objects/\");\n-\tif (!is_directory(objects_directory.buf)) {\n+\tobjects_directory = git_path_submodule(path, \"objects/\");\n+\tif (!is_directory(objects_directory)) {\n \t\tret = -1;\n \t\tgoto done;\n \t}\n+\n \t/* avoid adding it twice */\n \tfor (alt_odb = alt_odb_list; alt_odb; alt_odb = alt_odb->next)\n-\t\tif (alt_odb->name - alt_odb->base == objects_directory.len &&\n-\t\t\t\t!strncmp(alt_odb->base, objects_directory.buf,\n-\t\t\t\t\tobjects_directory.len))\n+\t\tif (alt_odb->name - alt_odb->base == strlen(objects_directory) &&\n+\t\t\t\t!strcmp(alt_odb->base, objects_directory))\n \t\t\tgoto done;\n \n-\talt_odb = xmalloc(objects_directory.len + 42 + sizeof(*alt_odb));\n+\talt_odb = xmalloc(strlen(objects_directory) + 42 + sizeof(*alt_odb));\n \talt_odb->next = alt_odb_list;\n-\tstrcpy(alt_odb->base, objects_directory.buf);\n-\talt_odb->name = alt_odb->base + objects_directory.len;\n+\tstrcpy(alt_odb->base, objects_directory);\n+\talt_odb->name = alt_odb->base + strlen(objects_directory);\n \talt_odb->name[2] = '/';\n \talt_odb->name[40] = '\\0';\n \talt_odb->name[41] = '\\0';\n \talt_odb_list = alt_odb;\n \n \t/* add possible alternates from the submodule */\n-\tread_info_alternates(objects_directory.buf, 0);\n+\tread_info_alternates(objects_directory, 0);\n \tprepare_alt_odb();\n done:\n-\tstrbuf_release(&objects_directory);\n \treturn ret;\n }\n \n-- \n2.0.1.1697.g73c6810\n"},{"id":"250524","messageId":"1413090791-14428-4-git-send-email-max@max630.net","threadId":"37720","inReplyTo":"1413090791-14428-1-git-send-email-max@max630.net","subject":"[PATCH 3/4] git-common-dir: make \"modules/\" per-working-directory directory","fromName":"Max Kirillov","fromEmail":"max@max630.net","sentAt":"2014-10-12T05:13:10Z","receivedAt":"2014-10-12T05:13:10Z","isPatch":true,"sender":{"key":"max@max630.net","avatar":"https://avatars.githubusercontent.com/u/381560?v=4"},"body":"Each working directory of main repository has its own working directory\nof submodule, and in most cases they should be checked out to different\nrevisions. So they should be separated.\n\nIt looks logical to make submodule instances in different working\ndirectories to reuse the submodule directory in the common dir of\nthe main repository, and probably this is how \"checkout --to\" should\ninitialize them called on the main repository, but they also should work\nfine being completely separated clones.\n\nTestfile t7410-submodule-checkout-to.sh demostrates the behavior.\n\nSigned-off-by: Max Kirillov <max@max630.net>\n---\n Documentation/gitrepository-layout.txt |  4 +--\n path.c                                 |  2 +-\n t/t7410-submodule-checkout-to.sh       | 50 ++++++++++++++++++++++++++++++++++\n 3 files changed, 52 insertions(+), 4 deletions(-)\n create mode 100755 t/t7410-submodule-checkout-to.sh\n\ndiff --git a/Documentation/gitrepository-layout.txt b/Documentation/gitrepository-layout.txt\nindex 2b30a92..7173b38 100644\n--- a/Documentation/gitrepository-layout.txt\n+++ b/Documentation/gitrepository-layout.txt\n@@ -248,9 +248,7 @@ commondir::\n \tincomplete without the repository pointed by \"commondir\".\n \n modules::\n-\tContains the git-repositories of the submodules. This\n-\tdirectory is ignored if $GIT_COMMON_DIR is set and\n-\t\"$GIT_COMMON_DIR/modules\" will be used instead.\n+\tContains the git-repositories of the submodules.\n \n worktrees::\n \tContains worktree specific information of linked\ndiff --git a/path.c b/path.c\nindex 35d498e..a5c51a3 100644\n--- a/path.c\n+++ b/path.c\n@@ -92,7 +92,7 @@ static void replace_dir(struct strbuf *buf, int len, const char *newdir)\n }\n \n static const char *common_list[] = {\n-\t\"/branches\", \"/hooks\", \"/info\", \"!/logs\", \"/lost-found\", \"/modules\",\n+\t\"/branches\", \"/hooks\", \"/info\", \"!/logs\", \"/lost-found\",\n \t\"/objects\", \"/refs\", \"/remotes\", \"/worktrees\", \"/rr-cache\", \"/svn\",\n \t\"config\", \"!gc.pid\", \"packed-refs\", \"shallow\",\n \tNULL\ndiff --git a/t/t7410-submodule-checkout-to.sh b/t/t7410-submodule-checkout-to.sh\nnew file mode 100755\nindex 0000000..8f30aed\n--- /dev/null\n+++ b/t/t7410-submodule-checkout-to.sh\n@@ -0,0 +1,50 @@\n+#!/bin/sh\n+\n+test_description='Combination of submodules and multiple workdirs'\n+\n+. ./test-lib.sh\n+\n+base_path=$(pwd -P)\n+\n+test_expect_success 'setup: make origin' \\\n+    'mkdir -p origin/sub && ( cd origin/sub && git init &&\n+\techo file1 >file1 &&\n+\tgit add file1 &&\n+\tgit commit -m file1 ) &&\n+    mkdir -p origin/main && ( cd origin/main && git init &&\n+\tgit submodule add ../sub &&\n+\tgit commit -m \"add sub\" ) &&\n+    ( cd origin/sub &&\n+\techo file1updated >file1 &&\n+\tgit add file1 &&\n+\tgit commit -m \"file1 updated\" ) &&\n+    ( cd origin/main/sub && git pull ) &&\n+    ( cd origin/main &&\n+\tgit add sub &&\n+\tgit commit -m \"sub updated\" )'\n+\n+test_expect_success 'setup: clone' \\\n+    'mkdir clone && ( cd clone &&\n+\tgit clone --recursive \"$base_path/origin/main\")'\n+\n+rev1_hash_main=$(git --git-dir=origin/main/.git show --pretty=format:%h -q \"HEAD~1\")\n+rev1_hash_sub=$(git --git-dir=origin/sub/.git show --pretty=format:%h -q \"HEAD~1\")\n+\n+test_expect_success 'checkout main' \\\n+    'mkdir default_checkout &&\n+    (cd clone/main &&\n+\tgit checkout --to \"$base_path/default_checkout/main\" \"$rev1_hash_main\")'\n+\n+test_expect_failure 'can see submodule diffs just after checkout' \\\n+    '(cd default_checkout/main && git diff --submodule master\"^!\" | grep \"file1 updated\")'\n+\n+test_expect_success 'checkout main and initialize independed clones' \\\n+    'mkdir fully_cloned_submodule &&\n+    (cd clone/main &&\n+\tgit checkout --to \"$base_path/fully_cloned_submodule/main\" \"$rev1_hash_main\") &&\n+    (cd fully_cloned_submodule/main && git submodule update)'\n+\n+test_expect_success 'can see submodule diffs after independed cloning' \\\n+    '(cd fully_cloned_submodule/main && git diff --submodule master\"^!\" | grep \"file1 updated\")'\n+\n+test_done\n-- \n2.0.1.1697.g73c6810\n"},{"id":"250526","messageId":"1413090791-14428-5-git-send-email-max@max630.net","threadId":"37720","inReplyTo":"1413090791-14428-1-git-send-email-max@max630.net","subject":"[PATCH 4/4] path: implement common_dir handling in git_path_submodule()","fromName":"Max Kirillov","fromEmail":"max@max630.net","sentAt":"2014-10-12T05:13:11Z","receivedAt":"2014-10-12T05:13:11Z","isPatch":true,"sender":{"key":"max@max630.net","avatar":"https://avatars.githubusercontent.com/u/381560?v=4"},"body":"Currently git_path_submodule() does not handle submodules being a linked\ncheckout. The visible result is that \"git diff --submodule\" fails to\nreport changes in the submodule.\n\nPerform the same resolution as with parent repository, but ignore the\nGIT_COMMON_DIR environment variable, because it would mean common\ndirectory for the parent repository and does not make sense for\nsubmodule.\n\nAlso add test for functionality which uses this call.\n\nSigned-off-by: Max Kirillov <max@max630.net>\n---\n cache.h                          |  1 +\n path.c                           | 24 ++++++++++++++++++++----\n setup.c                          | 17 ++++++++++++-----\n t/t7410-submodule-checkout-to.sh | 12 ++++++++++++\n 4 files changed, 45 insertions(+), 9 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex ab3adb6..09821df 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -437,6 +437,7 @@ extern char *get_object_directory(void);\n extern char *get_index_file(void);\n extern char *get_graft_file(void);\n extern int set_git_dir(const char *path);\n+extern int get_common_dir_noenv(struct strbuf *sb, const char *gitdir);\n extern int get_common_dir(struct strbuf *sb, const char *gitdir);\n extern const char *get_git_namespace(void);\n extern const char *strip_namespace(const char *namespaced_ref);\ndiff --git a/path.c b/path.c\nindex a5c51a3..78f718f 100644\n--- a/path.c\n+++ b/path.c\n@@ -98,7 +98,7 @@ static const char *common_list[] = {\n \tNULL\n };\n \n-static void update_common_dir(struct strbuf *buf, int git_dir_len)\n+static void update_common_dir(struct strbuf *buf, int git_dir_len, const char* common_dir)\n {\n \tchar *base = buf->buf + git_dir_len;\n \tconst char **p;\n@@ -115,12 +115,17 @@ static void update_common_dir(struct strbuf *buf, int git_dir_len)\n \t\t\tpath++;\n \t\t\tis_dir = 1;\n \t\t}\n+\n+\t\tif (!common_dir) {\n+\t\t\tcommon_dir = get_git_common_dir();\n+\t\t}\n+\n \t\tif (is_dir && dir_prefix(base, path)) {\n-\t\t\treplace_dir(buf, git_dir_len, get_git_common_dir());\n+\t\t\treplace_dir(buf, git_dir_len, common_dir);\n \t\t\treturn;\n \t\t}\n \t\tif (!is_dir && !strcmp(base, path)) {\n-\t\t\treplace_dir(buf, git_dir_len, get_git_common_dir());\n+\t\t\treplace_dir(buf, git_dir_len, common_dir);\n \t\t\treturn;\n \t\t}\n \t}\n@@ -160,7 +165,7 @@ static void adjust_git_path(struct strbuf *buf, int git_dir_len)\n \telse if (git_db_env && dir_prefix(base, \"objects\"))\n \t\treplace_dir(buf, git_dir_len + 7, get_object_directory());\n \telse if (git_common_dir_env)\n-\t\tupdate_common_dir(buf, git_dir_len);\n+\t\tupdate_common_dir(buf, git_dir_len, NULL);\n }\n \n static void do_git_path(struct strbuf *buf, const char *fmt, va_list args)\n@@ -256,6 +261,8 @@ const char *git_path_submodule(const char *path, const char *fmt, ...)\n {\n \tstruct strbuf *buf = get_pathname();\n \tconst char *git_dir;\n+\tstruct strbuf git_submodule_common_dir = STRBUF_INIT;\n+\tstruct strbuf git_submodule_dir = STRBUF_INIT;\n \tva_list args;\n \n \tstrbuf_addstr(buf, path);\n@@ -269,11 +276,20 @@ const char *git_path_submodule(const char *path, const char *fmt, ...)\n \t\tstrbuf_addstr(buf, git_dir);\n \t}\n \tstrbuf_addch(buf, '/');\n+\tstrbuf_addstr(&git_submodule_dir, buf->buf);\n \n \tva_start(args, fmt);\n \tstrbuf_vaddf(buf, fmt, args);\n \tva_end(args);\n+\n+\tif (get_common_dir_noenv(&git_submodule_common_dir, git_submodule_dir.buf)) {\n+\t\tupdate_common_dir(buf, git_submodule_dir.len, git_submodule_common_dir.buf);\n+\t}\n+\n \tstrbuf_cleanup_path(buf);\n+\n+\tstrbuf_release(&git_submodule_dir);\n+\tstrbuf_release(&git_submodule_common_dir);\n \treturn buf->buf;\n }\n \ndiff --git a/setup.c b/setup.c\nindex fb61860..ffda622 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -226,14 +226,21 @@ void verify_non_filename(const char *prefix, const char *arg)\n \n int get_common_dir(struct strbuf *sb, const char *gitdir)\n {\n+\tconst char *git_env_common_dir = getenv(GIT_COMMON_DIR_ENVIRONMENT);\n+\tif (git_env_common_dir) {\n+\t\tstrbuf_addstr(sb, git_env_common_dir);\n+\t\treturn 1;\n+\t} else {\n+\t\treturn get_common_dir_noenv(sb, gitdir);\n+\t}\n+}\n+\n+int get_common_dir_noenv(struct strbuf *sb, const char *gitdir)\n+{\n \tstruct strbuf data = STRBUF_INIT;\n \tstruct strbuf path = STRBUF_INIT;\n-\tconst char *git_common_dir = getenv(GIT_COMMON_DIR_ENVIRONMENT);\n \tint ret = 0;\n-\tif (git_common_dir) {\n-\t\tstrbuf_addstr(sb, git_common_dir);\n-\t\treturn 1;\n-\t}\n+\n \tstrbuf_addf(&path, \"%s/commondir\", gitdir);\n \tif (file_exists(path.buf)) {\n \t\tif (strbuf_read_file(&data, path.buf, 0) <= 0)\ndiff --git a/t/t7410-submodule-checkout-to.sh b/t/t7410-submodule-checkout-to.sh\nindex 8f30aed..153f5c5 100755\n--- a/t/t7410-submodule-checkout-to.sh\n+++ b/t/t7410-submodule-checkout-to.sh\n@@ -47,4 +47,16 @@ test_expect_success 'checkout main and initialize independed clones' \\\n test_expect_success 'can see submodule diffs after independed cloning' \\\n     '(cd fully_cloned_submodule/main && git diff --submodule master\"^!\" | grep \"file1 updated\")'\n \n+test_expect_success 'checkout sub manually' \\\n+    'mkdir linked_submodule &&\n+    (cd clone/main &&\n+\tgit checkout --to \"$base_path/linked_submodule/main\" \"$rev1_hash_main\") &&\n+    (cd clone/main/sub &&\n+\tgit checkout --to \"$base_path/linked_submodule/main/sub\" \"$rev1_hash_sub\") &&\n+    mkdir clone/main/.git/worktrees/main/modules &&\n+\techo \"gitdir: ../../modules/sub/worktrees/sub\" > clone/main/.git/worktrees/main/modules/sub'\n+\n+test_expect_success 'can see submodule diffs after manual checkout of linked submodule' \\\n+    '(cd linked_submodule/main && git diff --submodule master\"^!\" | grep \"file1 updated\")'\n+\n test_done\n-- \n2.0.1.1697.g73c6810\n"},{"id":"250605","messageId":"CACsJy8BUtkWKE+P_sHgpAY6wJ9tpzxZRtZHULiLoO=dGnBjkHQ@mail.gmail.com","threadId":"37720","inReplyTo":"1413090791-14428-1-git-send-email-max@max630.net","subject":"Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-10-14T12:17:54Z","receivedAt":"2014-10-14T12:17:54Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Oct 12, 2014 at 12:13 PM, Max Kirillov <max@max630.net> wrote:\n> These are fixes of issues with submodules with use of multiple working\n> trees.\n\nI think the patches look fine from the nd/multiple-work-trees writer's\nperspective. I know too little about submodules to judge if this is\nthe right way and not that way..\n-- \nDuy\n"},{"id":"250615","messageId":"543D58D9.5060606@web.de","threadId":"37720","inReplyTo":"CACsJy8BUtkWKE+P_sHgpAY6wJ9tpzxZRtZHULiLoO=dGnBjkHQ@mail.gmail.com","subject":"Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2014-10-14T17:09:45Z","receivedAt":"2014-10-14T17:09:45Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 14.10.2014 um 14:17 schrieb Duy Nguyen:\n> On Sun, Oct 12, 2014 at 12:13 PM, Max Kirillov <max@max630.net> wrote:\n>> These are fixes of issues with submodules with use of multiple working\n>> trees.\n>\n> I think the patches look fine from the nd/multiple-work-trees writer's\n> perspective. I know too little about submodules to judge if this is\n> the right way and not that way..\n\nSorry, I was too busy to review this work until now, but here we go:\n\nIf I understand multiple work trees correctly, everything except work\ntree local stuff is redirected into GIT_COMMON_DIR. This is a very cool\nfeature I'd love to see on our CI server to reduce disk footprint and\nclone times, especially for submodules!\n\nBut I can't see how that can work by just sharing the modules directory\ntree, as that contains work tree related files - e.g. the index - for\neach submodule. AFAICS sharing them between work trees will work only\nif the content of the modules directory is partly present in GIT_DIR -\nfor work tree related files - and only the common stuff is taken from\nGIT_COMMON_DIR (Or did I just miss the magic that already does that?).\nAnd I didn't try to wrap my head around recursive submodules yet ...\n\nUntil that problem is solved it looks wrong to pass GIT_COMMON_DIR into\nsubmodule recursion, I believe GIT_COMMON_DIR should be added to the\nlocal_repo_env array (and even if it is passed on later, we might have\nto append \"/modules/<submodule_name>\" to make it point to the correct\nlocation).\n\nBut maybe I'm missing something?\n"},{"id":"250616","messageId":"xmqqoatezhnx.fsf@gitster.dls.corp.google.com","threadId":"37720","inReplyTo":"543D58D9.5060606@web.de","subject":"Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-10-14T17:26:42Z","receivedAt":"2014-10-14T17:26:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jens Lehmann <Jens.Lehmann@web.de> writes:\n\n> But I can't see how that can work by just sharing the modules directory\n> tree, as that contains work tree related files - e.g. the index - for\n> each submodule. AFAICS sharing them between work trees will work only\n> if the content of the modules directory is partly present in GIT_DIR -\n> for work tree related files - and only the common stuff is taken from\n> GIT_COMMON_DIR (Or did I just miss the magic that already does that?).\n\nThe first time I saw the patch 3/4 in this series, my reaction was\n\"Huh, why should the repository data and branch tips be separated\nout into multiple independent copies for the same module?  Do we\nforce users to synchronise between these copies?  It does not make\nany sense at all.\"\n\nBut that was until I read your message ;-) You are right that the\nindex and HEAD are dependent to a particular working tree that is\nchecked out.  There may be other things that logically are per-\nworking tree.\n\nAnd multiple-worktree _is_ about keeping the same repository and\nhistory data (i.e. object database, refs, rerere database, reflogs\nfor refs/*) only once, while allowing multiple working trees attached\nto that single copy.\n\nSo it appears to me that to create a checkout-to copy of a\nsuperproject with a submodule, a checkout-to copy of the\nsuperproject would have a submodule, which is a checkout-to copy of\nthe submodule in the superproject.\n\n> And I didn't try to wrap my head around recursive submodules yet ...\n>\n> Until that problem is solved it looks wrong to pass GIT_COMMON_DIR into\n> submodule recursion, I believe GIT_COMMON_DIR should be added to the\n> local_repo_env array (and even if it is passed on later, we might have\n> to append \"/modules/<submodule_name>\" to make it point to the correct\n> location).\n"},{"id":"250620","messageId":"20141014183431.GA8157@wheezy.local","threadId":"37720","inReplyTo":"xmqqoatezhnx.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes","fromName":"Max Kirillov","fromEmail":"max@max630.net","sentAt":"2014-10-14T18:34:31Z","receivedAt":"2014-10-14T18:34:31Z","isPatch":true,"sender":{"key":"max@max630.net","avatar":"https://avatars.githubusercontent.com/u/381560?v=4"},"body":"On Tue, Oct 14, 2014 at 10:26:42AM -0700, Junio C Hamano wrote:\n> And multiple-worktree _is_ about keeping the same repository and\n> history data (i.e. object database, refs, rerere database, reflogs for\n> refs/*) only once, while allowing multiple working trees attached to\n> that single copy.\n> \n> So it appears to me that to create a checkout-to copy of a\n> superproject with a submodule, a checkout-to copy of the superproject\n> would have a submodule, which is a checkout-to copy of the submodule\n> in the superproject.\n\nThat's right, this linking should be more implicit.\n\nBut here are a lot of nuances. For example, it makes sense to have a\nsuperproject checkout without submodules being initialized (so that they\ndon't waste space and machine time for working tree, which often is more\nthan repository data). And it may happen so that this checkout is the\nmaster repository for superproject checkouts. But this should not\nprevent users from using initialized submodules in other checkouts.\n\nThen, a checkout copy of a submodule can be standalone (for example, git\nand git-html-docs are submodules of msysgit). Or, it can even belong to\nsome other superproject. And in that cases they still should be able to\nbe linked.\n\nConsidering all above, and also the thing that I am quite new to\nsubmodules (but have to use them currently), I did not intend to create\nany new UI, only to make backend handle the already existing linked\ncheckouts, which can be made manually.\n\n-- \nMax\n"},{"id":"250625","messageId":"543D7EBA.4040206@web.de","threadId":"37720","inReplyTo":"20141014183431.GA8157@wheezy.local","subject":"Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2014-10-14T19:51:22Z","receivedAt":"2014-10-14T19:51:22Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 14.10.2014 um 20:34 schrieb Max Kirillov:\n> On Tue, Oct 14, 2014 at 10:26:42AM -0700, Junio C Hamano wrote:\n>> And multiple-worktree _is_ about keeping the same repository and\n>> history data (i.e. object database, refs, rerere database, reflogs for\n>> refs/*) only once, while allowing multiple working trees attached to\n>> that single copy.\n>>\n>> So it appears to me that to create a checkout-to copy of a\n>> superproject with a submodule, a checkout-to copy of the superproject\n>> would have a submodule, which is a checkout-to copy of the submodule\n>> in the superproject.\n>\n> That's right, this linking should be more implicit.\n\nYep. And for the submodule of a submodule too ... ;-)\n\n> But here are a lot of nuances. For example, it makes sense to have a\n> superproject checkout without submodules being initialized (so that they\n> don't waste space and machine time for working tree, which often is more\n> than repository data).\n\nHmm, I'm not sure if this is a problem. If the GIT_COMMON_DIR does have\nthe submodule repo but it isn't initialized locally, we shouldn't have a\nproblem (except for wasting some disk space if not a single checkout-to\nsuperproject initializes this submodule). And if GIT_COMMON_DIR does not\nhave the submodule repo yet, wouldn't it be cloned the moment we init\nthe submodule in the checkout-to? Or would that need extra functionality?\n\n > And it may happen so that this checkout is the\n> master repository for superproject checkouts. But this should not\n> prevent users from using initialized submodules in other checkouts.\n\nI could live with the restriction that submodule's GIT_COMMON_DIRs always\nlive in their checkout-to superproject's GIT_COMMON_DIR. This would still\nbe an improvement for CI servers that have multiple clones of a super-\nproject, as they would all share their submodule common dirs at least\nper superproject.\n\n> Then, a checkout copy of a submodule can be standalone (for example, git\n> and git-html-docs are submodules of msysgit). Or, it can even belong to\n> some other superproject. And in that cases they still should be able to\n> be linked.\n\nMaybe such configurations would have to be handled manually to achieve\nmaximum savings. At least I could live with that.\n\n> Considering all above, and also the thing that I am quite new to\n> submodules (but have to use them currently), I did not intend to create\n> any new UI, only to make backend handle the already existing linked\n> checkouts, which can be made manually.\n\nMaybe the way to go is to restrict GIT_COMMON_DIR to superprojects and\nhave a distinct GIT_COMMON_MODULES_DIR? We could even set that to the\nsame location for all submodules of different superprojects to achieve\nminimum disk footprint (assuming they all have different names across\nall superprojects and recursion levels).\n\nHmm, so I tend towards adding GIT_COMMON_DIR to local_repo_env until\nwe figured out how to handle this. Without that I fear bad things will\nhappen, at least for a superproject with multiple checkout-to work trees\nwhere the same submodule is initialized more than once ...\n"},{"id":"250627","messageId":"20141014203114.GB8157@wheezy.local","threadId":"37720","inReplyTo":"543D58D9.5060606@web.de","subject":"Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes","fromName":"Max Kirillov","fromEmail":"max@max630.net","sentAt":"2014-10-14T20:31:14Z","receivedAt":"2014-10-14T20:31:14Z","isPatch":true,"sender":{"key":"max@max630.net","avatar":"https://avatars.githubusercontent.com/u/381560?v=4"},"body":"On Tue, Oct 14, 2014 at 07:09:45PM +0200, Jens Lehmann wrote:\n> Until that problem is solved it looks wrong to pass\n> GIT_COMMON_DIR into submodule recursion, I believe\n> GIT_COMMON_DIR should be added to the local_repo_env array\n> (and even if it is passed on later, we might have to\n> append \"/modules/<submodule_name>\" to make it point to the\n> correct location).\n \nActually, why there should be an _environment_ variable\nGIT_COMMON_DIR at all? I mean, gitdir is resolved to some\ndirectory (through link or environment), and it contains the\nshared data directly or referes to it with the commondir\nlink. In which case anyone would want to override that\nlocation?\n\nI searched though tests but they don't cover this.\n"},{"id":"250630","messageId":"20141014221509.GA10580@wheezy.local","threadId":"37720","inReplyTo":"543D7EBA.4040206@web.de","subject":"Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes","fromName":"Max Kirillov","fromEmail":"max@max630.net","sentAt":"2014-10-14T22:15:09Z","receivedAt":"2014-10-14T22:15:09Z","isPatch":true,"sender":{"key":"max@max630.net","avatar":"https://avatars.githubusercontent.com/u/381560?v=4"},"body":"On Tue, Oct 14, 2014 at 09:51:22PM +0200, Jens Lehmann wrote:\n> Am 14.10.2014 um 20:34 schrieb Max Kirillov:\n>> But here are a lot of nuances. For example, it makes\n>> sense to have a superproject checkout without submodules\n>> being initialized (so that they don't waste space and\n>> machine time for working tree, which often is more than\n>> repository data).\n> \n> Hmm, I'm not sure if this is a problem. If the\n> GIT_COMMON_DIR does have the submodule repo but it isn't\n> initialized locally, we shouldn't have a problem (except\n> for wasting some disk space if not a single checkout-to\n> superproject initializes this submodule).\n\nIf initially a repository is clone without submodules, it\nwill not have anything in the GIT_COMMON_DIR.\n\n> And if GIT_COMMON_DIR does not have the submodule repo\n> yet, wouldn't it be cloned the moment we init the\n> submodule in the checkout-to? Or would that need extra\n> functionality?\n\nI cannot say I like this. Network operations should be\ncaused only by clone and submodules.\n\nI think the logic can be simple: it a submodule is not\nchecked-out in the repository \"checkout --to\" is called\nfrom, then it is not checked-out to the new one also. If it\nis, then checkout calls itself recursively in the submodule\nand works like being run in standalone repository.\n\n>> Then, a checkout copy of a submodule can be standalone\n>> (for example, git and git-html-docs are submodules of\n>> msysgit). Or, it can even belong to some other\n>> superproject. And in that cases they still should be able\n>> to be linked.\n> \n> Maybe such configurations would have to be handled\n> manually to achieve maximum savings. At least I could live\n> with that.\n\nTo make manual handling of the cases, and to skip\nchecking-out a module.\n\nI would think about the following interface:\n\n$ git checkout --to ... - does not checkout submodules,\ncreates empty directory.\n\n$ git checkout --recursive --to ... - if a submodule is\nchecked-out in source repository, recursed there and run\n\"checkout --recursive\" again. If a submodule is not\nchecked-out, does not checkout it, creates an empty\ndirectory.\n\nBy the way, I have found your branch\nrecursive_submodule_checkout. Would you like to revive it?\nThen it can be done with the same option.\n\n> Hmm, so I tend towards adding GIT_COMMON_DIR to\n> local_repo_env until we figured out how to handle this.\n> Without that I fear bad things will happen, at least for a\n> superproject with multiple checkout-to work trees where\n> the same submodule is initialized more than once ...\n\nI learned about local_repo_env and agree it should include\nGIT_COMMON_DIR. Unless it is removed at all...\n"},{"id":"250676","messageId":"CACsJy8AmBr2YTJkVw4BDD95RVE91EEBtEyakOpb77NDXaVBzJA@mail.gmail.com","threadId":"37720","inReplyTo":"20141014203114.GB8157@wheezy.local","subject":"Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-10-15T13:08:33Z","receivedAt":"2014-10-15T13:08:33Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Oct 15, 2014 at 3:31 AM, Max Kirillov <max@max630.net> wrote:\n> On Tue, Oct 14, 2014 at 07:09:45PM +0200, Jens Lehmann wrote:\n>> Until that problem is solved it looks wrong to pass\n>> GIT_COMMON_DIR into submodule recursion, I believe\n>> GIT_COMMON_DIR should be added to the local_repo_env array\n>> (and even if it is passed on later, we might have to\n>> append \"/modules/<submodule_name>\" to make it point to the\n>> correct location).\n>\n> Actually, why there should be an _environment_ variable\n> GIT_COMMON_DIR at all? I mean, gitdir is resolved to some\n> directory (through link or environment), and it contains the\n> shared data directly or referes to it with the commondir\n> link. In which case anyone would want to override that\n> location?\n\nIt's how the implementation was built up. First I split the repo in\ntwo and I need something to point the new/shared part.\n$GIT_DIR/worktrees comes later to glue them up. Keeping it an\nenvironment variable gives us a possibility to glue things up in a\ndifferent way than using $GIT_DIR/worktrees. Of course the odds of\nanybody doing that are probably small or even near zero.\n\nStill consuming the rest of this thread. Thanks all for working on the\nsubmodule support for this.\n-- \nDuy\n"},{"id":"250677","messageId":"CACsJy8DEY=Dh7ErG7wN-w-v9pu8Ms0FZNX06cL0dkxpJTZefWQ@mail.gmail.com","threadId":"37720","inReplyTo":"20141014221509.GA10580@wheezy.local","subject":"Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-10-15T14:14:10Z","receivedAt":"2014-10-15T14:14:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Oct 15, 2014 at 5:15 AM, Max Kirillov <max@max630.net> wrote:\n>> Hmm, so I tend towards adding GIT_COMMON_DIR to\n>> local_repo_env until we figured out how to handle this.\n>> Without that I fear bad things will happen, at least for a\n>> superproject with multiple checkout-to work trees where\n>> the same submodule is initialized more than once ...\n>\n> I learned about local_repo_env and agree it should include\n> GIT_COMMON_DIR. Unless it is removed at all...\n\nIt's in the same class as GIT_DIR and GIT_WORK_TREE so yeah it should\nbe in local_repo_env. Removing it would break t0060-path-utils.sh at\nleast. Unless we have a very good reason to remove it, I think we\nshould keep it as is.\n-- \nDuy\n"},{"id":"250691","messageId":"xmqq1tq9xnrs.fsf@gitster.dls.corp.google.com","threadId":"37720","inReplyTo":"CACsJy8AmBr2YTJkVw4BDD95RVE91EEBtEyakOpb77NDXaVBzJA@mail.gmail.com","subject":"Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-10-15T17:09:59Z","receivedAt":"2014-10-15T17:09:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen <pclouds@gmail.com> writes:\n\n> On Wed, Oct 15, 2014 at 3:31 AM, Max Kirillov <max@max630.net> wrote:\n>> On Tue, Oct 14, 2014 at 07:09:45PM +0200, Jens Lehmann wrote:\n>>> Until that problem is solved it looks wrong to pass\n>>> GIT_COMMON_DIR into submodule recursion, I believe\n>>> GIT_COMMON_DIR should be added to the local_repo_env array\n>>> (and even if it is passed on later, we might have to\n>>> append \"/modules/<submodule_name>\" to make it point to the\n>>> correct location).\n>>\n>> Actually, why there should be an _environment_ variable\n>> GIT_COMMON_DIR at all? I mean, gitdir is resolved to some\n>> directory (through link or environment), and it contains the\n>> shared data directly or referes to it with the commondir\n>> link. In which case anyone would want to override that\n>> location?\n>\n> It's how the implementation was built up. First I split the repo in\n> two and I need something to point the new/shared part.\n> $GIT_DIR/worktrees comes later to glue them up. Keeping it an\n> environment variable gives us a possibility to glue things up in a\n> different way than using $GIT_DIR/worktrees. Of course the odds of\n> anybody doing that are probably small or even near zero.\n>\n> Still consuming the rest of this thread. Thanks all for working on the\n> submodule support for this.\n\nHmph.  I was hoping that the multiple-work-trees topic was ready for\n'next' by now, but we may want to wait to see how the interaction\nwith submodule plays out to have another chance of a clean reroll\nbefore it happens.  This is a topic with large impact and is quite\nintrusive code-wise, even though the intrusive parts are cleanly\ndone.  So we'd want to take more time to unleash it to the general\npublic than more usual small scale topics, anyway.\n\nThanks.\n"},{"id":"250696","messageId":"543EC390.4000709@web.de","threadId":"37720","inReplyTo":"20141014221509.GA10580@wheezy.local","subject":"Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2014-10-15T18:57:20Z","receivedAt":"2014-10-15T18:57:20Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 15.10.2014 um 00:15 schrieb Max Kirillov:\n> On Tue, Oct 14, 2014 at 09:51:22PM +0200, Jens Lehmann wrote:\n>> Am 14.10.2014 um 20:34 schrieb Max Kirillov:\n>>> But here are a lot of nuances. For example, it makes\n>>> sense to have a superproject checkout without submodules\n>>> being initialized (so that they don't waste space and\n>>> machine time for working tree, which often is more than\n>>> repository data).\n>>\n>> Hmm, I'm not sure if this is a problem. If the\n>> GIT_COMMON_DIR does have the submodule repo but it isn't\n>> initialized locally, we shouldn't have a problem (except\n>> for wasting some disk space if not a single checkout-to\n>> superproject initializes this submodule).\n>\n> If initially a repository is clone without submodules, it\n> will not have anything in the GIT_COMMON_DIR.\n\nOk.\n\n>> And if GIT_COMMON_DIR does not have the submodule repo\n>> yet, wouldn't it be cloned the moment we init the\n>> submodule in the checkout-to? Or would that need extra\n>> functionality?\n>\n> I cannot say I like this. Network operations should be\n> caused only by clone and submodules.\n\nSure (and please add fetch to the list ;-). Maybe I confused\nyou by saying \"init\" when I meant the \"submodule update\" run\nafter initializing the submodule?\n\n> I think the logic can be simple: it a submodule is not\n> checked-out in the repository \"checkout --to\" is called\n> from, then it is not checked-out to the new one also. If it\n> is, then checkout calls itself recursively in the submodule\n> and works like being run in standalone repository.\n\nBut when I later decide to populate the submodule in a\n\"checkout --to\" work tree, should it automagically also\nuse the central storage, creating the modules/<name>\ndirectory there if it doesn't exist yet? I think that'd\nmake sense to avoid having the work tree layout depend\non the order commands were ran in. And imagine new\nsubmodules, they should not be handled differently from\nthose already present.\n\n>>> Then, a checkout copy of a submodule can be standalone\n>>> (for example, git and git-html-docs are submodules of\n>>> msysgit). Or, it can even belong to some other\n>>> superproject. And in that cases they still should be able\n>>> to be linked.\n>>\n>> Maybe such configurations would have to be handled\n>> manually to achieve maximum savings. At least I could live\n>> with that.\n>\n> To make manual handling of the cases, and to skip\n> checking-out a module.\n>\n> I would think about the following interface:\n>\n> $ git checkout --to ... - does not checkout submodules,\n> creates empty directory.\n\nThis is what checkout should always do (at least until it\nlearns --recurse-submodules, then it would populate the\nsubmodule directories).\n\n> $ git checkout --recursive --to ... - if a submodule is\n> checked-out in source repository, recursed there and run\n> \"checkout --recursive\" again. If a submodule is not\n> checked-out, does not checkout it, creates an empty\n> directory.\n\nHmm, I believe that when the user requests recursion\nexplicitly it should always be checked out, no matter in\nwhat state the GIT_COMMON_DIR is in. Otherwise we'll see\nproblems when a new submodule shows up and is populated\ndepending on the point in time the \"checkout --to\" was\ndone ... not good.\n\n> By the way, I have found your branch\n> recursive_submodule_checkout. Would you like to revive it?\n> Then it can be done with the same option.\n\nNo need to revive it, I'm currently working on that branch\nwhenever I find some time (but I'll still need some time\nbefore I can post the next iteration).\n"},{"id":"250772","messageId":"20141016205453.GA8441@wheezy.local","threadId":"37720","inReplyTo":"543EC390.4000709@web.de","subject":"Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes","fromName":"Max Kirillov","fromEmail":"max@max630.net","sentAt":"2014-10-16T20:54:53Z","receivedAt":"2014-10-16T20:54:53Z","isPatch":true,"sender":{"key":"max@max630.net","avatar":"https://avatars.githubusercontent.com/u/381560?v=4"},"body":"On Wed, Oct 15, 2014 at 08:57:20PM +0200, Jens Lehmann wrote:\n> Am 15.10.2014 um 00:15 schrieb Max Kirillov:\n>> I think the logic can be simple: it a submodule is not\n>> checked-out in the repository \"checkout --to\" is called\n>> from, then it is not checked-out to the new one also. If it\n>> is, then checkout calls itself recursively in the submodule\n>> and works like being run in standalone repository.\n> \n> But when I later decide to populate the submodule in a\n> \"checkout --to\" work tree, should it automagically also\n> use the central storage, creating the modules/<name>\n> directory there if it doesn't exist yet? I think that'd\n> make sense to avoid having the work tree layout depend\n> on the order commands were ran in. And imagine new\n> submodules, they should not be handled differently from\n> those already present.\n\nLike place the common directory to\n$MAIN_REPO/.git/modules/$SUB/ and worktree-specific part to\n$MAIN_REPO/.git/worktrees/$WORKTREE/modules/$SUB, rather\nthan placing all into the socond one? It would make sense to\nmake, but then it would be imposible to checkout a diferent\nrepository into the same submodule in different superproject\ncheckouts. However stupid is sounds, there could be cases\nif, for example, at some moment submodule is being replaced\nby another one, and older worktrees should work with older\nsubmodule, while newer uses the newer submodule.\n\nMaybe, there could be some options to tell the command which\npopulates submodules (which commands that are? \"submodule update\"\nand other submodule subcommands? or there is something\nelse?) to use the curent checkout space or the main one. But\nI would still leave it depend on what user explicitly calls\nand where the initial submodule update is executed.\n\nAlso, could you clarify the usage of the /modules/\ndirectory. I did not notice it to affect anything after the\nsubmofule is placed there. Submodule operations use the\nsubmodule repositories directly (through the git link, which\ncan point anywhere), or in .gitmodules file, or maybe in\n.git/config. So there is actually no need to have that\ngitdir there. Is it correct?\n\n-- \nMax\n"},{"id":"250795","messageId":"CACsJy8ABm93QTZm8wnJUYaJ6zC2bXBDyois+Wrq-vCB=DCJmVQ@mail.gmail.com","threadId":"37720","inReplyTo":"xmqq1tq9xnrs.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-10-17T09:14:19Z","receivedAt":"2014-10-17T09:14:19Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Oct 16, 2014 at 12:09 AM, Junio C Hamano <gitster@pobox.com> wrote:\n> Hmph.  I was hoping that the multiple-work-trees topic was ready for\n> 'next' by now, but we may want to wait to see how the interaction\n> with submodule plays out to have another chance of a clean reroll\n> before it happens.  This is a topic with large impact and is quite\n> intrusive code-wise, even though the intrusive parts are cleanly\n> done.  So we'd want to take more time to unleash it to the general\n> public than more usual small scale topics, anyway.\n\nOriginally I wanted to get it merged without submodule support, but I\nfailed to spot the local_repo_env problem and could have caused a\nregression for submodules. So yeah delaying the series does not sound\nbad. Not sure about the reroll (i.e. rewriting current patches). I\nthink putting patches on top with explanation is better. But we can\nkeep it in 'pu' and see if we really need to reroll.\n-- \nDuy\n"},{"id":"250832","messageId":"54441147.5080204@web.de","threadId":"37720","inReplyTo":"20141016205453.GA8441@wheezy.local","subject":"Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2014-10-19T19:30:15Z","receivedAt":"2014-10-19T19:30:15Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 16.10.2014 um 22:54 schrieb Max Kirillov:\n> On Wed, Oct 15, 2014 at 08:57:20PM +0200, Jens Lehmann wrote:\n>> Am 15.10.2014 um 00:15 schrieb Max Kirillov:\n>>> I think the logic can be simple: it a submodule is not\n>>> checked-out in the repository \"checkout --to\" is called\n>>> from, then it is not checked-out to the new one also. If it\n>>> is, then checkout calls itself recursively in the submodule\n>>> and works like being run in standalone repository.\n>>\n>> But when I later decide to populate the submodule in a\n>> \"checkout --to\" work tree, should it automagically also\n>> use the central storage, creating the modules/<name>\n>> directory there if it doesn't exist yet? I think that'd\n>> make sense to avoid having the work tree layout depend\n>> on the order commands were ran in. And imagine new\n>> submodules, they should not be handled differently from\n>> those already present.\n>\n> Like place the common directory to\n> $MAIN_REPO/.git/modules/$SUB/ and worktree-specific part to\n> $MAIN_REPO/.git/worktrees/$WORKTREE/modules/$SUB, rather\n> than placing all into the socond one? It would make sense to\n> make, but then it would be imposible to checkout a diferent\n> repository into the same submodule in different superproject\n> checkouts. However stupid is sounds, there could be cases\n> if, for example, at some moment submodule is being replaced\n> by another one, and older worktrees should work with older\n> submodule, while newer uses the newer submodule.\n\nYes, but I believe that the user must be careful to not\nreuse the same submodule name for a different repo anyways,\nno matter if shared or not. Currently you'll get a warning\nabout that when trying to add a submodule whose name is\nalready found in .git/modules to avoid such confusion.\n\n> Maybe, there could be some options to tell the command which\n> populates submodules (which commands that are? \"submodule update\"\n> and other submodule subcommands? or there is something\n> else?) to use the curent checkout space or the main one. But\n> I would still leave it depend on what user explicitly calls\n> and where the initial submodule update is executed.\n\nCurrently only \"submodule update\" populates submodules, but\nI'm currently working hard on teaching commands like checkout\n(and lots of others) to do the same. I agree that the user\nshould be able to choose and for our CI server I would also\nlike to see a solution that could share submodules across\ndifferent superprojects. So having another environment\nvariable to decide where to put the work tree independent\nparts of the .git/modules directory might make sense here.\n\n> Also, could you clarify the usage of the /modules/\n> directory. I did not notice it to affect anything after the\n> submofule is placed there. Submodule operations use the\n> submodule repositories directly (through the git link, which\n> can point anywhere), or in .gitmodules file, or maybe in\n> .git/config. So there is actually no need to have that\n> gitdir there. Is it correct?\n\nNope. When submodules are cloned their git directory is\nplaced under .git/modules/<submodule name>, the .git file\nin the work tree points there and the core.worktree setting\npoints back from there to the work tree.\n"},{"id":"250833","messageId":"54441249.9030206@web.de","threadId":"37720","inReplyTo":"CACsJy8ABm93QTZm8wnJUYaJ6zC2bXBDyois+Wrq-vCB=DCJmVQ@mail.gmail.com","subject":"Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2014-10-19T19:34:33Z","receivedAt":"2014-10-19T19:34:33Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 17.10.2014 um 11:14 schrieb Duy Nguyen:\n> On Thu, Oct 16, 2014 at 12:09 AM, Junio C Hamano <gitster@pobox.com> wrote:\n>> Hmph.  I was hoping that the multiple-work-trees topic was ready for\n>> 'next' by now, but we may want to wait to see how the interaction\n>> with submodule plays out to have another chance of a clean reroll\n>> before it happens.  This is a topic with large impact and is quite\n>> intrusive code-wise, even though the intrusive parts are cleanly\n>> done.  So we'd want to take more time to unleash it to the general\n>> public than more usual small scale topics, anyway.\n>\n> Originally I wanted to get it merged without submodule support, but I\n> failed to spot the local_repo_env problem and could have caused a\n> regression for submodules. So yeah delaying the series does not sound\n> bad. Not sure about the reroll (i.e. rewriting current patches). I\n> think putting patches on top with explanation is better. But we can\n> keep it in 'pu' and see if we really need to reroll.\n\nI didn't look into your series in detail, but it looks to my like\nexcepting the .git/modules directory from sharing (by putting it\ninto local_repo_env array) and adding a test for that (just to be\nsure that no modules directory shows up where it shouldn't) should\nbe sufficient to get your stuff merged without submodule support.\nIt might be better to handle submodule support in a follow up series.\n\nDoes that make sense?\n"},{"id":"250844","messageId":"20141020041109.GA5784@wheezy.local","threadId":"37720","inReplyTo":"54441147.5080204@web.de","subject":"Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes","fromName":"Max Kirillov","fromEmail":"max@max630.net","sentAt":"2014-10-20T04:11:09Z","receivedAt":"2014-10-20T04:11:09Z","isPatch":true,"sender":{"key":"max@max630.net","avatar":"https://avatars.githubusercontent.com/u/381560?v=4"},"body":"On Sun, Oct 19, 2014 at 09:30:15PM +0200, Jens Lehmann wrote:\n> Am 16.10.2014 um 22:54 schrieb Max Kirillov:\n>> On Wed, Oct 15, 2014 at 08:57:20PM +0200, Jens Lehmann wrote:\n>>> Am 15.10.2014 um 00:15 schrieb Max Kirillov:\n>>>> I think the logic can be simple: it a submodule is not\n>>>> checked-out in the repository \"checkout --to\" is called\n>>>> from, then it is not checked-out to the new one also. If it\n>>>> is, then checkout calls itself recursively in the submodule\n>>>> and works like being run in standalone repository.\n\n>>> But when I later decide to populate the submodule in a\n>>> \"checkout --to\" work tree, should it automagically also\n>>> use the central storage, creating the modules/<name>\n>>> directory there if it doesn't exist yet? I think that'd\n>>> make sense to avoid having the work tree layout depend\n>>> on the order commands were ran in. And imagine new\n>>> submodules, they should not be handled differently from\n>>> those already present.\n\n>> Like place the common directory to\n>> $MAIN_REPO/.git/modules/$SUB/ and worktree-specific part to\n>> $MAIN_REPO/.git/worktrees/$WORKTREE/modules/$SUB, rather\n>> than placing all into the socond one? It would make sense to\n>> make, but then it would be imposible to checkout a diferent\n>> repository into the same submodule in different superproject\n>> checkouts. However stupid is sounds, there could be cases\n>> if, for example, at some moment submodule is being replaced\n>> by another one, and older worktrees should work with older\n>> submodule, while newer uses the newer submodule.\n\n> Yes, but I believe that the user must be careful to not\n> reuse the same submodule name for a different repo anyways,\n> no matter if shared or not. Currently you'll get a warning\n> about that when trying to add a submodule whose name is\n> already found in .git/modules to avoid such confusion.\n\nYes, while trying to write tests for this case I discovered\nthat there are warnings and the recommended way is to use\ndifferent names for different repositories even if they are\npointing to the same path. Then always placing common\ndirectory into the .git/modules/<module> could be a good\nidea, and in very special cases users could manually create\nrepositories with custom placement.\n\n>> Also, could you clarify the usage of the /modules/\n>> directory. I did not notice it to affect anything after the\n>> submofule is placed there. Submodule operations use the\n>> submodule repositories directly (through the git link, which\n>> can point anywhere), or in .gitmodules file, or maybe in\n>> .git/config. So there is actually no need to have that\n>> gitdir there. Is it correct?\n\n> Nope. When submodules are cloned their git directory is\n> placed under .git/modules/<submodule name>, the .git file\n> in the work tree points there and the core.worktree setting\n> points back from there to the work tree.\n\nI meant is the fact that gitdir is placed in modules, rather\nthan in any other place, is used anywhere. There are 2\nplaces to put the gitdir of submodule in linked copy:\n1. $MAIN_REPO/.git/worktrees/$WORKTREE/modules/$SUB\n2. $MAIN_REPO/.git/modules/$SUB/worktrees/$SUB_WTNAME\nFirst one is suggested by submodule way of placing gitdirs,\nand the second one by worrktree way. There are reasons to\nhave the second one - garbage collection and check that 2\nbranch is not checked out twice. Are there resons to have\nthe 1st one? The one is to prevent use of different\nrepositories with the same name, anything else?\n\n-- \nMax\n"},{"id":"251299","messageId":"CACsJy8C34PyK4rPQC_wFgms=gVCs2FN_5aUSMfzJawErZHHwFg@mail.gmail.com","threadId":"37720","inReplyTo":"20141020041109.GA5784@wheezy.local","subject":"Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-11-03T12:54:39Z","receivedAt":"2014-11-03T12:54:39Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Ping.. any idea how to go from here..\n\nOn Mon, Oct 20, 2014 at 11:11 AM, Max Kirillov <max@max630.net> wrote:\n> On Sun, Oct 19, 2014 at 09:30:15PM +0200, Jens Lehmann wrote:\n>> Am 16.10.2014 um 22:54 schrieb Max Kirillov:\n>>> On Wed, Oct 15, 2014 at 08:57:20PM +0200, Jens Lehmann wrote:\n>>>> Am 15.10.2014 um 00:15 schrieb Max Kirillov:\n>>>>> I think the logic can be simple: it a submodule is not\n>>>>> checked-out in the repository \"checkout --to\" is called\n>>>>> from, then it is not checked-out to the new one also. If it\n>>>>> is, then checkout calls itself recursively in the submodule\n>>>>> and works like being run in standalone repository.\n>\n>>>> But when I later decide to populate the submodule in a\n>>>> \"checkout --to\" work tree, should it automagically also\n>>>> use the central storage, creating the modules/<name>\n>>>> directory there if it doesn't exist yet? I think that'd\n>>>> make sense to avoid having the work tree layout depend\n>>>> on the order commands were ran in. And imagine new\n>>>> submodules, they should not be handled differently from\n>>>> those already present.\n>\n>>> Like place the common directory to\n>>> $MAIN_REPO/.git/modules/$SUB/ and worktree-specific part to\n>>> $MAIN_REPO/.git/worktrees/$WORKTREE/modules/$SUB, rather\n>>> than placing all into the socond one? It would make sense to\n>>> make, but then it would be imposible to checkout a diferent\n>>> repository into the same submodule in different superproject\n>>> checkouts. However stupid is sounds, there could be cases\n>>> if, for example, at some moment submodule is being replaced\n>>> by another one, and older worktrees should work with older\n>>> submodule, while newer uses the newer submodule.\n>\n>> Yes, but I believe that the user must be careful to not\n>> reuse the same submodule name for a different repo anyways,\n>> no matter if shared or not. Currently you'll get a warning\n>> about that when trying to add a submodule whose name is\n>> already found in .git/modules to avoid such confusion.\n>\n> Yes, while trying to write tests for this case I discovered\n> that there are warnings and the recommended way is to use\n> different names for different repositories even if they are\n> pointing to the same path. Then always placing common\n> directory into the .git/modules/<module> could be a good\n> idea, and in very special cases users could manually create\n> repositories with custom placement.\n>\n>>> Also, could you clarify the usage of the /modules/\n>>> directory. I did not notice it to affect anything after the\n>>> submofule is placed there. Submodule operations use the\n>>> submodule repositories directly (through the git link, which\n>>> can point anywhere), or in .gitmodules file, or maybe in\n>>> .git/config. So there is actually no need to have that\n>>> gitdir there. Is it correct?\n>\n>> Nope. When submodules are cloned their git directory is\n>> placed under .git/modules/<submodule name>, the .git file\n>> in the work tree points there and the core.worktree setting\n>> points back from there to the work tree.\n>\n> I meant is the fact that gitdir is placed in modules, rather\n> than in any other place, is used anywhere. There are 2\n> places to put the gitdir of submodule in linked copy:\n> 1. $MAIN_REPO/.git/worktrees/$WORKTREE/modules/$SUB\n> 2. $MAIN_REPO/.git/modules/$SUB/worktrees/$SUB_WTNAME\n> First one is suggested by submodule way of placing gitdirs,\n> and the second one by worrktree way. There are reasons to\n> have the second one - garbage collection and check that 2\n> branch is not checked out twice. Are there resons to have\n> the 1st one? The one is to prevent use of different\n> repositories with the same name, anything else?\n>\n> --\n> Max\n\n\n\n-- \nDuy\n"},{"id":"251358","messageId":"5457EC29.8040005@web.de","threadId":"37720","inReplyTo":"CACsJy8C34PyK4rPQC_wFgms=gVCs2FN_5aUSMfzJawErZHHwFg@mail.gmail.com","subject":"Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2014-11-03T20:57:13Z","receivedAt":"2014-11-03T20:57:13Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 03.11.2014 um 13:54 schrieb Duy Nguyen:\n> Ping.. any idea how to go from here..\n\nI didn't dig deep enough into the multiple worktrees topic to\nknow what \"$MAIN_REPO/.git/worktrees/$WORKTREE/modules/$SUB\"\nmight mean, but a submodule whose repo lives under\n.git/modules/$SUBMODULE_NAME should have its worktree in\n$SUBMODULE_PATH of the superproject's worktree.\n\nSo if we share submodule repos, the sharable stuff should\nlive under $MAIN_REPO/.git/modules/$SUBMODULE_NAME/. But\neverything that can't be put there should stay under the\n.git/modules/$SUBMODULE_NAME directory in every worktree.\nThe same redirection mechanism the superproject uses to\nredirect to $MAIN_REPO/.git should be used to redirect\nthe sharable stuff from .git/modules/$SUBMODULE_NAME to\n$MAIN_REPO/.git/modules/$SUBMODULE_NAME/.\n\nI believe this should be the default. Optionally we might\nwant to enable the user to put $MAIN_REPO/.git/modules\nsomeplace else (outside the superprojects $MAIN_REPO) to\nbe able to reuse the object store of submodules that are\nshared between different superprojects.\n\nDoes that make sense?\n\n> On Mon, Oct 20, 2014 at 11:11 AM, Max Kirillov <max@max630.net> wrote:\n>> On Sun, Oct 19, 2014 at 09:30:15PM +0200, Jens Lehmann wrote:\n>>> Am 16.10.2014 um 22:54 schrieb Max Kirillov:\n>>>> On Wed, Oct 15, 2014 at 08:57:20PM +0200, Jens Lehmann wrote:\n>>>>> Am 15.10.2014 um 00:15 schrieb Max Kirillov:\n>>>>>> I think the logic can be simple: it a submodule is not\n>>>>>> checked-out in the repository \"checkout --to\" is called\n>>>>>> from, then it is not checked-out to the new one also. If it\n>>>>>> is, then checkout calls itself recursively in the submodule\n>>>>>> and works like being run in standalone repository.\n>>\n>>>>> But when I later decide to populate the submodule in a\n>>>>> \"checkout --to\" work tree, should it automagically also\n>>>>> use the central storage, creating the modules/<name>\n>>>>> directory there if it doesn't exist yet? I think that'd\n>>>>> make sense to avoid having the work tree layout depend\n>>>>> on the order commands were ran in. And imagine new\n>>>>> submodules, they should not be handled differently from\n>>>>> those already present.\n>>\n>>>> Like place the common directory to\n>>>> $MAIN_REPO/.git/modules/$SUB/ and worktree-specific part to\n>>>> $MAIN_REPO/.git/worktrees/$WORKTREE/modules/$SUB, rather\n>>>> than placing all into the socond one? It would make sense to\n>>>> make, but then it would be imposible to checkout a diferent\n>>>> repository into the same submodule in different superproject\n>>>> checkouts. However stupid is sounds, there could be cases\n>>>> if, for example, at some moment submodule is being replaced\n>>>> by another one, and older worktrees should work with older\n>>>> submodule, while newer uses the newer submodule.\n>>\n>>> Yes, but I believe that the user must be careful to not\n>>> reuse the same submodule name for a different repo anyways,\n>>> no matter if shared or not. Currently you'll get a warning\n>>> about that when trying to add a submodule whose name is\n>>> already found in .git/modules to avoid such confusion.\n>>\n>> Yes, while trying to write tests for this case I discovered\n>> that there are warnings and the recommended way is to use\n>> different names for different repositories even if they are\n>> pointing to the same path. Then always placing common\n>> directory into the .git/modules/<module> could be a good\n>> idea, and in very special cases users could manually create\n>> repositories with custom placement.\n>>\n>>>> Also, could you clarify the usage of the /modules/\n>>>> directory. I did not notice it to affect anything after the\n>>>> submofule is placed there. Submodule operations use the\n>>>> submodule repositories directly (through the git link, which\n>>>> can point anywhere), or in .gitmodules file, or maybe in\n>>>> .git/config. So there is actually no need to have that\n>>>> gitdir there. Is it correct?\n>>\n>>> Nope. When submodules are cloned their git directory is\n>>> placed under .git/modules/<submodule name>, the .git file\n>>> in the work tree points there and the core.worktree setting\n>>> points back from there to the work tree.\n>>\n>> I meant is the fact that gitdir is placed in modules, rather\n>> than in any other place, is used anywhere. There are 2\n>> places to put the gitdir of submodule in linked copy:\n>> 1. $MAIN_REPO/.git/worktrees/$WORKTREE/modules/$SUB\n>> 2. $MAIN_REPO/.git/modules/$SUB/worktrees/$SUB_WTNAME\n>> First one is suggested by submodule way of placing gitdirs,\n>> and the second one by worrktree way. There are reasons to\n>> have the second one - garbage collection and check that 2\n>> branch is not checked out twice. Are there resons to have\n>> the 1st one? The one is to prevent use of different\n>> repositories with the same name, anything else?\n>>\n>> --\n>> Max\n>\n>\n>\n"},{"id":"251365","messageId":"20141103220702.GA23548@wheezy.local","threadId":"37720","inReplyTo":"CACsJy8C34PyK4rPQC_wFgms=gVCs2FN_5aUSMfzJawErZHHwFg@mail.gmail.com","subject":"Re: [PATCH 0/4] Multiple worktrees vs. submodules fixes","fromName":"Max Kirillov","fromEmail":"max@max630.net","sentAt":"2014-11-03T22:07:02Z","receivedAt":"2014-11-03T22:07:02Z","isPatch":true,"sender":{"key":"max@max630.net","avatar":"https://avatars.githubusercontent.com/u/381560?v=4"},"body":"On Mon, Nov 03, 2014 at 07:54:39PM +0700, Duy Nguyen wrote:\n> Ping.. any idea how to go from here..\n\nI'm sorry, I happen to have little time since the last\nconversation.\n\nAs far as I understand, my patches are correct about\nhandling existing submodules, but they may be not enough\nregarding _initialization_ of the submodules. There can be\ndesired changes to how 'git submodule update --init' behaves\nin the linked working trees. If others agree to leave it to\nsome later development, and continue now with the reading\npart, I would keep it all unchanged, except adding the\nGIT_COMMON_DIR to local_repo_env (or would you like to do\nit?) and couple of additional tests which I have drafted\nduring discussion. Otherwise I'll be trying but it's hard to\nmake any estimates.\n\nPS: the change of 'git submodule update --init' would be\nbehavior change, but git anyway must support non-linked\nsubmodules in different worktrees, so there should be no\ncompatibility break.\n\n-- \nMax\n"}]}