# [PATCH 0/2] git clone with --separate-git-dir destroys existing directory content 14 messages from 2020-07-01 to 2020-07-10. Participants: Ben Wijen, Eric Sunshine, Ben, Junio C Hamano. Thread: https://gitlist.dev/t/53781 ## Ben Wijen, 2020-07-01 09:36 Subject: [PATCH 0/2] git clone with --separate-git-dir destroys existing directory content Message-ID: <20200701093653.3706-1-ben@wijen.net> URL: https://gitlist.dev/e/20200701093653.3706-1-ben%40wijen.net ``` Hi, I found an issue with git clone, when using --separate-git-dir and that directory already exists, it's content is destroyed. I have created a patch set on origin/next, but (as I have learned this the hard way) please let me know if this needs to be backported... Ben... Ben Wijen (2): git clone: check for non-empty directory git clone: don't clone into non-empty directory builtin/clone.c | 12 ++++++++++-- t/t5601-clone.sh | 3 ++- 2 files changed, 12 insertions(+), 3 deletions(-) -- 2.27.0 ``` ## Ben Wijen, 2020-07-01 09:36 Subject: [PATCH 1/2] git clone: check for non-empty directory Message-ID: <20200701093653.3706-2-ben@wijen.net> URL: https://gitlist.dev/e/20200701093653.3706-2-ben%40wijen.net In-Reply-To: <20200701093653.3706-1-ben@wijen.net> ``` When using git clone with --separate-git-dir realgitdir and realgitdir already exists, it's content is destroyed. Extend test to make sure content is left intact Signed-off-by: Ben Wijen --- t/t5601-clone.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh index 84ea2a3eb7..03901c55f2 100755 --- a/t/t5601-clone.sh +++ b/t/t5601-clone.sh @@ -271,7 +271,8 @@ test_expect_success 'fetch from gitfile parent' ' test_expect_success 'clone separate gitdir where target already exists' ' rm -rf dst && - test_must_fail git clone --separate-git-dir realgitdir src dst + test_must_fail git clone --separate-git-dir realgitdir src dst && + test -f realgitdir/config ' test_expect_success 'clone --reference from original' ' -- 2.27.0 ``` ## Ben Wijen, 2020-07-01 09:36 Subject: [PATCH 2/2] git clone: don't clone into non-empty directory Message-ID: <20200701093653.3706-3-ben@wijen.net> URL: https://gitlist.dev/e/20200701093653.3706-3-ben%40wijen.net In-Reply-To: <20200701093653.3706-1-ben@wijen.net> ``` When using git clone with --separate-git-dir realgitdir and realgitdir already exists, it's content is destroyed. Make sure we don't clone into an existing non-empty directory. Signed-off-by: Ben Wijen --- builtin/clone.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index bef70745c0..54a91acf06 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -946,7 +946,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix) int is_bundle = 0, is_local; const char *repo_name, *repo, *work_tree, *git_dir; char *path, *dir, *display_repo = NULL; - int dest_exists; + int dest_exists, real_dest_exists = 0; const struct ref *refs, *remote_head; const struct ref *remote_head_points_at; const struct ref *our_head_points_at; @@ -1021,6 +1021,14 @@ int cmd_clone(int argc, const char **argv, const char *prefix) die(_("destination path '%s' already exists and is not " "an empty directory."), dir); + if (real_git_dir) { + real_dest_exists = path_exists(real_git_dir); + if (real_dest_exists && !is_empty_dir(real_git_dir)) + die(_("destination path '%s' already exists and is not " + "an empty directory."), real_git_dir); + } + + strbuf_addf(&reflog_msg, "clone: from %s", display_repo ? display_repo : repo); free(display_repo); @@ -1057,7 +1065,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix) } if (real_git_dir) { - if (path_exists(real_git_dir)) + if (real_dest_exists) junk_git_dir_flags |= REMOVE_DIR_KEEP_TOPLEVEL; junk_git_dir = real_git_dir; } else { -- 2.27.0 ``` ## Eric Sunshine, 2020-07-01 15:39 Subject: Re: [PATCH 0/2] git clone with --separate-git-dir destroys existing directory content Message-ID: URL: https://gitlist.dev/e/CAPig%2BcS6XnUK9MQL68_yoR9Gve4_itmawsd_0-yJr3S5%3D3-zPQ%40mail.gmail.com In-Reply-To: <20200701093653.3706-1-ben@wijen.net> ``` On Wed, Jul 1, 2020 at 5:46 AM Ben Wijen wrote: > I found an issue with git clone, when using --separate-git-dir and that directory already exists, > it's content is destroyed. Thanks for fixing this. > I have created a patch set on origin/next, but (as I have learned this the hard way) please let me know > if this needs to be backported... Does this patch series depend on certain topics in 'next'? If not, please base it on 'master' instead. This will allow it to graduate from 'seen' to 'next' and eventually 'master' without being held back by any other topics in flight. On the other hand, if it does depend upon some topic in 'next', then it is a good idea to specify which specific topics it needs. That way, your patches will be able to graduate after those specific topics graduate rather than having to wait until _all_ of 'next' graduates (which could take forever). ``` ## Eric Sunshine, 2020-07-01 16:00 Subject: Re: [PATCH 1/2] git clone: check for non-empty directory Message-ID: URL: https://gitlist.dev/e/CAPig%2BcT%2BcXuM3Asu6%2BZ25pGV2uRm6K1iAeYVhic9kqk1mS84-g%40mail.gmail.com In-Reply-To: <20200701093653.3706-2-ben@wijen.net> ``` On Wed, Jul 1, 2020 at 5:46 AM Ben Wijen wrote: > When using git clone with --separate-git-dir realgitdir and > realgitdir already exists, it's content is destroyed. > > Extend test to make sure content is left intact > > Signed-off-by: Ben Wijen > --- > diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh > @@ -271,7 +271,8 @@ test_expect_success 'fetch from gitfile parent' ' > test_expect_success 'clone separate gitdir where target already exists' ' > rm -rf dst && > - test_must_fail git clone --separate-git-dir realgitdir src dst > + test_must_fail git clone --separate-git-dir realgitdir src dst && > + test -f realgitdir/config > ' This addresses the immediate problem of the directory content being destroyed, which is good. But, we can future-proof it even more by also verifying (to some degree) that the existing content has not been disturbed. Doing so would give us greater confidence against some future change breaking "realgitdir" in some fashion other than merely emptying it. For instance, we might do this: rm -rf dst && echo foo=bar >realgitdir/config && test_must_fail git clone --separate-git-dir realgitdir src dst && grep foo=bar realgitdir/config One other observation: To preserve bisectability (git-bisect) of git.git itself, we want to ensure that the entire test suite still passes at each point in a patch series. Making this change to the test in its own patch introduces a failure into the test suite, which is undesirable. One way to address this shortcoming would be to temporarily change this test from 'test_must_succeed' to 'test_must_fail', and then flip it back to 'test_must_succeed' in patch 2/2. A cleaner approach is simply to combine the two patches so that the fix and updated test are bundled together (which makes sense anyhow in this case since this patch is so short and need not stand on its own). ``` ## Eric Sunshine, 2020-07-01 16:10 Subject: Re: [PATCH 2/2] git clone: don't clone into non-empty directory Message-ID: URL: https://gitlist.dev/e/CAPig%2BcTEpajuD98RH5xepJ%2BrK2f75SROrTvgjzKnyfjxbuA1AQ%40mail.gmail.com In-Reply-To: <20200701093653.3706-3-ben@wijen.net> ``` On Wed, Jul 1, 2020 at 5:46 AM Ben Wijen wrote: > When using git clone with --separate-git-dir realgitdir and > realgitdir already exists, it's content is destroyed. > > Make sure we don't clone into an existing non-empty directory. > > Signed-off-by: Ben Wijen > --- > diff --git a/builtin/clone.c b/builtin/clone.c > @@ -1021,6 +1021,14 @@ int cmd_clone(int argc, const char **argv, const char *prefix) > die(_("destination path '%s' already exists and is not " > "an empty directory."), dir); > > + if (real_git_dir) { > + real_dest_exists = path_exists(real_git_dir); > + if (real_dest_exists && !is_empty_dir(real_git_dir)) "git init --separate-git-dir=" checks only whether the path exists, and errors out if it does; it doesn't care whether the directory is empty or not. I'm wondering, therefore, if this check should be tightened to more closely align the behavior of the two commands. (Using the tighter semantic now doesn't prohibit loosening it in the future, whereas it's harder to tighten behavior which started out loose.) > + die(_("destination path '%s' already exists and is not " > + "an empty directory."), real_git_dir); This message might be a bit clearer if "destination" is changed to "repository". (And drop the bit about not being empty -- if we go that route.) ``` ## Eric Sunshine, 2020-07-01 17:40 Subject: Re: [PATCH 1/2] git clone: check for non-empty directory Message-ID: URL: https://gitlist.dev/e/CAPig%2BcRj6p4tdDhQOeoXpoqp%2BWb94rKaVim-Yddj_Gz13phQpg%40mail.gmail.com In-Reply-To: ``` On Wed, Jul 1, 2020 at 12:00 PM Eric Sunshine wrote: > [...] For instance, we might do this: > rm -rf dst && > echo foo=bar >realgitdir/config && I meant to use '>>' here, not '>'. ``` ## Eric Sunshine, 2020-07-02 05:32 Subject: Re: [PATCH 1/2] git clone: check for non-empty directory Message-ID: URL: https://gitlist.dev/e/CAPig%2BcQwJKSVMP8G7E62bESERYsAFTUN2PBX32YWMhECjU%3DqTQ%40mail.gmail.com In-Reply-To: ``` On Wed, Jul 1, 2020 at 12:00 PM Eric Sunshine wrote: > One other observation: To preserve bisectability (git-bisect) of > git.git itself, we want to ensure that the entire test suite still > passes at each point in a patch series. Making this change to the test > in its own patch introduces a failure into the test suite, which is > undesirable. One way to address this shortcoming would be to > temporarily change this test from 'test_must_succeed' to > 'test_must_fail', and then flip it back to 'test_must_succeed' in > patch 2/2. [...] And another correction: I meant 'test_expect_success' and 'test_expect_failure', respectively. ``` ## Ben, 2020-07-02 07:50 Subject: Re: [PATCH 2/2] git clone: don't clone into non-empty directory Message-ID: URL: https://gitlist.dev/e/fe053a22-16a1-a071-520c-9bb5be7dc09f%40wijen.net In-Reply-To: ``` On 01-07-2020 18:10, Eric Sunshine wrote: > On Wed, Jul 1, 2020 at 5:46 AM Ben Wijen wrote: > > "git init --separate-git-dir=" checks only whether the path exists, > and errors out if it does; it doesn't care whether the directory is > empty or not. I'm wondering, therefore, if this check should be > tightened to more closely align the behavior of the two commands. > (Using the tighter semantic now doesn't prohibit loosening it in the > future, whereas it's harder to tighten behavior which started out > loose.) > About `git init`, I did take a look at it, but saw 'exist_ok' is always true when called from 'cmd_init_db' (see builtin/init_db.c:654) This means `git init` always allows when git_dir/real_git_dir already exists, which I understand for the worktree, but I doubt if one wants this for the repository. (when --separate-git-dir is used) Also, because 'exist_ok' is always true for `git init` and - with this patch - `git clone` checks the git_dir/real_git_dir before the 'init_db' call I'm wondering if the code in 'init_db' (builtin/init-db.c:394-400) can be removed. Then, even if we do take that route (no 'is_empty_dir', only 'path_exists') we must also address junk_git_dir_flags, as REMOVE_DIR_KEEP_TOPLEVEL would invalidate a second git clone. So, as this patch only fixes the problem at hand, IMHO separate patch-sets should be created for mentioned issues. Ben... ``` ## Ben Wijen, 2020-07-02 08:13 Subject: [PATCH v2 0/1] git clone: don't clone into non-empty directory Message-ID: <20200702081335.16786-1-ben@wijen.net> URL: https://gitlist.dev/e/20200702081335.16786-1-ben%40wijen.net In-Reply-To: ``` Eric, thank you for your review. With this update I have: * rebased (and squashed) on master * included your proposals for the test * changed "destination path..." to "repository path..." Ben... Ben Wijen (1): git clone: don't clone into non-empty directory builtin/clone.c | 12 ++++++++++-- t/t5601-clone.sh | 4 +++- 2 files changed, 13 insertions(+), 3 deletions(-) -- 2.27.0 ``` ## Ben Wijen, 2020-07-02 08:13 Subject: [PATCH v2 1/1] git clone: don't clone into non-empty directory Message-ID: <20200702081335.16786-2-ben@wijen.net> URL: https://gitlist.dev/e/20200702081335.16786-2-ben%40wijen.net In-Reply-To: ``` When using git clone with --separate-git-dir realgitdir and realgitdir already exists, it's content is destroyed. So, make sure we don't clone into an existing non-empty directory. Signed-off-by: Ben Wijen --- builtin/clone.c | 12 ++++++++++-- t/t5601-clone.sh | 4 +++- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index 2a8e3aaaed..e470193bb8 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -946,7 +946,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix) int is_bundle = 0, is_local; const char *repo_name, *repo, *work_tree, *git_dir; char *path, *dir, *display_repo = NULL; - int dest_exists; + int dest_exists, real_dest_exists = 0; const struct ref *refs, *remote_head; const struct ref *remote_head_points_at; const struct ref *our_head_points_at; @@ -1021,6 +1021,14 @@ int cmd_clone(int argc, const char **argv, const char *prefix) die(_("destination path '%s' already exists and is not " "an empty directory."), dir); + if (real_git_dir) { + real_dest_exists = path_exists(real_git_dir); + if (real_dest_exists && !is_empty_dir(real_git_dir)) + die(_("repository path '%s' already exists and is not " + "an empty directory."), real_git_dir); + } + + strbuf_addf(&reflog_msg, "clone: from %s", display_repo ? display_repo : repo); free(display_repo); @@ -1057,7 +1065,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix) } if (real_git_dir) { - if (path_exists(real_git_dir)) + if (real_dest_exists) junk_git_dir_flags |= REMOVE_DIR_KEEP_TOPLEVEL; junk_git_dir = real_git_dir; } else { diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh index 84ea2a3eb7..eb9a093e25 100755 --- a/t/t5601-clone.sh +++ b/t/t5601-clone.sh @@ -271,7 +271,9 @@ test_expect_success 'fetch from gitfile parent' ' test_expect_success 'clone separate gitdir where target already exists' ' rm -rf dst && - test_must_fail git clone --separate-git-dir realgitdir src dst + echo foo=bar >>realgitdir/config && + test_must_fail git clone --separate-git-dir realgitdir src dst && + grep foo=bar realgitdir/config ' test_expect_success 'clone --reference from original' ' -- 2.27.0 ``` ## Junio C Hamano, 2020-07-06 22:40 Subject: Re: [PATCH v2 1/1] git clone: don't clone into non-empty directory Message-ID: URL: https://gitlist.dev/e/xmqqv9j08dxo.fsf%40gitster.c.googlers.com In-Reply-To: <20200702081335.16786-2-ben@wijen.net> ``` Ben Wijen writes: > When using git clone with --separate-git-dir realgitdir and > realgitdir already exists, it's content is destroyed. > > So, make sure we don't clone into an existing non-empty directory. This came from d45420c1 (clone: do not clean up directories we didn't create, 2018-01-02), where we claimed Because we know that the only directory we'll write into is an empty one, we can handle this case by just passing the KEEP_TOPLEVEL flag to our recursive delete (if we could write into populated directories, we'd have to keep track of what we wrote and what we did not, which would be much harder). The assumed use case is that somebody created an empty directory for our use in a grandparent directory where we have no write permission and gave it to us, and we want to do a "git clone" into it while keeping that directory alive. But we didn't make sure the directory was not empty ourselves---we just assumed it to be empty. And the originally envisioned use case does not have anything to do with the use of --separate-git-dir at all. So from that point of view, this change does not break the originally envisioned use case, so this probably is not regressing any existing and valid use case that may not involve "--separate-git-dir", but I'd rather see that the commit message explain these things to its readers to justify itself. Thanks. ``` ## Ben Wijen, 2020-07-10 08:47 Subject: [PATCH v3 1/1] git clone: don't clone into non-empty directory Message-ID: <20200710084732.5152-2-ben@wijen.net> URL: https://gitlist.dev/e/20200710084732.5152-2-ben%40wijen.net In-Reply-To: ``` When using git clone with --separate-git-dir realgitdir and realgitdir already exists, it's content is destroyed. So, make sure we don't clone into an existing non-empty directory. Note: #d45420c1: "clone: do not clean up directories we didn't create" assumed we always write into an empty directory, but missed the check for real_git_dir, this commit fixed it by adding the check. Signed-off-by: Ben Wijen --- builtin/clone.c | 12 ++++++++++-- t/t5601-clone.sh | 4 +++- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index bef70745c0..a9f3312a54 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -946,7 +946,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix) int is_bundle = 0, is_local; const char *repo_name, *repo, *work_tree, *git_dir; char *path, *dir, *display_repo = NULL; - int dest_exists; + int dest_exists, real_dest_exists = 0; const struct ref *refs, *remote_head; const struct ref *remote_head_points_at; const struct ref *our_head_points_at; @@ -1021,6 +1021,14 @@ int cmd_clone(int argc, const char **argv, const char *prefix) die(_("destination path '%s' already exists and is not " "an empty directory."), dir); + if (real_git_dir) { + real_dest_exists = path_exists(real_git_dir); + if (real_dest_exists && !is_empty_dir(real_git_dir)) + die(_("repository path '%s' already exists and is not " + "an empty directory."), real_git_dir); + } + + strbuf_addf(&reflog_msg, "clone: from %s", display_repo ? display_repo : repo); free(display_repo); @@ -1057,7 +1065,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix) } if (real_git_dir) { - if (path_exists(real_git_dir)) + if (real_dest_exists) junk_git_dir_flags |= REMOVE_DIR_KEEP_TOPLEVEL; junk_git_dir = real_git_dir; } else { diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh index 84ea2a3eb7..eb9a093e25 100755 --- a/t/t5601-clone.sh +++ b/t/t5601-clone.sh @@ -271,7 +271,9 @@ test_expect_success 'fetch from gitfile parent' ' test_expect_success 'clone separate gitdir where target already exists' ' rm -rf dst && - test_must_fail git clone --separate-git-dir realgitdir src dst + echo foo=bar >>realgitdir/config && + test_must_fail git clone --separate-git-dir realgitdir src dst && + grep foo=bar realgitdir/config ' test_expect_success 'clone --reference from original' ' -- 2.27.0 ``` ## Ben Wijen, 2020-07-10 08:47 Subject: [PATCH v3 0/1] git clone: don't clone into non-empty directory Message-ID: <20200710084732.5152-1-ben@wijen.net> URL: https://gitlist.dev/e/20200710084732.5152-1-ben%40wijen.net In-Reply-To: ``` Hi Junio, I added a small note to the commit message. Please let me know if you require more. Thank you, Ben Wijen (1): git clone: don't clone into non-empty directory builtin/clone.c | 12 ++++++++++-- t/t5601-clone.sh | 4 +++- 2 files changed, 13 insertions(+), 3 deletions(-) -- 2.27.0 ```