threads / patch / 46612

patchadd test for bug in git-mv with nested submodules

Subject: [PATCH] add test for bug in git-mv with nested submodules

## tl;dr

8 messages between Aug 17, 2017 and Sep 20, 2017. Diffs are folded; open one to read it.

replies: 7people: 3as markdown or json

Heiko Voigt· Aug 17, 2017, 10:34 UTC · lore

When using git-mv with a submodule it will detect that and update the paths for its configurations (.gitmodules, worktree and gitfile). This does not work for nested submodules where a user renames the root submodule.

We discovered this fact when working on on-demand fetch for renamed submodules. Lets add a test to document.

Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
---
 t/t7001-mv.sh | 9 +++++++++
 1 file changed, 9 insertions(+)
Show changes to t/t7001-mv.sh +9 −0
diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
index e365d1f..39f8aed 100755
--- a/t/t7001-mv.sh
+++ b/t/t7001-mv.sh
@@ -491,4 +491,13 @@ test_expect_success 'moving a submodule in nested directories' '
 	test_cmp actual expect
 '
 
+test_expect_failure 'moving nested submodules' '
+	git commit -am "cleanup commit" &&
+	git submodule add ./. sub_nested &&
+	git commit -m "add sub_nested" &&
+	git submodule update --init --recursive &&
+	git mv sub_nested sub_nested_moved &&
+	git status
+'
+
 test_done
-- 
2.0.0.274.g6b2cd91
Stefan Beller· Aug 17, 2017, 19:05 UTC · re: Heiko Voigt · lore

Re: [PATCH] add test for bug in git-mv with nested submodules

On Thu, Aug 17, 2017 at 3:34 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
Show 24 quoted lines
> When using git-mv with a submodule it will detect that and update the
> paths for its configurations (.gitmodules, worktree and gitfile). This
> does not work for nested submodules where a user renames the root
> submodule.
>
> We discovered this fact when working on on-demand fetch for renamed
> submodules. Lets add a test to document.
>
> Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
> ---
>  t/t7001-mv.sh | 9 +++++++++
>  1 file changed, 9 insertions(+)
>
> diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
> index e365d1f..39f8aed 100755
> --- a/t/t7001-mv.sh
> +++ b/t/t7001-mv.sh
> @@ -491,4 +491,13 @@ test_expect_success 'moving a submodule in nested directories' '
>         test_cmp actual expect
>  '
>
> +test_expect_failure 'moving nested submodules' '
> +       git commit -am "cleanup commit" &&
> +       git submodule add ./. sub_nested &&

If possible, I would avoid adding the repo itself as a submodule as it is unrealistic in the wild.

While it may be ok for the test here, later down the road other tests making use of it it may become an issue with the URL of the submodule.

Show 10 quoted lines
> +       git commit -m "add sub_nested" &&
> +       git submodule update --init --recursive &&
> +       git mv sub_nested sub_nested_moved &&
> +       git status
> +'
> +
>  test_done
> --
> 2.0.0.274.g6b2cd91
>
Heiko Voigt· Aug 18, 2017, 16:06 UTC · re: Stefan Beller · lore

Re: [PATCH] add test for bug in git-mv with nested submodules

On Thu, Aug 17, 2017 at 12:05:56PM -0700, Stefan Beller wrote:
Show 32 quoted lines
> On Thu, Aug 17, 2017 at 3:34 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> > When using git-mv with a submodule it will detect that and update the
> > paths for its configurations (.gitmodules, worktree and gitfile). This
> > does not work for nested submodules where a user renames the root
> > submodule.
> >
> > We discovered this fact when working on on-demand fetch for renamed
> > submodules. Lets add a test to document.
> >
> > Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
> > ---
> >  t/t7001-mv.sh | 9 +++++++++
> >  1 file changed, 9 insertions(+)
> >
> > diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
> > index e365d1f..39f8aed 100755
> > --- a/t/t7001-mv.sh
> > +++ b/t/t7001-mv.sh
> > @@ -491,4 +491,13 @@ test_expect_success 'moving a submodule in nested directories' '
> >         test_cmp actual expect
> >  '
> >
> > +test_expect_failure 'moving nested submodules' '
> > +       git commit -am "cleanup commit" &&
> > +       git submodule add ./. sub_nested &&
> 
> If possible, I would avoid adding the repo itself
> as a submodule as it is unrealistic in the wild.
> 
> While it may be ok for the test here, later down the road
> other tests making use of it it may become an issue with
> the URL of the submodule.

I just copied the shortcut that they were adding themselfes as submodule in 'setup submodule'. The whole setup of submodules in this test is like this. This way we already had a nested submodule structure which I could just add.

I agree that this is unrealistic so I can change that in the test I am adding. But from what I have seen, this shortcut is taken in quite some places when dealing with submodules.

Cheers Heiko
Stefan Beller· Aug 18, 2017, 19:04 UTC · re: Heiko Voigt · lore

Re: [PATCH] add test for bug in git-mv with nested submodules

Show 8 quoted lines
> I just copied the shortcut that they were adding themselfes as submodule
> in 'setup submodule'. The whole setup of submodules in this test is like
> this. This way we already had a nested submodule structure which I could
> just add.
>
> I agree that this is unrealistic so I can change that in the test I am
> adding. But from what I have seen, this shortcut is taken in quite some
> places when dealing with submodules.

Please do not make it worse. Once upon a time (late '16 IIRC) I had a series floating on the list removing all occurrences, but there were issues with the series and it did not land.

Heiko Voigt· Sep 15, 2017, 11:50 UTC · re: Stefan Beller · lore

[PATCH v2] add test for bug in git-mv for recursive submodules

When using git-mv with a submodule it will detect that and update the paths for its configurations (.gitmodules, worktree and gitfile). This does not work for recursive submodules where a user renames the root submodule.

We discovered this fact when working on on-demand fetch for renamed submodules. Lets add a test to document.

Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
---
On Fri, Aug 18, 2017 at 12:04:03PM -0700, Stefan Beller wrote:
Show 13 quoted lines
> > I just copied the shortcut that they were adding themselfes as submodule
> > in 'setup submodule'. The whole setup of submodules in this test is like
> > this. This way we already had a nested submodule structure which I could
> > just add.
> >
> > I agree that this is unrealistic so I can change that in the test I am
> > adding. But from what I have seen, this shortcut is taken in quite some
> > places when dealing with submodules.
> 
> Please do not make it worse.
> Once upon a time (late '16 IIRC) I had a series floating on the
> list removing all occurrences, but there were issues with the
> series and it did not land.

Took a little while but here is a more clean patch creating individual submodules for the nesting.

Cheers Heiko
 t/t7001-mv.sh | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
Show changes to t/t7001-mv.sh +25 −0
diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
index e365d1ff77..cbc5fb37fe 100755
--- a/t/t7001-mv.sh
+++ b/t/t7001-mv.sh
@@ -491,4 +491,29 @@ test_expect_success 'moving a submodule in nested directories' '
 	test_cmp actual expect
 '
 
+test_expect_failure 'moving nested submodules' '
+	git commit -am "cleanup commit" &&
+	mkdir sub_nested_nested &&
+	(cd sub_nested_nested &&
+		touch nested_level2 &&
+		git init &&
+		git add . &&
+		git commit -m "nested level 2"
+	) &&
+	mkdir sub_nested &&
+	(cd sub_nested &&
+		touch nested_level1 &&
+		git init &&
+		git add . &&
+		git commit -m "nested level 1"
+		git submodule add ../sub_nested_nested &&
+		git commit -m "add nested level 2"
+	) &&
+	git submodule add ./sub_nested nested_move &&
+	git commit -m "add nested_move" &&
+	git submodule update --init --recursive &&
+	git mv nested_move sub_nested_moved &&
+	git status
+'
+
 test_done
-- 
2.14.1.145.gb3622a4
Junio C Hamano· Sep 17, 2017, 00:46 UTC · re: Heiko Voigt · lore

Re: [PATCH v2] add test for bug in git-mv for recursive submodules

Heiko Voigt <hvoigt@hvoigt.net> writes:
Show 29 quoted lines
> When using git-mv with a submodule it will detect that and update the
> paths for its configurations (.gitmodules, worktree and gitfile). This
> does not work for recursive submodules where a user renames the root
> submodule.
>
> We discovered this fact when working on on-demand fetch for renamed
> submodules. Lets add a test to document.
>
> Signed-off-by: Heiko Voigt <hvoigt@hvoigt.net>
> ---
> On Fri, Aug 18, 2017 at 12:04:03PM -0700, Stefan Beller wrote:
>> > I just copied the shortcut that they were adding themselfes as submodule
>> > in 'setup submodule'. The whole setup of submodules in this test is like
>> > this. This way we already had a nested submodule structure which I could
>> > just add.
>> >
>> > I agree that this is unrealistic so I can change that in the test I am
>> > adding. But from what I have seen, this shortcut is taken in quite some
>> > places when dealing with submodules.
>> 
>> Please do not make it worse.
>> Once upon a time (late '16 IIRC) I had a series floating on the
>> list removing all occurrences, but there were issues with the
>> series and it did not land.
>
> Took a little while but here is a more clean patch creating individual
> submodules for the nesting.
>
> Cheers Heiko
Thanks.  Stefan, does this look good to you now?

It is not quite clear which step is expected to fail with the current code by reading the test or the proposed log message. Does "mv" refuse to work and we do not get to run "status", or does "status" report a failure, or do we fail well before that?

The log message that only says "This does not work when ..." is not helpful in figuring it out, either. Something like "This does not work and fails to update the paths for its configurations" or whatever that describes "what actually happens" (in contrast to "what ought to happen", which you described clearly) should be there.

Description on how you happened to have discovered the issue feels a lot less relevant compared to that, and it is totally useless if it is unclear what the issue is in the first place.

Show 37 quoted lines
>  t/t7001-mv.sh | 25 +++++++++++++++++++++++++
>  1 file changed, 25 insertions(+)
>
> diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
> index e365d1ff77..cbc5fb37fe 100755
> --- a/t/t7001-mv.sh
> +++ b/t/t7001-mv.sh
> @@ -491,4 +491,29 @@ test_expect_success 'moving a submodule in nested directories' '
>  	test_cmp actual expect
>  '
>  
> +test_expect_failure 'moving nested submodules' '
> +	git commit -am "cleanup commit" &&
> +	mkdir sub_nested_nested &&
> +	(cd sub_nested_nested &&
> +		touch nested_level2 &&
> +		git init &&
> +		git add . &&
> +		git commit -m "nested level 2"
> +	) &&
> +	mkdir sub_nested &&
> +	(cd sub_nested &&
> +		touch nested_level1 &&
> +		git init &&
> +		git add . &&
> +		git commit -m "nested level 1"
> +		git submodule add ../sub_nested_nested &&
> +		git commit -m "add nested level 2"
> +	) &&
> +	git submodule add ./sub_nested nested_move &&
> +	git commit -m "add nested_move" &&
> +	git submodule update --init --recursive &&
> +	git mv nested_move sub_nested_moved &&
> +	git status
> +'
> +
>  test_done
Stefan Beller· Sep 18, 2017, 20:03 UTC · re: Junio C Hamano · lore

Re: [PATCH v2] add test for bug in git-mv for recursive submodules

>> Took a little while but here is a more clean patch creating individual
>> submodules for the nesting.
>>
>> Cheers Heiko
Thanks for writing this test!
>
> Thanks.  Stefan, does this look good to you now?
Yes, though there are nits below.
> It is not quite clear which step is expected to fail with the
> current code by reading the test or the proposed log message.  Does
> "mv" refuse to work and we do not get to run "status", or does
> "status" report a failure, or do we fail well before that?

git-mv failing seems like a new possibility without incurring another process spawn with the new repository object. (Though then we could also just fix the recursed submodule)

Show 26 quoted lines
> The log message that only says "This does not work when ..." is not
> helpful in figuring it out, either.  Something like "This does not
> work and fails to update the paths for its configurations" or
> whatever that describes "what actually happens" (in contrast to
> "what ought to happen", which you described clearly) should be
> there.
>
> Description on how you happened to have discovered the issue feels a
> lot less relevant compared to that, and it is totally useless if it
> is unclear what the issue is in the first place.
>
>>  t/t7001-mv.sh | 25 +++++++++++++++++++++++++
>>  1 file changed, 25 insertions(+)
>>
>> diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
>> index e365d1ff77..cbc5fb37fe 100755
>> --- a/t/t7001-mv.sh
>> +++ b/t/t7001-mv.sh
>> @@ -491,4 +491,29 @@ test_expect_success 'moving a submodule in nested directories' '
>>       test_cmp actual expect
>>  '
>>
>> +test_expect_failure 'moving nested submodules' '
>> +     git commit -am "cleanup commit" &&
>> +     mkdir sub_nested_nested &&
>> +     (cd sub_nested_nested &&
We seem to have different styles for nested shell. I prefer
  outside command &&
  (
      first nested command here &&
      ...

as that aligns indentation to the nesting level. I have seen the style you use a lot in the test suite, and we do not have a guideline in Documentation/CodingGuidelines, so I do not complain too loudly. ;)

Show 17 quoted lines
>> +             touch nested_level2 &&
>> +             git init &&
>> +             git add . &&
>> +             git commit -m "nested level 2"
>> +     ) &&
>> +     mkdir sub_nested &&
>> +     (cd sub_nested &&
>> +             touch nested_level1 &&
>> +             git init &&
>> +             git add . &&
>> +             git commit -m "nested level 1"
>> +             git submodule add ../sub_nested_nested &&
>> +             git commit -m "add nested level 2"
>> +     ) &&
>> +     git submodule add ./sub_nested nested_move &&
>> +     git commit -m "add nested_move" &&
>> +     git submodule update --init --recursive &&
So far a nice setup!
>> +     git mv nested_move sub_nested_moved &&

This is the offending command that produces the bug, as it will break most subsequent commands, such as

>> +     git status

git-status is one of the basic commands. Without status to function, I think it is hard to recover your repo without a lot of in-depth knowledge of Git (submodules).

I wonder if git-status should complain more gracefully and fallback to one of --ignore-submodules={dirty, all}, that actually still works.

Maybe we could introduce a new default mode for this flag, that is "none-except-on-error", though this sounds as if we're fixing symptoms instead of the root cause.

Heiko Voigt· Sep 20, 2017, 13:46 UTC · re: Stefan Beller · lore

Re: [PATCH v2] add test for bug in git-mv for recursive submodules

On Mon, Sep 18, 2017 at 01:03:32PM -0700, Stefan Beller wrote:
Show 6 quoted lines
> >> Took a little while but here is a more clean patch creating individual
> >> submodules for the nesting.
> >>
> >> Cheers Heiko
> 
> Thanks for writing this test!
No worries. :)
Show 12 quoted lines
> > Thanks.  Stefan, does this look good to you now?
> 
> Yes, though there are nits below.
> 
> > It is not quite clear which step is expected to fail with the
> > current code by reading the test or the proposed log message.  Does
> > "mv" refuse to work and we do not get to run "status", or does
> > "status" report a failure, or do we fail well before that?
> 
> git-mv failing seems like a new possibility without incurring
> another process spawn with the new repository object.
> (Though then we could also just fix the recursed submodule)

It is mv that fails to update everything necessary when using it with recursively nested submodules. So the git-mv command does not report a failure here. As an interim fix it could maybe report an error when encountering nested submodules but the real fix would be to teach it to recursively spawn the appropriate git-mv commands.

Show 10 quoted lines
> > The log message that only says "This does not work when ..." is not
> > helpful in figuring it out, either.  Something like "This does not
> > work and fails to update the paths for its configurations" or
> > whatever that describes "what actually happens" (in contrast to
> > "what ought to happen", which you described clearly) should be
> > there.
> >
> > Description on how you happened to have discovered the issue feels a
> > lot less relevant compared to that, and it is totally useless if it
> > is unclear what the issue is in the first place.

Sorry about being a bit brief here. How about dropping that information how I discovered the bug then and change the commit message to something like this:

    add test for bug in git-mv for recursive submodules
    When using git-mv with a submodule it will detect that and update
    the paths for its configurations (.gitmodules, worktree and
    gitfile). This does not work in case it encounters nested
    submodules. In that case it only updates the configurations for the
    submodule directly underneath the superproject and fails to update
    the paths for the submodules nested more deeply. This in turn leads
    to the symptom that git status reports that it can not chdir to the
    nested submodule in its old location.
    Lets add a test to document.
?
Show 27 quoted lines
> >>  t/t7001-mv.sh | 25 +++++++++++++++++++++++++
> >>  1 file changed, 25 insertions(+)
> >>
> >> diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
> >> index e365d1ff77..cbc5fb37fe 100755
> >> --- a/t/t7001-mv.sh
> >> +++ b/t/t7001-mv.sh
> >> @@ -491,4 +491,29 @@ test_expect_success 'moving a submodule in nested directories' '
> >>       test_cmp actual expect
> >>  '
> >>
> >> +test_expect_failure 'moving nested submodules' '
> >> +     git commit -am "cleanup commit" &&
> >> +     mkdir sub_nested_nested &&
> >> +     (cd sub_nested_nested &&
> 
> We seem to have different styles for nested shell. I prefer
> 
>   outside command &&
>   (
>       first nested command here &&
>       ...
> 
> as that aligns indentation to the nesting level. I have seen
> the style you use a lot in the  test suite, and we do not have
> a guideline in Documentation/CodingGuidelines, so I do not
> complain too loudly. ;)

Yeah we have some different styles it seems ;) So here some reasoning behind my style:

I actually would agree on your style if 'first nested command' was any arbitrary command but when I use my style it is always when I use a nested shell for changing into some directory, doing something there and then being able to return to the previous directory by closing the nested shell. So for me the 'cd somewhere' belongs to the brackets similarly like a condition definition belongs to the if it is used with.

Show 19 quoted lines
> >> +             touch nested_level2 &&
> >> +             git init &&
> >> +             git add . &&
> >> +             git commit -m "nested level 2"
> >> +     ) &&
> >> +     mkdir sub_nested &&
> >> +     (cd sub_nested &&
> >> +             touch nested_level1 &&
> >> +             git init &&
> >> +             git add . &&
> >> +             git commit -m "nested level 1"
> >> +             git submodule add ../sub_nested_nested &&
> >> +             git commit -m "add nested level 2"
> >> +     ) &&
> >> +     git submodule add ./sub_nested nested_move &&
> >> +     git commit -m "add nested_move" &&
> >> +     git submodule update --init --recursive &&
> 
> So far a nice setup!
Thanks.
> >> +     git mv nested_move sub_nested_moved &&
> 
> This is the offending command that produces the bug,
> as it will break most subsequent commands, such as
Yes.
Show 13 quoted lines
> >> +     git status
> 
> git-status is one of the basic commands. Without
> status to function, I think it is hard to recover your repo without
> a lot of in-depth knowledge of Git (submodules).
> 
> I wonder if git-status should complain more gracefully
> and fallback to one of --ignore-submodules={dirty, all},
> that actually still works.
> 
> Maybe we could introduce a new default mode for this
> flag, that is "none-except-on-error", though this sounds
> as if we're fixing symptoms instead of the root cause.

I think we should rather fix the root cause. For me git-mv is actually breaking the repository and as described above one possible interim solution for me would be for 'git-mv' to error out and tell the user that it does currently not work on recursively nested submodules.

Cheers Heiko

← back to recent threads