threads / bug / 42660

[BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

Subject: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

## tl;dr

17 messages between Jun 19, 2016 and Apr 19, 2017.

replies: 16people: 7as markdown or json

Vadim Eisenberg· Jun 19, 2016, 07:17 UTC · lore

/usr/local/bin/git clone --recursive --depth 10 https://github.com/IBM-Swift/Kitura-net.git Cloning into 'Kitura-net'... remote: Counting objects: 253, done. remote: Compressing objects: 100% (142/142), done. remote: Total 253 (delta 134), reused 188 (delta 86), pack-reused 0 Receiving objects: 100% (253/253), 63.28 KiB | 0 bytes/s, done. Resolving deltas: 100% (134/134), done. Checking connectivity... done. Submodule 'Kitura-Build' (https://github.com/IBM-Swift/Kitura-Build.git) registered for path 'Kitura-Build' Cloning into '/home/vadime/Kitura-net/Kitura-Build'... error: no such remote ref d0d9d6c739a79627641e6438fe4f39bd0eba83bb Fetched in submodule path 'Kitura-Build', but it did not contain d0d9d6c739a79627641e6438fe4f39bd0eba83bb. Direct fetching of that commit failed.

Linux version: 4.2.0-25-generic #30-Ubuntu SMP Mon Jan 18 12:31:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Jeff King· Jun 19, 2016, 10:00 UTC · re: Vadim Eisenberg · lore

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

On Sun, Jun 19, 2016 at 10:17:36AM +0300, Vadim Eisenberg wrote:
Show 16 quoted lines
> /usr/local/bin/git clone --recursive --depth 10 
> https://github.com/IBM-Swift/Kitura-net.git
> Cloning into 'Kitura-net'...
> remote: Counting objects: 253, done.
> remote: Compressing objects: 100% (142/142), done.
> remote: Total 253 (delta 134), reused 188 (delta 86), pack-reused 0
> Receiving objects: 100% (253/253), 63.28 KiB | 0 bytes/s, done.
> Resolving deltas: 100% (134/134), done.
> Checking connectivity... done.
> Submodule 'Kitura-Build' (https://github.com/IBM-Swift/Kitura-Build.git) 
> registered for path 'Kitura-Build'
> Cloning into '/home/vadime/Kitura-net/Kitura-Build'...
> error: no such remote ref d0d9d6c739a79627641e6438fe4f39bd0eba83bb
> Fetched in submodule path 'Kitura-Build', but it did not contain 
> d0d9d6c739a79627641e6438fe4f39bd0eba83bb. Direct fetching of that commit 
> failed.

The problem seems to be the shallow clone. The super-project points to a commit in the submodule that is not near the tip of any branch, so shallow-cloning the submodule means we don't get that commit. Prior to d22eb04 (clone: add `--shallow-submodules` flag, 2016-04-25), submodules were _always_ cloned fully.

The immediate workaround is to add "--no-shallow-submodules" to your clone invocation.

Stefan, I think it might be worth revisiting the default set by d22eb04 to propagate shallowness from the super-project clone. In an ideal world, we would be asking each submodule for the actual commit we are interested in, and shallowness would not matter. But until uploadpack.allowReachableSHA1InWant works everywhere, I suspect this is going to be a problem.

-Peff
Vadim Eisenberg· Jun 19, 2016, 13:07 UTC · re: Jeff King · lore

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

The problem is with other tools that use git, like Swift Package Manager - https://swift.org/package-manager/. Versions of git before 2.9.0 have no option --no-shallow-submodules. So the tools that use git would have to check if the version of git is greater than or equal 2.9.0 to know to specify that option.

Best Regards, Vadim

Jeff King <peff@peff.net> wrote on 06/19/2016 01:00:51 PM:
Show 19 quoted lines
> From: Jeff King <peff@peff.net>
> To: Vadim Eisenberg/Haifa/IBM@IBMIL
> Cc: Stefan Beller <sbeller@google.com>, git@vger.kernel.org
> Date: 06/19/2016 01:01 PM
> Subject: Re: [BUG REPORT] git 2.9.0 clone --recursive fails on 
> cloning a submodule
> 
> On Sun, Jun 19, 2016 at 10:17:36AM +0300, Vadim Eisenberg wrote:
> 
> > /usr/local/bin/git clone --recursive --depth 10 
> > https://github.com/IBM-Swift/Kitura-net.git
> > Cloning into 'Kitura-net'...
> > remote: Counting objects: 253, done.
> > remote: Compressing objects: 100% (142/142), done.
> > remote: Total 253 (delta 134), reused 188 (delta 86), pack-reused 0
> > Receiving objects: 100% (253/253), 63.28 KiB | 0 bytes/s, done.
> > Resolving deltas: 100% (134/134), done.
> > Checking connectivity... done.
> > Submodule 'Kitura-Build' (
https://github.com/IBM-Swift/Kitura-Build.git) 
Show 5 quoted lines
> > registered for path 'Kitura-Build'
> > Cloning into '/home/vadime/Kitura-net/Kitura-Build'...
> > error: no such remote ref d0d9d6c739a79627641e6438fe4f39bd0eba83bb
> > Fetched in submodule path 'Kitura-Build', but it did not contain 
> > d0d9d6c739a79627641e6438fe4f39bd0eba83bb. Direct fetching of that 
commit 
Show 20 quoted lines
> > failed.
> 
> The problem seems to be the shallow clone. The super-project points to a
> commit in the submodule that is not near the tip of any branch, so
> shallow-cloning the submodule means we don't get that commit. Prior to
> d22eb04 (clone: add `--shallow-submodules` flag, 2016-04-25), submodules
> were _always_ cloned fully.
> 
> The immediate workaround is to add "--no-shallow-submodules" to your
> clone invocation.
> 
> Stefan, I think it might be worth revisiting the default set by d22eb04
> to propagate shallowness from the super-project clone. In an ideal
> world, we would be asking each submodule for the actual commit we are
> interested in, and shallowness would not matter. But until
> uploadpack.allowReachableSHA1InWant works everywhere, I suspect this is
> going to be a problem.
> 
> -Peff
> 
Jeff King· Jun 19, 2016, 14:46 UTC · re: Vadim Eisenberg · lore

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

On Sun, Jun 19, 2016 at 04:07:41PM +0300, Vadim Eisenberg wrote:
Show 5 quoted lines
> The problem is with other tools that use git, like Swift Package Manager - 
> https://swift.org/package-manager/.
> Versions of git before 2.9.0 have no option --no-shallow-submodules. So 
> the tools that use git would have to check if the version of git is 
> greater than or equal 2.9.0 to know to specify that option.

Oh, I agree it's something we should treat as a regression. Using `--no-shallow-submodules` is just a workaround.

-Peff
Junio C Hamano· Jun 19, 2016, 20:51 UTC · re: Jeff King · lore

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

Jeff King <peff@peff.net> writes:
Show 6 quoted lines
> Stefan, I think it might be worth revisiting the default set by d22eb04
> to propagate shallowness from the super-project clone. In an ideal
> world, we would be asking each submodule for the actual commit we are
> interested in, and shallowness would not matter. But until
> uploadpack.allowReachableSHA1InWant works everywhere, I suspect this is
> going to be a problem.

Yup, something like this on top of d22eb04 to be merged before v2.9.1 for the maintenance track would be necessary.

-- >8 --
Subject: clone: do not let --depth imply --shallow-submodules

In v2.9.0, we prematurely flipped the default to force cloning submodules shallowly, when the superproject is getting cloned shallowly. This is likely to fail when the upstream repositories submodules are cloned from a repository that is not prepared to serve histories that ends at a commit that is not at the tip of a branch, and we know the world is not yet ready.

Use a safer default to clone the submodules fully, unless the user tells us that she knows that the upstream repository of the submodules are willing to cooperate with "--shallow-submodules" option.

Noticed-by: Vadim Eisenberg <VADIME@il.ibm.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 Documentation/git-clone.txt | 5 ++---
 builtin/clone.c             | 5 ++---
 t/t5614-clone-submodules.sh | 4 ++--
 3 files changed, 6 insertions(+), 8 deletions(-)
diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index e1a21b7..c5a1ce2 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -192,9 +192,8 @@ objects from the source repository into a pack in the cloned repository.
 	Create a 'shallow' clone with a history truncated to the
 	specified number of revisions. Implies `--single-branch` unless
 	`--no-single-branch` is given to fetch the histories near the
-	tips of all branches. This implies `--shallow-submodules`. If
-	you want to have a shallow superproject clone, but full submodules,
-	also pass `--no-shallow-submodules`.
+	tips of all branches. If you want to clone submodules shallowly,
+	also pass `--shallow-submodules`.
 
 --[no-]single-branch::
 	Clone only the history leading to the tip of a single branch,
diff --git a/builtin/clone.c b/builtin/clone.c
index ecdf308..f267742 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -40,7 +40,7 @@ static const char * const builtin_clone_usage[] = {
 
 static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
 static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;
-static int option_shallow_submodules = -1;
+static int option_shallow_submodules;
 static char *option_template, *option_depth;
 static char *option_origin = NULL;
 static char *option_branch = NULL;
@@ -730,8 +730,7 @@ static int checkout(void)
 		struct argv_array args = ARGV_ARRAY_INIT;
 		argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
 
-		if (option_shallow_submodules == 1
-		    || (option_shallow_submodules == -1 && option_depth))
+		if (option_shallow_submodules == 1)
 			argv_array_push(&args, "--depth=1");
 
 		if (max_jobs != -1)
diff --git a/t/t5614-clone-submodules.sh b/t/t5614-clone-submodules.sh
index 62044c5..f7c630b 100755
--- a/t/t5614-clone-submodules.sh
+++ b/t/t5614-clone-submodules.sh
@@ -37,9 +37,9 @@ test_expect_success 'nonshallow clone implies nonshallow submodule' '
 	)
 '
 
-test_expect_success 'shallow clone implies shallow submodule' '
+test_expect_success 'shallow clone does not imply shallow submodule' '
 	test_when_finished "rm -rf super_clone" &&
-	git clone --recurse-submodules --depth 2 "file://$pwd/." super_clone &&
+	git clone --recurse-submodules --depth 2 --shallow-submodules "file://$pwd/." super_clone &&
 	(
 		cd super_clone &&
 		git log --oneline >lines &&
Jeff King· Jun 20, 2016, 00:13 UTC · re: Junio C Hamano · lore

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

On Sun, Jun 19, 2016 at 01:51:56PM -0700, Junio C Hamano wrote:
Show 17 quoted lines
> Yup, something like this on top of d22eb04 to be merged before
> v2.9.1 for the maintenance track would be necessary.
> 
> -- >8 --
> Subject: clone: do not let --depth imply --shallow-submodules
> 
> In v2.9.0, we prematurely flipped the default to force cloning
> submodules shallowly, when the superproject is getting cloned
> shallowly.  This is likely to fail when the upstream repositories
> submodules are cloned from a repository that is not prepared to
> serve histories that ends at a commit that is not at the tip of a
> branch, and we know the world is not yet ready.
> 
> Use a safer default to clone the submodules fully, unless the user
> tells us that she knows that the upstream repository of the
> submodules are willing to cooperate with "--shallow-submodules"
> option.
Yeah, this looks good. To minor comments:
Show 8 quoted lines
> @@ -730,8 +730,7 @@ static int checkout(void)
>  		struct argv_array args = ARGV_ARRAY_INIT;
>  		argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
>  
> -		if (option_shallow_submodules == 1
> -		    || (option_shallow_submodules == -1 && option_depth))
> +		if (option_shallow_submodules == 1)
>  			argv_array_push(&args, "--depth=1");

I hadn't paid much attention to this topic originally, but was surprised that "--depth 10" in the clone implies "--depth 1" in the submodule. This is not really related to your patch (in fact, your patch makes the logic go away). But maybe something to consider if it's ever resurrected (or possibly if somebody runs "--shallow-submodules --depth 5" we should pass --depth=1; I dunno).

Show 8 quoted lines
> -test_expect_success 'shallow clone implies shallow submodule' '
> +test_expect_success 'shallow clone does not imply shallow submodule' '
>  	test_when_finished "rm -rf super_clone" &&
> -	git clone --recurse-submodules --depth 2 "file://$pwd/." super_clone &&
> +	git clone --recurse-submodules --depth 2 --shallow-submodules "file://$pwd/." super_clone &&
>  	(
>  		cd super_clone &&
>  		git log --oneline >lines &&

We are not really testing "does not imply" here, but "passing --shallow-submodules works". The "does not imply" test would be cloning without the option and checking that the resulting submodules are not shallow.

-Peff
Stefan Beller· Jun 20, 2016, 01:09 UTC · re: Jeff King · lore

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

On Sun, Jun 19, 2016 at 5:13 PM, Jeff King <peff@peff.net> wrote:
Show 21 quoted lines
> On Sun, Jun 19, 2016 at 01:51:56PM -0700, Junio C Hamano wrote:
>
>> Yup, something like this on top of d22eb04 to be merged before
>> v2.9.1 for the maintenance track would be necessary.
>>
>> -- >8 --
>> Subject: clone: do not let --depth imply --shallow-submodules
>>
>> In v2.9.0, we prematurely flipped the default to force cloning
>> submodules shallowly, when the superproject is getting cloned
>> shallowly.  This is likely to fail when the upstream repositories
>> submodules are cloned from a repository that is not prepared to
>> serve histories that ends at a commit that is not at the tip of a
>> branch, and we know the world is not yet ready.
>>
>> Use a safer default to clone the submodules fully, unless the user
>> tells us that she knows that the upstream repository of the
>> submodules are willing to cooperate with "--shallow-submodules"
>> option.
>
> Yeah, this looks good. To minor comments:
I agree, but I find the second concern a bit more than just minor.
Show 16 quoted lines
>
>> @@ -730,8 +730,7 @@ static int checkout(void)
>>               struct argv_array args = ARGV_ARRAY_INIT;
>>               argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
>>
>> -             if (option_shallow_submodules == 1
>> -                 || (option_shallow_submodules == -1 && option_depth))
>> +             if (option_shallow_submodules == 1)
>>                       argv_array_push(&args, "--depth=1");
>
> I hadn't paid much attention to this topic originally, but was surprised
> that "--depth 10" in the clone implies "--depth 1" in the submodule.
> This is not really related to your patch (in fact, your patch makes the
> logic go away). But maybe something to consider if it's ever resurrected
> (or possibly if somebody runs "--shallow-submodules --depth 5" we should
> pass --depth=1; I dunno).

How often do we see a depth != 1 in practice? I have the impression (and no data to back up my claim) that a binary switch for nonshallow or depth 1 would serve us just as good, which is why I did not want to ad complexity to the submodule depth. (What if you want submodule A with depth 2 and B with 5? In that case get them all shallow and deepen as appropriate, would be my answer)

Show 14 quoted lines
>
>> -test_expect_success 'shallow clone implies shallow submodule' '
>> +test_expect_success 'shallow clone does not imply shallow submodule' '
>>       test_when_finished "rm -rf super_clone" &&
>> -     git clone --recurse-submodules --depth 2 "file://$pwd/." super_clone &&
>> +     git clone --recurse-submodules --depth 2 --shallow-submodules "file://$pwd/." super_clone &&
>>       (
>>               cd super_clone &&
>>               git log --oneline >lines &&
>
> We are not really testing "does not imply" here, but "passing
> --shallow-submodules works". The "does not imply" test would be cloning
> without the option and checking that the resulting submodules are not
> shallow.

In case we want to be sure that it works for 2.9.1, i.e. we treat it as a regression, we need to test the "does not imply" a bit more I would think. I can send that test on top tomorrow if you'd like to.

Thanks, Stefan

>
> -Peff
Vadim Eisenberg· Jun 20, 2016, 03:01 UTC · re: Stefan Beller · lore

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

> How often do we see a depth != 1 in practice? 
FYI: Here is an example of depth 10 from Swift Package Manager which is 
git-based 
https://github.com/apple/swift-package-manager/blob/61bdff3e9c2b8456be31861b4cda87a31440d517/Sources/Get/Git.swift
try system(Git.tool, "clone",
        "--recursive",   // get submodules too so that developers can use 
these if they so choose
        "--depth", "10",
Stefan Beller <sbeller@google.com> wrote on 06/20/2016 04:09:28 AM:
Show 41 quoted lines
> From: Stefan Beller <sbeller@google.com>
> To: Jeff King <peff@peff.net>
> Cc: Junio C Hamano <gitster@pobox.com>, Vadim Eisenberg/Haifa/
> IBM@IBMIL, "git@vger.kernel.org" <git@vger.kernel.org>
> Date: 06/20/2016 04:09 AM
> Subject: Re: [BUG REPORT] git 2.9.0 clone --recursive fails on 
> cloning a submodule
> 
> On Sun, Jun 19, 2016 at 5:13 PM, Jeff King <peff@peff.net> wrote:
> > On Sun, Jun 19, 2016 at 01:51:56PM -0700, Junio C Hamano wrote:
> >
> >> Yup, something like this on top of d22eb04 to be merged before
> >> v2.9.1 for the maintenance track would be necessary.
> >>
> >> -- >8 --
> >> Subject: clone: do not let --depth imply --shallow-submodules
> >>
> >> In v2.9.0, we prematurely flipped the default to force cloning
> >> submodules shallowly, when the superproject is getting cloned
> >> shallowly.  This is likely to fail when the upstream repositories
> >> submodules are cloned from a repository that is not prepared to
> >> serve histories that ends at a commit that is not at the tip of a
> >> branch, and we know the world is not yet ready.
> >>
> >> Use a safer default to clone the submodules fully, unless the user
> >> tells us that she knows that the upstream repository of the
> >> submodules are willing to cooperate with "--shallow-submodules"
> >> option.
> >
> > Yeah, this looks good. To minor comments:
> 
> I agree, but I find the second concern a bit more than just minor.
> 
> >
> >> @@ -730,8 +730,7 @@ static int checkout(void)
> >>               struct argv_array args = ARGV_ARRAY_INIT;
> >>               argv_array_pushl(&args, "submodule", "update", "--
> init", "--recursive", NULL);
> >>
> >> -             if (option_shallow_submodules == 1
> >> -                 || (option_shallow_submodules == -1 && 
option_depth))
> >> +             if (option_shallow_submodules == 1)
> >>                       argv_array_push(&args, "--depth=1");
> >
> > I hadn't paid much attention to this topic originally, but was 
surprised
> > that "--depth 10" in the clone implies "--depth 1" in the submodule.
> > This is not really related to your patch (in fact, your patch makes 
the
> > logic go away). But maybe something to consider if it's ever 
resurrected
> > (or possibly if somebody runs "--shallow-submodules --depth 5" we 
should
Show 5 quoted lines
> > pass --depth=1; I dunno).
> 
> How often do we see a depth != 1 in practice?
> I have the impression (and no data to back up my claim) that a binary
> switch for nonshallow or depth 1 would serve us just as good, which is 
why
Show 7 quoted lines
> I did not want to ad complexity to the submodule depth.
> (What if you want submodule A with depth 2 and B with 5? In that
> case get them all shallow and deepen as appropriate, would be my answer)
> 
> >
> >> -test_expect_success 'shallow clone implies shallow submodule' '
> >> +test_expect_success 'shallow clone does not imply shallow submodule' 
'
Show 11 quoted lines
> >>       test_when_finished "rm -rf super_clone" &&
> >> -     git clone --recurse-submodules --depth 2 "file://$pwd/." 
> super_clone &&
> >> +     git clone --recurse-submodules --depth 2 --shallow-submodules "
> file://$pwd/." super_clone &&
> >>       (
> >>               cd super_clone &&
> >>               git log --oneline >lines &&
> >
> > We are not really testing "does not imply" here, but "passing
> > --shallow-submodules works". The "does not imply" test would be 
cloning
Show 6 quoted lines
> > without the option and checking that the resulting submodules are not
> > shallow.
> 
> In case we want to be sure that it works for 2.9.1, i.e. we treat it
> as a regression,
> we need to test the "does not imply" a bit more I would think. I cansend 
that
Show 8 quoted lines
> test on top tomorrow if you'd like to.
> 
> Thanks,
> Stefan
> 
> >
> > -Peff
> 
Jeff King· Jun 20, 2016, 10:02 UTC · re: Stefan Beller · lore

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

On Sun, Jun 19, 2016 at 06:09:28PM -0700, Stefan Beller wrote:
Show 13 quoted lines
> > I hadn't paid much attention to this topic originally, but was surprised
> > that "--depth 10" in the clone implies "--depth 1" in the submodule.
> > This is not really related to your patch (in fact, your patch makes the
> > logic go away). But maybe something to consider if it's ever resurrected
> > (or possibly if somebody runs "--shallow-submodules --depth 5" we should
> > pass --depth=1; I dunno).
> 
> How often do we see a depth != 1 in practice?
> I have the impression (and no data to back up my claim) that a binary
> switch for nonshallow or depth 1 would serve us just as good, which is why
> I did not want to ad complexity to the submodule depth.
> (What if you want submodule A with depth 2 and B with 5? In that
> case get them all shallow and deepen as appropriate, would be my answer)

To be honest, I don't know why people use anything except --depth=1, but it's clear from my experience that they do. This example has --depth=10, and on the server side at GitHub I have seen similar numbers from clients, especially CI services.

(I take special note of such cases because --shallow quite often causes performance problems on the server side, though generally --depth=10 is not any worse than --depth=1. The worst case is really "--no-single-branch --depth=1", which wants a ton of objects but has to throw away all of the on-disk deltas).

Show 9 quoted lines
> > We are not really testing "does not imply" here, but "passing
> > --shallow-submodules works". The "does not imply" test would be cloning
> > without the option and checking that the resulting submodules are not
> > shallow.
> 
> In case we want to be sure that it works for 2.9.1, i.e. we treat it
> as a regression,
> we need to test the "does not imply" a bit more I would think. I can send that
> test on top tomorrow if you'd like to.

I think it's worth doing (and testing both: the default behavior, and that the --shallow-submodules feature works). Thanks.

-Peff
Stefan Beller· Jun 20, 2016, 16:59 UTC · re: Junio C Hamano · lore

[PATCH] shallow clone to not imply shallow submodules

Signed-off-by: Stefan Beller <sbeller@google.com>
---
Hi Junio, Peff,

I thought about this patch squashed into "clone: do not let --depth imply --shallow-submodules" will actually test for the regression.

Thanks, Stefan

 t/t5614-clone-submodules.sh | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/t/t5614-clone-submodules.sh b/t/t5614-clone-submodules.sh
index f7c630b..a9aaa01 100755
--- a/t/t5614-clone-submodules.sh
+++ b/t/t5614-clone-submodules.sh
@@ -37,7 +37,7 @@ test_expect_success 'nonshallow clone implies nonshallow submodule' '
 	)
 '
 
-test_expect_success 'shallow clone does not imply shallow submodule' '
+test_expect_success 'shallow clone with shallow submodule' '
 	test_when_finished "rm -rf super_clone" &&
 	git clone --recurse-submodules --depth 2 --shallow-submodules "file://$pwd/." super_clone &&
 	(
@@ -52,6 +52,21 @@ test_expect_success 'shallow clone does not imply shallow submodule' '
 	)
 '
 
+test_expect_success 'shallow clone does not imply shallow submodule' '
+	test_when_finished "rm -rf super_clone" &&
+	git clone --recurse-submodules --depth 2 "file://$pwd/." super_clone &&
+	(
+		cd super_clone &&
+		git log --oneline >lines &&
+		test_line_count = 2 lines
+	) &&
+	(
+		cd super_clone/sub &&
+		git log --oneline >lines &&
+		test_line_count = 3 lines
+	)
+'
+
 test_expect_success 'shallow clone with non shallow submodule' '
 	test_when_finished "rm -rf super_clone" &&
 	git clone --recurse-submodules --depth 2 --no-shallow-submodules "file://$pwd/." super_clone &&
-- 
2.7.0.rc0.40.g5328432.dirty
Jeff King· Jun 20, 2016, 17:13 UTC · re: Stefan Beller · lore

Re: [PATCH] shallow clone to not imply shallow submodules

On Mon, Jun 20, 2016 at 09:59:58AM -0700, Stefan Beller wrote:
Show 8 quoted lines
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
> 
> Hi Junio, Peff,
> 
> I thought about this patch squashed into  
> "clone: do not let --depth imply --shallow-submodules" will actually test
> for the regression.
Yep, it looks good to me.
Show 14 quoted lines
> +test_expect_success 'shallow clone does not imply shallow submodule' '
> +	test_when_finished "rm -rf super_clone" &&
> +	git clone --recurse-submodules --depth 2 "file://$pwd/." super_clone &&
> +	(
> +		cd super_clone &&
> +		git log --oneline >lines &&
> +		test_line_count = 2 lines
> +	) &&
> +	(
> +		cd super_clone/sub &&
> +		git log --oneline >lines &&
> +		test_line_count = 3 lines
> +	)
> +'

This follows the style of the other tests, so it's the right thing here. But as a style suggestion, I think:

  git -C super_clone/sub log --oneline >lines &&
  test_line_count = 3 lines
is nicer than the subshell. It's more succinct, and it saves a process.
-Peff
Stefan Beller· Jun 20, 2016, 17:14 UTC · re: Jeff King · lore

Re: [PATCH] shallow clone to not imply shallow submodules

On Mon, Jun 20, 2016 at 10:13 AM, Jeff King <peff@peff.net> wrote:
Show 35 quoted lines
> On Mon, Jun 20, 2016 at 09:59:58AM -0700, Stefan Beller wrote:
>
>> Signed-off-by: Stefan Beller <sbeller@google.com>
>> ---
>>
>> Hi Junio, Peff,
>>
>> I thought about this patch squashed into
>> "clone: do not let --depth imply --shallow-submodules" will actually test
>> for the regression.
>
> Yep, it looks good to me.
>
>> +test_expect_success 'shallow clone does not imply shallow submodule' '
>> +     test_when_finished "rm -rf super_clone" &&
>> +     git clone --recurse-submodules --depth 2 "file://$pwd/." super_clone &&
>> +     (
>> +             cd super_clone &&
>> +             git log --oneline >lines &&
>> +             test_line_count = 2 lines
>> +     ) &&
>> +     (
>> +             cd super_clone/sub &&
>> +             git log --oneline >lines &&
>> +             test_line_count = 3 lines
>> +     )
>> +'
>
> This follows the style of the other tests, so it's the right thing here.
> But as a style suggestion, I think:
>
>   git -C super_clone/sub log --oneline >lines &&
>   test_line_count = 3 lines
>
> is nicer than the subshell. It's more succinct, and it saves a process.

which we would want to refactor to in a follow up, but not merge it through to 2.9.1.

Thanks, Stefan

>
> -Peff
Jeff King· Jun 20, 2016, 17:18 UTC · re: Stefan Beller · lore

Re: [PATCH] shallow clone to not imply shallow submodules

On Mon, Jun 20, 2016 at 10:14:47AM -0700, Stefan Beller wrote:
Show 10 quoted lines
> > This follows the style of the other tests, so it's the right thing here.
> > But as a style suggestion, I think:
> >
> >   git -C super_clone/sub log --oneline >lines &&
> >   test_line_count = 3 lines
> >
> > is nicer than the subshell. It's more succinct, and it saves a process.
> 
> which we would want to refactor to in a follow up, but not merge it
> through to 2.9.1.
Yeah, exactly. That was what I meant by "here".
-Peff
Ævar Arnfjörð Bjarmason· Apr 19, 2017, 11:30 UTC · re: Junio C Hamano · lore

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

On Sun, Jun 19, 2016 at 10:51 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 73 quoted lines
> Jeff King <peff@peff.net> writes:
>
>> Stefan, I think it might be worth revisiting the default set by d22eb04
>> to propagate shallowness from the super-project clone. In an ideal
>> world, we would be asking each submodule for the actual commit we are
>> interested in, and shallowness would not matter. But until
>> uploadpack.allowReachableSHA1InWant works everywhere, I suspect this is
>> going to be a problem.
>
> Yup, something like this on top of d22eb04 to be merged before
> v2.9.1 for the maintenance track would be necessary.
>
> -- >8 --
> Subject: clone: do not let --depth imply --shallow-submodules
>
> In v2.9.0, we prematurely flipped the default to force cloning
> submodules shallowly, when the superproject is getting cloned
> shallowly.  This is likely to fail when the upstream repositories
> submodules are cloned from a repository that is not prepared to
> serve histories that ends at a commit that is not at the tip of a
> branch, and we know the world is not yet ready.
>
> Use a safer default to clone the submodules fully, unless the user
> tells us that she knows that the upstream repository of the
> submodules are willing to cooperate with "--shallow-submodules"
> option.
>
> Noticed-by: Vadim Eisenberg <VADIME@il.ibm.com>
> Helped-by: Jeff King <peff@peff.net>
> Signed-off-by: Junio C Hamano <gitster@pobox.com>
> ---
>  Documentation/git-clone.txt | 5 ++---
>  builtin/clone.c             | 5 ++---
>  t/t5614-clone-submodules.sh | 4 ++--
>  3 files changed, 6 insertions(+), 8 deletions(-)
>
> diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
> index e1a21b7..c5a1ce2 100644
> --- a/Documentation/git-clone.txt
> +++ b/Documentation/git-clone.txt
> @@ -192,9 +192,8 @@ objects from the source repository into a pack in the cloned repository.
>         Create a 'shallow' clone with a history truncated to the
>         specified number of revisions. Implies `--single-branch` unless
>         `--no-single-branch` is given to fetch the histories near the
> -       tips of all branches. This implies `--shallow-submodules`. If
> -       you want to have a shallow superproject clone, but full submodules,
> -       also pass `--no-shallow-submodules`.
> +       tips of all branches. If you want to clone submodules shallowly,
> +       also pass `--shallow-submodules`.
>
>  --[no-]single-branch::
>         Clone only the history leading to the tip of a single branch,
> diff --git a/builtin/clone.c b/builtin/clone.c
> index ecdf308..f267742 100644
> --- a/builtin/clone.c
> +++ b/builtin/clone.c
> @@ -40,7 +40,7 @@ static const char * const builtin_clone_usage[] = {
>
>  static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
>  static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;
> -static int option_shallow_submodules = -1;
> +static int option_shallow_submodules;
>  static char *option_template, *option_depth;
>  static char *option_origin = NULL;
>  static char *option_branch = NULL;
> @@ -730,8 +730,7 @@ static int checkout(void)
>                 struct argv_array args = ARGV_ARRAY_INIT;
>                 argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
>
> -               if (option_shallow_submodules == 1
> -                   || (option_shallow_submodules == -1 && option_depth))
> +               if (option_shallow_submodules == 1)
>                         argv_array_push(&args, "--depth=1");

Very late reply, since I'm just looking at this now with the --no-tags opti,n, but that == 1 makes no sense anymore, and should just be `if (option_shallow_submodules)` shouldn't it? I.e. this used to be an int for the depth, now is a bool, but the current == 1 check is left over probably from an earlier version where the depth was configurable.

Show 21 quoted lines
>                 if (max_jobs != -1)
> diff --git a/t/t5614-clone-submodules.sh b/t/t5614-clone-submodules.sh
> index 62044c5..f7c630b 100755
> --- a/t/t5614-clone-submodules.sh
> +++ b/t/t5614-clone-submodules.sh
> @@ -37,9 +37,9 @@ test_expect_success 'nonshallow clone implies nonshallow submodule' '
>         )
>  '
>
> -test_expect_success 'shallow clone implies shallow submodule' '
> +test_expect_success 'shallow clone does not imply shallow submodule' '
>         test_when_finished "rm -rf super_clone" &&
> -       git clone --recurse-submodules --depth 2 "file://$pwd/." super_clone &&
> +       git clone --recurse-submodules --depth 2 --shallow-submodules "file://$pwd/." super_clone &&
>         (
>                 cd super_clone &&
>                 git log --oneline >lines &&
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
Stefan Beller· Apr 19, 2017, 18:54 UTC · re: Ævar Arnfjörð Bjarmason · lore

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

On Wed, Apr 19, 2017 at 4:30 AM, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

Show 80 quoted lines
> On Sun, Jun 19, 2016 at 10:51 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Jeff King <peff@peff.net> writes:
>>
>>> Stefan, I think it might be worth revisiting the default set by d22eb04
>>> to propagate shallowness from the super-project clone. In an ideal
>>> world, we would be asking each submodule for the actual commit we are
>>> interested in, and shallowness would not matter. But until
>>> uploadpack.allowReachableSHA1InWant works everywhere, I suspect this is
>>> going to be a problem.
>>
>> Yup, something like this on top of d22eb04 to be merged before
>> v2.9.1 for the maintenance track would be necessary.
>>
>> -- >8 --
>> Subject: clone: do not let --depth imply --shallow-submodules
>>
>> In v2.9.0, we prematurely flipped the default to force cloning
>> submodules shallowly, when the superproject is getting cloned
>> shallowly.  This is likely to fail when the upstream repositories
>> submodules are cloned from a repository that is not prepared to
>> serve histories that ends at a commit that is not at the tip of a
>> branch, and we know the world is not yet ready.
>>
>> Use a safer default to clone the submodules fully, unless the user
>> tells us that she knows that the upstream repository of the
>> submodules are willing to cooperate with "--shallow-submodules"
>> option.
>>
>> Noticed-by: Vadim Eisenberg <VADIME@il.ibm.com>
>> Helped-by: Jeff King <peff@peff.net>
>> Signed-off-by: Junio C Hamano <gitster@pobox.com>
>> ---
>>  Documentation/git-clone.txt | 5 ++---
>>  builtin/clone.c             | 5 ++---
>>  t/t5614-clone-submodules.sh | 4 ++--
>>  3 files changed, 6 insertions(+), 8 deletions(-)
>>
>> diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
>> index e1a21b7..c5a1ce2 100644
>> --- a/Documentation/git-clone.txt
>> +++ b/Documentation/git-clone.txt
>> @@ -192,9 +192,8 @@ objects from the source repository into a pack in the cloned repository.
>>         Create a 'shallow' clone with a history truncated to the
>>         specified number of revisions. Implies `--single-branch` unless
>>         `--no-single-branch` is given to fetch the histories near the
>> -       tips of all branches. This implies `--shallow-submodules`. If
>> -       you want to have a shallow superproject clone, but full submodules,
>> -       also pass `--no-shallow-submodules`.
>> +       tips of all branches. If you want to clone submodules shallowly,
>> +       also pass `--shallow-submodules`.
>>
>>  --[no-]single-branch::
>>         Clone only the history leading to the tip of a single branch,
>> diff --git a/builtin/clone.c b/builtin/clone.c
>> index ecdf308..f267742 100644
>> --- a/builtin/clone.c
>> +++ b/builtin/clone.c
>> @@ -40,7 +40,7 @@ static const char * const builtin_clone_usage[] = {
>>
>>  static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
>>  static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;
>> -static int option_shallow_submodules = -1;
>> +static int option_shallow_submodules;
>>  static char *option_template, *option_depth;
>>  static char *option_origin = NULL;
>>  static char *option_branch = NULL;
>> @@ -730,8 +730,7 @@ static int checkout(void)
>>                 struct argv_array args = ARGV_ARRAY_INIT;
>>                 argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
>>
>> -               if (option_shallow_submodules == 1
>> -                   || (option_shallow_submodules == -1 && option_depth))
>> +               if (option_shallow_submodules == 1)
>>                         argv_array_push(&args, "--depth=1");
>
> Very late reply, since I'm just looking at this now with the --no-tags
> opti,n, but that == 1 makes no sense anymore, and should just be `if
> (option_shallow_submodules)` shouldn't it? I.e. this used to be an int
> for the depth, now is a bool, but the current == 1 check is left over
> probably from an earlier version where the depth was configurable.
Yes we can drop the "== 1" here.

Thanks, Stefan

Duy Nguyen· Jun 21, 2016, 16:48 UTC · re: Jeff King · lore

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

On Sun, Jun 19, 2016 at 12:00 PM, Jeff King <peff@peff.net> wrote:
Show 6 quoted lines
> Stefan, I think it might be worth revisiting the default set by d22eb04
> to propagate shallowness from the super-project clone. In an ideal
> world, we would be asking each submodule for the actual commit we are
> interested in, and shallowness would not matter. But until
> uploadpack.allowReachableSHA1InWant works everywhere, I suspect this is
> going to be a problem.

Maybe we can pass an option to subsequent clones that say "if allow-...-sha1-in-want is advertised, do a shallow clone, otherwise fall back to full clone"?

-- 
Duy

← back to recent threads