threads / patch / 61716

patch, 2 partsclone: shallow-submodules should be single-branch by default

Subject: [PATCH 0/2] clone: shallow-submodules should be single-branch by default

## tl;dr

7 messages between Jul 2, 2024 and Jul 25, 2024. Diffs are folded; open one to read it.

replies: 6people: 2as markdown or json

Bruce Perry via GitGitGadget· Jul 2, 2024, 19:07 UTC · lore

I noticed a couple places where the behavior of recursive clones for shallow submodules does not match what is implied by the documentation. Shallow submodules should be, but aren't, single branch by default. It would also be useful to allow users to override the shallow specification in gitmodules on the command line for clones as they can for submodule update. The modification here for the former is a bit ugly, but hopefully at least gets the point across to start a discussion. First time submitting a patch here, hopefully I'm getting the process right.

Bruce Perry (2):
  clone: shallow-submodules should be single-branch by default
  clone: no-shallow-submodules clone overrides option in gitmodules
 Documentation/git-clone.txt         |  3 ++
 Documentation/gitmodules.txt        |  4 +--
 builtin/clone.c                     | 10 ++++--
 t/t5614-clone-submodules-shallow.sh | 52 +++++++++++++++++++++++------
 4 files changed, 53 insertions(+), 16 deletions(-)
base-commit: daed0c68e94967bfbb3f87e15f7c9090dc1aa1e1
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1740%2Fbaperry2%2Fsubmods-clone-bug-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1740/baperry2/submods-clone-bug-v1
Pull-Request: https://github.com/git/git/pull/1740
-- 
gitgitgadget
Bruce Perry via GitGitGadget· Jul 2, 2024, 19:07 UTC · re: Bruce Perry via GitGitGadget · lore

[PATCH 1/2] clone: shallow-submodules should be single-branch by default

From: Bruce Perry <bruce.perry@nrel.gov>

Recursive cloning with with the `--shallow-submodules` command line option (or the `shallow = true` option in .gitmodules) and no specification of `--[no-]single-branch` does result in a shallow submodule clone, but obtains all available branches. This behavior is unexpected because in other circumstances shallow clones imply single-branch clones. It also violates the documented behavior: the documentation states that performing a clone with `--recurse-submodules` is equivalent to doing a nonrecursive clone then immediately running `git submodule update --init --recursive`. However, with the `shallow = true` option in .gitmodules, the former results in a shallow but not single-branch clone of the submodules, while the latter results submodules that are both shallow and single-branch.

Modify the logic for git clone with `--recurse-submodules` so that if no option is specified for `--[no-]single-branch`, then no `--[no-]single-branch` option is passed to the internal call to `git submodule update --init --recursive`. As a result, the default will be used, which is to make shallow clones also single-branch and non-shallow clones multi-branch.

Modify the tests for shallow-submodules so that the submodule has multiple branches and the expected behavior in terms of obtaining branches is validated: shallow submodules must also be single-branch, while non-shallow submodules must obtain both branches.

Make a slight clarification to the documentation regarding the above behavior, although primarily the effect is to bring the behavior in line with the present documentation.

Signed-off-by: Bruce Perry <bruce.perry@nrel.gov>
---
 Documentation/git-clone.txt         |  3 +++
 Documentation/gitmodules.txt        |  4 ++--
 builtin/clone.c                     |  6 +++--
 t/t5614-clone-submodules-shallow.sh | 36 ++++++++++++++++++++---------
 4 files changed, 34 insertions(+), 15 deletions(-)
Show changes to 4 files +34 −15

Documentation/git-clone.txt, Documentation/gitmodules.txt, builtin/clone.c, t/t5614-clone-submodules-shallow.sh

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 5de18de2ab8..69248421390 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -297,6 +297,9 @@ or `--mirror` is given)
 
 `--`[`no-`]`shallow-submodules`::
 	All submodules which are cloned will be shallow with a depth of 1.
+	Also implies `--single-branch` for the submodule clones
+	unless `--no-single-branch` is passed, in which case histories near
+	the tips of all branches of the cloned submodules are fetched.
 
 `--`[`no-`]`remote-submodules`::
 	All submodules which are cloned will use the status of the submodule's
diff --git a/Documentation/gitmodules.txt b/Documentation/gitmodules.txt
index d9bec8b1875..c8160cd1df7 100644
--- a/Documentation/gitmodules.txt
+++ b/Documentation/gitmodules.txt
@@ -95,8 +95,8 @@ affected by this setting.
 
 submodule.<name>.shallow::
 	When set to true, a clone of this submodule will be performed as a
-	shallow clone (with a history depth of 1) unless the user explicitly
-	asks for a non-shallow clone.
+	shallow clone (single branch with a history depth of 1) unless the
+	user explicitly asks for a non-shallow clone.
 
 NOTES
 -----
diff --git a/builtin/clone.c b/builtin/clone.c
index b28f88eb43d..50ccce8902d 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -58,6 +58,7 @@ static const char * const builtin_clone_usage[] = {
 };
 
 static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
+static int option_single_branch_submodules;
 static int option_local = -1, option_no_hardlinks, option_shared;
 static int option_no_tags;
 static int option_shallow_submodules;
@@ -816,8 +817,8 @@ static int checkout(int submodule_progress, int filter_submodules)
 			strvec_pushf(&cmd.args, "--filter=%s",
 				     expand_list_objects_filter_spec(&filter_options));
 
-		if (option_single_branch >= 0)
-			strvec_push(&cmd.args, option_single_branch ?
+		if (option_single_branch_submodules >= 0)
+			strvec_push(&cmd.args, option_single_branch_submodules ?
 					       "--single-branch" :
 					       "--no-single-branch");
 
@@ -997,6 +998,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 
 	if (option_depth || option_since || option_not.nr)
 		deepen = 1;
+	option_single_branch_submodules = option_single_branch;
 	if (option_single_branch == -1)
 		option_single_branch = deepen ? 1 : 0;
 
diff --git a/t/t5614-clone-submodules-shallow.sh b/t/t5614-clone-submodules-shallow.sh
index c2a2bb453ee..b23c7d085aa 100755
--- a/t/t5614-clone-submodules-shallow.sh
+++ b/t/t5614-clone-submodules-shallow.sh
@@ -14,10 +14,14 @@ test_expect_success 'setup' '
 	mkdir sub &&
 	(
 		cd sub &&
-		git init &&
+		git init -b main &&
 		test_commit subcommit1 &&
 		test_commit subcommit2 &&
-		test_commit subcommit3
+		test_commit subcommit3 &&
+		git checkout -b branch &&
+		test_commit branchcommit1 &&
+		test_commit branchcommit2 &&
+		git checkout main
 	) &&
 	git submodule add "file://$pwd/sub" sub &&
 	git commit -m "add submodule"
@@ -30,16 +34,18 @@ test_expect_success 'nonshallow clone implies nonshallow submodule' '
 	git -C super_clone log --oneline >lines &&
 	test_line_count = 3 lines &&
 	git -C super_clone/sub log --oneline >lines &&
-	test_line_count = 3 lines
+	test_line_count = 3 lines &&
+	git -C super_clone/sub log --oneline --all >lines &&
+	test_line_count = 5 lines
 '
 
 test_expect_success 'shallow clone with shallow submodule' '
 	test_when_finished "rm -rf super_clone" &&
 	test_config_global protocol.file.allow always &&
 	git clone --recurse-submodules --depth 2 --shallow-submodules "file://$pwd/." super_clone &&
-	git -C super_clone log --oneline >lines &&
+	git -C super_clone log --oneline --all >lines &&
 	test_line_count = 2 lines &&
-	git -C super_clone/sub log --oneline >lines &&
+	git -C super_clone/sub log --oneline --all >lines &&
 	test_line_count = 1 lines
 '
 
@@ -50,7 +56,9 @@ test_expect_success 'shallow clone does not imply shallow submodule' '
 	git -C super_clone log --oneline >lines &&
 	test_line_count = 2 lines &&
 	git -C super_clone/sub log --oneline >lines &&
-	test_line_count = 3 lines
+	test_line_count = 3 lines  &&
+	git -C super_clone/sub log --oneline --all >lines &&
+	test_line_count = 5 lines
 '
 
 test_expect_success 'shallow clone with non shallow submodule' '
@@ -60,7 +68,9 @@ test_expect_success 'shallow clone with non shallow submodule' '
 	git -C super_clone log --oneline >lines &&
 	test_line_count = 2 lines &&
 	git -C super_clone/sub log --oneline >lines &&
-	test_line_count = 3 lines
+	test_line_count = 3 lines  &&
+	git -C super_clone/sub log --oneline --all >lines &&
+	test_line_count = 5 lines
 '
 
 test_expect_success 'non shallow clone with shallow submodule' '
@@ -69,7 +79,7 @@ test_expect_success 'non shallow clone with shallow submodule' '
 	git clone --recurse-submodules --no-local --shallow-submodules "file://$pwd/." super_clone &&
 	git -C super_clone log --oneline >lines &&
 	test_line_count = 3 lines &&
-	git -C super_clone/sub log --oneline >lines &&
+	git -C super_clone/sub log --oneline --all >lines &&
 	test_line_count = 1 lines
 '
 
@@ -87,7 +97,7 @@ test_expect_success 'clone follows shallow recommendation' '
 	) &&
 	(
 		cd super_clone/sub &&
-		git log --oneline >lines &&
+		git log --oneline --all >lines &&
 		test_line_count = 1 lines
 	)
 '
@@ -105,7 +115,9 @@ test_expect_success 'get unshallow recommended shallow submodule' '
 	(
 		cd super_clone/sub &&
 		git log --oneline >lines &&
-		test_line_count = 3 lines
+		test_line_count = 3 lines &&
+		git log --oneline --all >lines &&
+		test_line_count = 5 lines
 	)
 '
 
@@ -124,7 +136,9 @@ test_expect_success 'clone follows non shallow recommendation' '
 	(
 		cd super_clone/sub &&
 		git log --oneline >lines &&
-		test_line_count = 3 lines
+		test_line_count = 3 lines &&
+		git log --oneline --all >lines &&
+		test_line_count = 5 lines
 	)
 '
 
-- 
gitgitgadget
Bruce Perry via GitGitGadget· Jul 2, 2024, 19:07 UTC · re: Bruce Perry via GitGitGadget · lore

[PATCH 2/2] clone: no-shallow-submodules clone overrides option in gitmodules

From: Bruce Perry <bruce.perry@nrel.gov>

When the `--no-shallow-submodules` option is used for cloning, pass the `--no-recommend-shallow` option to the internal submodule update command, which will override an option for shallow in .gitmodules if present. This allows the user to force a non-shallow clone of the submodules if desired. It brings the behavior in line with the present documentation, which states the option in gitmodules is used unless explitcitly overridden by the user. Add a test to confirm that the override is done when using the clone command (similar to a current test that validates similar behavior for the submodule update command).

Signed-off-by: Bruce Perry <bruce.perry@nrel.gov>
---
 builtin/clone.c                     |  4 +++-
 t/t5614-clone-submodules-shallow.sh | 16 ++++++++++++++++
 2 files changed, 19 insertions(+), 1 deletion(-)
Show changes to 2 files +19 −1

builtin/clone.c, t/t5614-clone-submodules-shallow.sh

diff --git a/builtin/clone.c b/builtin/clone.c
index 50ccce8902d..63cfb48484c 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -61,7 +61,7 @@ static int option_no_checkout, option_bare, option_mirror, option_single_branch
 static int option_single_branch_submodules;
 static int option_local = -1, option_no_hardlinks, option_shared;
 static int option_no_tags;
-static int option_shallow_submodules;
+static int option_shallow_submodules = -1;
 static int option_reject_shallow = -1;    /* unspecified */
 static int config_reject_shallow = -1;    /* unspecified */
 static int deepen;
@@ -798,6 +798,8 @@ static int checkout(int submodule_progress, int filter_submodules)
 
 		if (option_shallow_submodules == 1)
 			strvec_push(&cmd.args, "--depth=1");
+		else if (option_shallow_submodules == 0)
+			strvec_push(&cmd.args, "--no-recommend-shallow");
 
 		if (max_jobs != -1)
 			strvec_pushf(&cmd.args, "--jobs=%d", max_jobs);
diff --git a/t/t5614-clone-submodules-shallow.sh b/t/t5614-clone-submodules-shallow.sh
index b23c7d085aa..8e2965248a2 100755
--- a/t/t5614-clone-submodules-shallow.sh
+++ b/t/t5614-clone-submodules-shallow.sh
@@ -102,6 +102,22 @@ test_expect_success 'clone follows shallow recommendation' '
 	)
 '
 
+test_expect_success 'no-shallow-submodules clone option overrides gitmodules' '
+	test_when_finished "rm -rf super_clone" &&
+	test_config_global protocol.file.allow always &&
+	git clone --recurse-submodules --no-shallow-submodules --no-local "file://$pwd/." super_clone &&
+	(
+		cd super_clone &&
+		git log --oneline >lines &&
+		test_line_count = 4 lines
+	) &&
+	(
+		cd super_clone/sub &&
+		git log --oneline --all >lines &&
+		test_line_count = 5 lines
+	)
+'
+
 test_expect_success 'get unshallow recommended shallow submodule' '
 	test_when_finished "rm -rf super_clone" &&
 	test_config_global protocol.file.allow always &&
-- 
gitgitgadget
rsbecker@nexbridge.com· Jul 2, 2024, 19:12 UTC · re: Bruce Perry via GitGitGadget · lore

RE: [PATCH 0/2] clone: shallow-submodules should be single-branch by default

On Tuesday, July 2, 2024 3:08 PM, Bruce Perry wrote:
Show 25 quoted lines
>I noticed a couple places where the behavior of recursive clones for shallow
>submodules does not match what is implied by the documentation. Shallow
>submodules should be, but aren't, single branch by default. It would also be useful
>to allow users to override the shallow specification in gitmodules on the command
>line for clones as they can for submodule update. The modification here for the
>former is a bit ugly, but hopefully at least gets the point across to start a discussion.
>First time submitting a patch here, hopefully I'm getting the process right.
>
>Bruce Perry (2):
>  clone: shallow-submodules should be single-branch by default
>  clone: no-shallow-submodules clone overrides option in gitmodules
>
> Documentation/git-clone.txt         |  3 ++
> Documentation/gitmodules.txt        |  4 +--
> builtin/clone.c                     | 10 ++++--
> t/t5614-clone-submodules-shallow.sh | 52 +++++++++++++++++++++++------
> 4 files changed, 53 insertions(+), 16 deletions(-)
>
>
>base-commit: daed0c68e94967bfbb3f87e15f7c9090dc1aa1e1
>Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-
>1740%2Fbaperry2%2Fsubmods-clone-bug-v1
>Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-
>1740/baperry2/submods-clone-bug-v1
>Pull-Request: https://github.com/git/git/pull/1740

I am concerned about this one. Many CI systems (including Jenkins GitSCM) assume a detached head for submodule clone/checkout. Adding a branch to the mix will change the expected semantics. Am I missing something? --Randall

Bruce Perry· Jul 3, 2024, 05:00 UTC · re: rsbecker@nexbridge.com · lore

Re: [PATCH 0/2] clone: shallow-submodules should be single-branch by default

Randall,

Perhaps I was using imprecise terminology. This change should not impact whether submodule clones land in a detached head state, so it should not impact anything that assumes submodule clones are detached head.

The change being made is this: "git clone --recurse-submodules --shallow-submodules" currently gives you a submodule with a detached head at the desired state, but also downloads data for the tips of all branches in the remote being cloned (potentially a lot of unneeded data as in my use case). The modification means the same command would give you a detached head at the desired state, plus the tip of only the default branch in the remote. The modified behavior matches the current behavior for a simple "git clone" followed by "git submodule update --init --recurse--submodules --depth=1".

Thanks, Bruce

(Resent due to a formatting failure)
On Tue, Jul 2, 2024, 1:12 PM <rsbecker@nexbridge.com> wrote:
Show 31 quoted lines
>
> On Tuesday, July 2, 2024 3:08 PM, Bruce Perry wrote:
> >I noticed a couple places where the behavior of recursive clones for shallow
> >submodules does not match what is implied by the documentation. Shallow
> >submodules should be, but aren't, single branch by default. It would also be useful
> >to allow users to override the shallow specification in gitmodules on the command
> >line for clones as they can for submodule update. The modification here for the
> >former is a bit ugly, but hopefully at least gets the point across to start a discussion.
> >First time submitting a patch here, hopefully I'm getting the process right.
> >
> >Bruce Perry (2):
> >  clone: shallow-submodules should be single-branch by default
> >  clone: no-shallow-submodules clone overrides option in gitmodules
> >
> > Documentation/git-clone.txt         |  3 ++
> > Documentation/gitmodules.txt        |  4 +--
> > builtin/clone.c                     | 10 ++++--
> > t/t5614-clone-submodules-shallow.sh | 52 +++++++++++++++++++++++------
> > 4 files changed, 53 insertions(+), 16 deletions(-)
> >
> >
> >base-commit: daed0c68e94967bfbb3f87e15f7c9090dc1aa1e1
> >Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-
> >1740%2Fbaperry2%2Fsubmods-clone-bug-v1
> >Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-
> >1740/baperry2/submods-clone-bug-v1
> >Pull-Request: https://github.com/git/git/pull/1740
>
> I am concerned about this one. Many CI systems (including Jenkins GitSCM) assume a detached head for submodule clone/checkout. Adding a branch to the mix will change the expected semantics. Am I missing something?
> --Randall
>
rsbecker@nexbridge.com· Jul 3, 2024, 14:05 UTC · re: Bruce Perry · lore

RE: [PATCH 0/2] clone: shallow-submodules should be single-branch by default

On Wednesday, July 3, 2024 1:00 AM, Bruce Perry wrote:
Show 53 quoted lines
>Perhaps I was using imprecise terminology. This change should not impact whether
>submodule clones land in a detached head state, so it should not impact anything
>that assumes submodule clones are detached head.
>
>The change being made is this: "git clone --recurse-submodules --shallow-
>submodules" currently gives you a submodule with a detached head at the desired
>state, but also downloads data for the tips of all branches in the remote being
>cloned (potentially a lot of unneeded data as in my use case). The modification
>means the same command would give you a detached head at the desired state,
>plus the tip of only the default branch in the remote. The modified behavior
>matches the current behavior for a simple "git clone" followed by "git submodule
>update --init --recurse--submodules --depth=1".
>
>Thanks,
>Bruce
>
>(Resent due to a formatting failure)
>
>On Tue, Jul 2, 2024, 1:12 PM <rsbecker@nexbridge.com> wrote:
>>
>> On Tuesday, July 2, 2024 3:08 PM, Bruce Perry wrote:
>> >I noticed a couple places where the behavior of recursive clones for
>> >shallow submodules does not match what is implied by the
>> >documentation. Shallow submodules should be, but aren't, single
>> >branch by default. It would also be useful to allow users to override
>> >the shallow specification in gitmodules on the command line for
>> >clones as they can for submodule update. The modification here for the former is
>a bit ugly, but hopefully at least gets the point across to start a discussion.
>> >First time submitting a patch here, hopefully I'm getting the process right.
>> >
>> >Bruce Perry (2):
>> >  clone: shallow-submodules should be single-branch by default
>> >  clone: no-shallow-submodules clone overrides option in gitmodules
>> >
>> > Documentation/git-clone.txt         |  3 ++
>> > Documentation/gitmodules.txt        |  4 +--
>> > builtin/clone.c                     | 10 ++++--
>> > t/t5614-clone-submodules-shallow.sh | 52
>> > +++++++++++++++++++++++------
>> > 4 files changed, 53 insertions(+), 16 deletions(-)
>> >
>> >
>> >base-commit: daed0c68e94967bfbb3f87e15f7c9090dc1aa1e1
>> >Published-As:
>> >https://github.com/gitgitgadget/git/releases/tag/pr-git-
>> >1740%2Fbaperry2%2Fsubmods-clone-bug-v1
>> >Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-
>> >1740/baperry2/submods-clone-bug-v1
>> >Pull-Request: https://github.com/git/git/pull/1740
>>
>> I am concerned about this one. Many CI systems (including Jenkins GitSCM)
>assume a detached head for submodule clone/checkout. Adding a branch to the mix
>will change the expected semantics. Am I missing something?

Thanks for clearing that up. Fine with me. Regards Randall

Bruce Perry· Jul 25, 2024, 17:10 UTC · re: rsbecker@nexbridge.com · lore

Re: [PATCH 0/2] clone: shallow-submodules should be single-branch by default

Any further comments on this or changes I should make so it can be merged?

The current code behavior does not match the documentation, so something should be changed. If these changes are not desired, I could submit an alternative patch that changes the documentation to clarify what currently happens instead.

On Wed, Jul 3, 2024 at 8:05 AM <rsbecker@nexbridge.com> wrote:
Show 60 quoted lines
>
> On Wednesday, July 3, 2024 1:00 AM, Bruce Perry wrote:
> >Perhaps I was using imprecise terminology. This change should not impact whether
> >submodule clones land in a detached head state, so it should not impact anything
> >that assumes submodule clones are detached head.
> >
> >The change being made is this: "git clone --recurse-submodules --shallow-
> >submodules" currently gives you a submodule with a detached head at the desired
> >state, but also downloads data for the tips of all branches in the remote being
> >cloned (potentially a lot of unneeded data as in my use case). The modification
> >means the same command would give you a detached head at the desired state,
> >plus the tip of only the default branch in the remote. The modified behavior
> >matches the current behavior for a simple "git clone" followed by "git submodule
> >update --init --recurse--submodules --depth=1".
> >
> >Thanks,
> >Bruce
> >
> >(Resent due to a formatting failure)
> >
> >On Tue, Jul 2, 2024, 1:12 PM <rsbecker@nexbridge.com> wrote:
> >>
> >> On Tuesday, July 2, 2024 3:08 PM, Bruce Perry wrote:
> >> >I noticed a couple places where the behavior of recursive clones for
> >> >shallow submodules does not match what is implied by the
> >> >documentation. Shallow submodules should be, but aren't, single
> >> >branch by default. It would also be useful to allow users to override
> >> >the shallow specification in gitmodules on the command line for
> >> >clones as they can for submodule update. The modification here for the former is
> >a bit ugly, but hopefully at least gets the point across to start a discussion.
> >> >First time submitting a patch here, hopefully I'm getting the process right.
> >> >
> >> >Bruce Perry (2):
> >> >  clone: shallow-submodules should be single-branch by default
> >> >  clone: no-shallow-submodules clone overrides option in gitmodules
> >> >
> >> > Documentation/git-clone.txt         |  3 ++
> >> > Documentation/gitmodules.txt        |  4 +--
> >> > builtin/clone.c                     | 10 ++++--
> >> > t/t5614-clone-submodules-shallow.sh | 52
> >> > +++++++++++++++++++++++------
> >> > 4 files changed, 53 insertions(+), 16 deletions(-)
> >> >
> >> >
> >> >base-commit: daed0c68e94967bfbb3f87e15f7c9090dc1aa1e1
> >> >Published-As:
> >> >https://github.com/gitgitgadget/git/releases/tag/pr-git-
> >> >1740%2Fbaperry2%2Fsubmods-clone-bug-v1
> >> >Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-
> >> >1740/baperry2/submods-clone-bug-v1
> >> >Pull-Request: https://github.com/git/git/pull/1740
> >>
> >> I am concerned about this one. Many CI systems (including Jenkins GitSCM)
> >assume a detached head for submodule clone/checkout. Adding a branch to the mix
> >will change the expected semantics. Am I missing something?
>
> Thanks for clearing that up. Fine with me.
> Regards
> Randall
>

← back to recent threads