threads / rfc / 40771

[RFC] Clone repositories recursive with depth 1

Subject: [RFC] Clone repositories recursive with depth 1

## tl;dr

6 messages between Nov 11, 2015 and Nov 14, 2015.

replies: 5people: 3as markdown or json

Lars Schneider· Nov 11, 2015, 14:09 UTC · lore
Hi,
I have a clean build machine and I want to clone my source code to this machine while transferring only the minimal necessary amount of data. Therefore I use this command:
git clone --recursive --depth 1 --single-branch <url>
Apparently this does not clone the submodules with "--depth 1" (using Git 2.4.9). As a workaround I tried:

git clone --depth 1 --single-branch <url> cd <repo-name> git submodule update --init --recursive --depth 1

However, this does not work either as I get: fatal: reference is not a tree: <correct sha1 of the submodule referenced by the main project> Unable to checkout <correct sha1 of the submodule referenced by the main project> in submodule path <submodule path>

How would you clone the repo? Is the behavior above expected? If not, should the "--depth 1" flag be applied recursively to all submodules on a clone --recursive? Has a patch implementing this a chance to get in?

Thanks, Lars

Stefan Beller· Nov 11, 2015, 19:19 UTC · re: Lars Schneider · lore

Re: [RFC] Clone repositories recursive with depth 1

On Wed, Nov 11, 2015 at 6:09 AM, Lars Schneider <larsxschneider@gmail.com> wrote:

Show 5 quoted lines
> Hi,
>
> I have a clean build machine and I want to clone my source code to this machine while transferring only the minimal necessary amount of data. Therefore I use this command:
>
> git clone --recursive --depth 1 --single-branch <url>

That *should* work, actually. However looking at the code it does not.

citing from builtin/clone.c:
    static struct option builtin_clone_options[] = {
        ...
        OPT_BOOL(0, "recursive", &option_recursive,
           N_("initialize submodules in the clone")),
        OPT_BOOL(0, "recurse-submodules", &option_recursive,
          N_("initialize submodules in the clone")),
        ...
    };
    ...
    static const char *argv_submodule[] = {
        "submodule", "update", "--init", "--recursive", NULL
    };
    if (!err && option_recursive)
        err = run_command_v_opt(argv_submodule, RUN_GIT_CMD);

So the --depth argument is not passed on, although "git submodule update" definitely supports --depth.

In an upcoming series (next version of origin/sb/submodule-parallel-update), this will slightly change, such it will be even easier to add the depth argument in there as we construct the argument list in code instead of hard coding argv_submodule.

This may require some discussion whether you expect --depth to be recursed. (What if you only want a top level shallow thing?, What if you want to have only submodules shallow? What is the user expectation here?)

Show 10 quoted lines
>
> Apparently this does not clone the submodules with "--depth 1" (using Git 2.4.9). As a workaround I tried:
>
> git clone --depth 1 --single-branch <url>
> cd <repo-name>
> git submodule update --init --recursive --depth 1
>
> However, this does not work either as I get:
> fatal: reference is not a tree: <correct sha1 of the submodule referenced by the main project>
> Unable to checkout <correct sha1 of the submodule referenced by the main project> in submodule path <submodule path>
That seems like another bug to me.

I just tried to clone a project and populate with submodules later and it works as expected without these error messages. (I am running some kind of xxx.dirty development version, most likely origin/sb/submodule-parallel-update, I'll check some other versions, too)

Stefan Beller· Nov 11, 2015, 20:09 UTC · re: Stefan Beller · lore

Re: [RFC] Clone repositories recursive with depth 1

On Wed, Nov 11, 2015 at 11:19 AM, Stefan Beller <sbeller@google.com> wrote:
Show 49 quoted lines
> On Wed, Nov 11, 2015 at 6:09 AM, Lars Schneider
> <larsxschneider@gmail.com> wrote:
>> Hi,
>>
>> I have a clean build machine and I want to clone my source code to this machine while transferring only the minimal necessary amount of data. Therefore I use this command:
>>
>> git clone --recursive --depth 1 --single-branch <url>
>
> That *should* work, actually.
> However looking at the code it does not.
>
> citing from builtin/clone.c:
>
>     static struct option builtin_clone_options[] = {
>         ...
>         OPT_BOOL(0, "recursive", &option_recursive,
>            N_("initialize submodules in the clone")),
>         OPT_BOOL(0, "recurse-submodules", &option_recursive,
>           N_("initialize submodules in the clone")),
>         ...
>     };
>     ...
>     static const char *argv_submodule[] = {
>         "submodule", "update", "--init", "--recursive", NULL
>     };
>
>     if (!err && option_recursive)
>         err = run_command_v_opt(argv_submodule, RUN_GIT_CMD);
>
> So the --depth argument is not passed on, although "git submodule update"
> definitely supports --depth.
>
> In an upcoming series (next version of origin/sb/submodule-parallel-update),
> this will slightly change, such it will be even easier to add the
> depth argument in
> there as we construct the argument list in code instead of hard coding
> argv_submodule.
>
> This may require some discussion whether you expect --depth to be recursed.
> (What if you only want a top level shallow thing?, What if you want to have only
> submodules shallow? What is the user expectation here?)
>
>>
>> Apparently this does not clone the submodules with "--depth 1" (using Git 2.4.9). As a workaround I tried:
>>
>> git clone --depth 1 --single-branch <url>
>> cd <repo-name>
>> git submodule update --init --recursive --depth 1
>>
The workaround works with the origin/master version for me.

Notice the other email thread, which suggests to include --remote into the call to git submodule update depending on a branch config option being present in the .gitmodules file.

Lars Schneider· Nov 12, 2015, 09:39 UTC · re: Stefan Beller · lore

Re: [RFC] Clone repositories recursive with depth 1

On 11 Nov 2015, at 21:09, Stefan Beller <sbeller@google.com> wrote:
Show 56 quoted lines
> On Wed, Nov 11, 2015 at 11:19 AM, Stefan Beller <sbeller@google.com> wrote:
>> On Wed, Nov 11, 2015 at 6:09 AM, Lars Schneider
>> <larsxschneider@gmail.com> wrote:
>>> Hi,
>>> 
>>> I have a clean build machine and I want to clone my source code to this machine while transferring only the minimal necessary amount of data. Therefore I use this command:
>>> 
>>> git clone --recursive --depth 1 --single-branch <url>
>> 
>> That *should* work, actually.
>> However looking at the code it does not.
>> 
>> citing from builtin/clone.c:
>> 
>>    static struct option builtin_clone_options[] = {
>>        ...
>>        OPT_BOOL(0, "recursive", &option_recursive,
>>           N_("initialize submodules in the clone")),
>>        OPT_BOOL(0, "recurse-submodules", &option_recursive,
>>          N_("initialize submodules in the clone")),
>>        ...
>>    };
>>    ...
>>    static const char *argv_submodule[] = {
>>        "submodule", "update", "--init", "--recursive", NULL
>>    };
>> 
>>    if (!err && option_recursive)
>>        err = run_command_v_opt(argv_submodule, RUN_GIT_CMD);
>> 
>> So the --depth argument is not passed on, although "git submodule update"
>> definitely supports --depth.
>> 
>> In an upcoming series (next version of origin/sb/submodule-parallel-update),
>> this will slightly change, such it will be even easier to add the
>> depth argument in
>> there as we construct the argument list in code instead of hard coding
>> argv_submodule.
>> 
>> This may require some discussion whether you expect --depth to be recursed.
>> (What if you only want a top level shallow thing?, What if you want to have only
>> submodules shallow? What is the user expectation here?)
>> 
>>> 
>>> Apparently this does not clone the submodules with "--depth 1" (using Git 2.4.9). As a workaround I tried:
>>> 
>>> git clone --depth 1 --single-branch <url>
>>> cd <repo-name>
>>> git submodule update --init --recursive --depth 1
>>> 
> 
> The workaround works with the origin/master version for me.
> 
> Notice the other email thread, which suggests to include --remote into the
> call to  git submodule update depending on a branch config option being
> present in the .gitmodules file.

Can you check "[PATCH v2] add test to demonstrate that shallow recursive clones fail"? This demonstrates the failure that I see. I also tried the "--remote" flag but this does not work either (see test case). Can you confirm this behavior?

Cheers, Lars

Stefan Beller· Nov 12, 2015, 23:47 UTC · re: Lars Schneider · lore

Re: [RFC] Clone repositories recursive with depth 1

+cc Stanislav, who came up with the other thread for passing --remote to git submodule

On Thu, Nov 12, 2015 at 1:39 AM, Lars Schneider <larsxschneider@gmail.com> wrote:

Show 9 quoted lines
>> Notice the other email thread, which suggests to include --remote into the
>> call to  git submodule update depending on a branch config option being
>> present in the .gitmodules file.
>
> Can you check "[PATCH v2] add test to demonstrate that shallow recursive clones fail"? This demonstrates the failure that I see. I also tried the "--remote" flag but this does not work either (see test case).
> Can you confirm this behavior?
>
> Cheers,
> Lars
I can confirm it breaks as expected here.
I may have confused you here by pointing to the --remote option.

(git clone is a bit stupid when it comes to submodule handling.) All it does currently is this:

    if --recurseSubmodules option or --recursive option is given:
        run: "git submodule update --init --recursive"

No attention is paid to any other option such as --depth. That's all I wanted to point out there.

Ideally we want to add:
    If there is a branch configured in the .gitmodules file,
    we would want to add the --remote command
    if we have given other options such as --depth or --reference
    we want to pass that along to the called submodule helper.

So I was looking at the internal code structure and think one of the next series I am going to send will touch the code such that we can incorporate the conditions as outlined above easier, because it is not hardcoded into an array ["git", "submodule", "update" "--init", "--recursive"], as I want to add yet another dynamic option to the submodule helper invocation. (I want to add --jobs <n> there)

Cheers, Stefan

Fredrik Gustafsson· Nov 14, 2015, 16:25 UTC · re: Lars Schneider · lore

Re: [RFC] Clone repositories recursive with depth 1

On Wed, Nov 11, 2015 at 03:09:18PM +0100, Lars Schneider wrote:
Show 9 quoted lines
> Apparently this does not clone the submodules with "--depth 1" (using Git 2.4.9). As a workaround I tried:
> 
> git clone --depth 1 --single-branch <url>
> cd <repo-name>
> git submodule update --init --recursive --depth 1
> 
> However, this does not work either as I get:
> fatal: reference is not a tree: <correct sha1 of the submodule referenced by the main project>
> Unable to checkout <correct sha1 of the submodule referenced by the main project> in submodule path <submodule path>

This looks like a familiar bug to me. I'm not sure if it's a bug or a known behaviour. When the depth argument was introduced to submodules a year (or two) ago there was a know bug. I not sure if it's fixed or not.

The problem is that git is/was unable to fetch a sha1 but only a branch or a tag. So fetching a submodule will fetch the HEAD of the submodule with the requested depth. Then git will try to checkout a sha1 of that submodule, which may or maynot exists.

Say that you fetch master of a submodule of depth 1 and do a checkout of a commit that exists. When someonen else has pushed to that submodule, the commit will not longer be reachable from depth 1 and if someone else tries to clone with depth 1 it will fail with the same error message as you got.

The solution to this is to allow git to fetch the sha1 the superproject points to direct when fetching the submodule.

-- 
Fredrik Gustafsson

phone: +46 733-608274
e-mail: iveqy@iveqy.com
website: http://www.iveqy.com

← back to recent threads