Re: [PATCHv2 0/2] Submodule shallow recommendation [WAS: Submodules: have a depth field in the .gitmodules file]

From: Stefan Beller <sbeller@google.com>
Date: May 26, 2016, 18:54 UTC
Message-ID: <CAGZ79kZKSC-vdUXg0uzb_u022TNVY-JHXuXLAc2LOT7OvpYckQ@mail.gmail.com>
In-Reply-To: <xmqq1t4owuue.fsf@gitster.mtv.corp.google.com>

On Thu, May 26, 2016 at 11:13 AM, Junio C Hamano <gitster@pobox.com> wrote:

Show 13 quoted lines

> Stefan Beller <sbeller@google.com> writes:
>
>> Sometimes the history of a submodule is not considered important by
>> the projects upstream. To make it easier for downstream users, allow
>> a field 'submodule.<name>.depth' in .gitmodules, which can be used
>> to indicate the recommended depth.
>
> I have a design-level question.
>
> If your project consists of 600 submodules, among which 40 of them
> you would recommend making a shallow clone, are there traits, other
> than "most people would not care about its history", that are shared
> across these 40 subprojects?

>From my understanding these 40 subprojects are a large file storage

done different than Git LFS. In the repo world this was choosen to be a separate repository, such that you had versioning available as the large files change a few times (like a precompiled kernel for special hardware, etc). And this is one of the missing pieces to translate the current repo-tool workflow to submodules.

Show 19 quoted lines

>
> What I am trying to get at is that after adding .shallow annotation
> to these 40 submodules in .gitmodules, the project may need to add a
> different annotation for the same 40 submodules to control another
> operation.  Which would be cumbersome, and a level of redirection
> might be a good way to solve it.
>
> IIRC, earlier you had talked about a vision where you can just say
> 'submodule init --group=framework' to prepare your top-level project
> tree with its submodules in a state suitable to work on 'the
> framework part of the project', and the 'app' folks can substitute
> 'framework' with 'app' in that command.  I thought the earlier
> defaultGroup work (and the attribute limited pathspec work that lays
> groundwork for it) was part of that effort.
>
> Perhaps when a user says "submodule init --group=framework", that
> "framework" can choose some "developer profile" that indirectly
> specifies things like which group of submodules to initialize, which
> group of submodules can be shallow, etc.?

So you are proposing another layer of indirection, i.e. instead of giving a pathspec with ":(attr:label-framework)" we would want to give a profile which then has the pathspec plus additional information on shallowness an such.

Show 9 quoted lines

>
> Just a strawman (without worrying about details and expressiveness
> of the syntax), I am wondering if you want something like this in
> your .gitmodules:
>
>     [profile "framework"]
>         initialize = $submodule_spec
>         shallow = $submodule_spec
>         ...

There could be more operations here, like update strategies.

>
>     [submodule "kernel"]
>         url = ...
>         path = ...

instead here we could also put a (no-)init recommendation additionally to the shallow recommendation.

Show 9 quoted lines

>
>     ...
>
> where $submodule_spec would be a way to choose modules by various
> means.  You may name them by their names.  You may name them by
> their paths.  With the submodule-pathspec topic graduated, you may
> use ":(attr:framework)*" to choose them by attribute limited
> pathspec.
>

And you reinvented submodule groups. ;) IIRC we had a discussion if we want to have the submodule groups stored at each submodule or at a central "profile/group" setting. The advantage of putting the setting to each submodule is that it is easier to organize, i.e. it produces better locality for the settings.

It is easier to know for the [submodule "kernel"] what to set for flags or labels when looking at that submodule as you're likely to have knowledge about it. However adding a new group/profile is cumbersome, so we'd want a

    git config --add multi-set
submodule.<pathspec-matching-many-submodules>.label "initialize"

Another idea for having profiles would be to add conditional recommendations

     [submodule "kernel"]
         url = ...
         path = ...
         shallow = true if selected via :(attr:framework)

I have a worse feeling about these conditionals than about the profile, though

I think the profiles are selected using different repo manifest files, i.e. that would be different .gitmodules/.gitattributes files on different branches. However in each of these branches we would want to have a way to recommend which submodules to initialize/checkout/shallow and such.

If keeping these different settings in different branches, we may desire better
merge support for .gitattributes, though (adjacent lines do not influence each
other, like in source code. So if one branch had a change like
     submodule0 ...
    - /submodule1 label-foo
    + /submodule1 label-foo label-bar
     submodule2 ...

and andother branch had

     submodule1 label-foo
    - /submodule2 label-baz
    + /submodule2 label-baz label-bar
     submodule3 ...

we would not want to see the merge conflict as we would with todays merge strategy.)

Thanks, Stefan