Re: [PATCHv2] builtin/clone: support submodule groups

From: Stefan Beller <sbeller@google.com>
Date: Nov 30, 2015, 19:31 UTC
Message-ID: <CAGZ79kbUktcGNw4C123dxGoUsi=W+h4vUPWmBm2rExipUOcXqA@mail.gmail.com>
In-Reply-To: <20151126045929.GA29107@tsaunders-iceball.corp.tor1.mozilla.com>

+ cc Duy, Michael, who discussed the sparse checkout recently

On Wed, Nov 25, 2015 at 9:00 PM, Trevor Saunders <tbsaunde@tbsaunde.org> wrote:

Show 8 quoted lines

> Seeing the recent sparse checkout discussion I realized it might be
> useful to have a similar sort of feature for sparse checkouts.  So say I
> had a mobile and desktop client in the same repo and wanted to be able to
> checkout the commoncode and one client without having to explicitly list
> all the paths I care about. It seems like UI wise you might want to use
> --group there too, or at least explaining the difference to users might
> be interesting, but maybe that's worrying way too much abouta possible
> future feature.

For reference what I was proposing (coverletter [RFC PATCH 0/5] Submodule Groups): -->8-- This is also available at https://github.com/stefanbeller/git/tree/submodule-groups It applies on top of the submodule-parallel-patch series I sent a few minutes ago.

Consider having a real large software project in Git with each component in a submodule (such as an operating system, Android, Debian, Fedora, no toy OS such as https://github.com/gittup/gittup as that doesn't quite demonstrate the scale of the problem).

If you have lots of submodules, you probably don't need all of them at once, but you have functional units. Some submodules are absolutely required, some are optional and only for very specific purposes.

This patch series adds meaning to a "groups" field in the .gitmodules file.

So you could have a .gitmodules file such as:

[submodule "gcc"]
        path = gcc
        url = git://...
        groups = default,devel
[submodule "linux"]
        path = linux
        url = git://...
        groups = default
[submodule "nethack"]
        path = nethack
        url = git://...
        groups = optional,games

and by this series you can work on an arbitrary subgroup of these submodules such using these commands:

    git clone --group default --group devel git://...
    # will clone the superproject and recursively
    # checkout any submodule being in at least one of the groups.

    git submodule add --group default --group devel git://... ..
    # will add a submodule, adding 2 submodule
    # groups to its entry in .gitmodule

    # as support for clone we want to have:
    git config submodule.groups default
    git submodule init --groups
    # will init all submodules from the default group

    # as support for clone we want to have:
    git config submodule.groups default
    git submodule update --groups
    # will update all submodules from the default group

Any feedback welcome, specially on the design level! (Do we want to have it stored in the .gitmodules file? Do we want to have the groups configured in .git/config as "submodule.groups", any other way to make it future proof and extend the groups syntax?) -->8--

I think the biggest advantage with the groups is to have it not depending on the path. Consider your one repository containing both mobile and desktop code, where you have a sparse checkout for mobile.

Now what happens if files are renamed, i.e. the leading directory? As this change comes in from your dear coworker, who has no idea how your .git/info/sparse-checkout looks like, it may happen that more files appear in your worktree as your patterns did not cover the renamed case. Or some file contents go missing as they are in one of the ignored paths.

This groups feature would solve that as the groups are not dependent on the paths or the data itself. However the groups in this proposal are only meant to be applied on a submodule level, not in a repository itself.

How would you do that? I could imagine a file like .gitgroups (It should be part of the repository, such that everybody talks about the same groups), with a similar syntax like .gitattributes where some file patterns are assigned one or more groups. And in either .git/config (as it is for the submodules here) or in a file .git/info/groups (as the sparse checkouts do it in that directory) you'd configure the groups you are interested in.

It would be cool to have the same mechanism for both sparse-group-checkout and submodules, so I'd propose to use a config option in .git/config, such as checkout-group which covers both submodules as well as the pattern groups as specified by .gitgroups.

--- Beware, this is just a first shot spinning around some ideas.

Thanks, Stefan