[Summit topic] Submodules and how to make them worth using

This session was led by Emily Shaffer. Supporting cast: brian m. carlson, Orgad Shaneh, Jonathan "jrnieder" Nieder, Jeff Hostetler, and Philip Oakley.

Notes:

 1. https://lore.kernel.org/git/YHofmWcIAidkvJiD@google.com/

    1. Internally at Google, a lot of use of “repo”

    2. Isn’t great, but not much alternative available

    3. Submodules are also not great, let’s make them better

    4. Some prior work: --recurse-submodules options

       1. I can run “git branch” with and without --recurse-submodules

    5. Being in recurse mode gives us a chance to be opinionated

    6. Don’t want to have a million options and create a lot of complexity

    7. Branches

       1. Superproject thinks “main” points to one set of states in submodules

       2. Submodules have “main” pointing elsewhere

       3. Which is right? The superproject is right, “git status” can show the
          difference

    8. Not trying to eliminate all complexity. There is some inherent
       complexity in stitching repositories together. But I want to make it
       predictable

    9. For specifics, see the RFC linked to above

 2. brian: Interested in current status, what’s been implemented

    1. Emily: workflow git clone / git branch / git commit / git push, all
       using submodule.recurse, worked well

    2. Intern Mahi Kolla sent a patch to recurse by default once you’ve done a
       --recurse-submodules clone

    3. Ran demo for an internal team, feedback was positive

    4. Used a hacky remote helper to map “git push” to “git push origin
       HEAD:refs/for/main”, we have plans for not needing that :)

    5. Partial clone with submodules is close to done, is another important
       part of this

    6. Glen and Josh have done some work on branching + setting tracking info.
       That’s key for making recursive push work in an intuitive way, because
       the branch you want to push to in each submodule is not always the same

    7. I also pushed a series storing a path in each submodule’s git directory
       to its superproject’s git directory. Use that as another phase in config
       parsing, inherited-from-superproject config. That combines well with
       config-based hooks (thanks Ævar for the help with that)

    8. Next steps are around fast-forward merges and rebases

    9. Specifics are in the doc linked to

 3. Interaction with Gerrit

    1. Orgad: when you push to a submodule and superproject, at merge time the
       submodule commit changes, what do you do in the superproject to handle
       this?

    2. jrnieder: This comes up in any review flow, not just Gerrit --- ideally
       you’d want to review the superproject and submodule changes together as
       one unit. There’s some work happening in Gerrit on “multi-change
       review”.

    3. What works today: Gerrit’s submodule subscription feature has the
       ability to update a superproject. If you have a set of submodule changes
       and a superproject change that are submitted together, then at submit
       time Gerrit will rewrite the superproject change to reflect what
       happened in the submodules.

    4. In the Android workflow the superproject only contains pointers to
       submodules so we don’t push changes for review to the superproject at
       all. So we handle this with submodule subscription.

    5. Emily: analogy to auto-generated merge commits

 4. Jeff Hostetler: back in 2014 Microsoft considered submodules, hit a can of
    worms

    1. Coordinating changes between submodule and superproject, this requires
       server-side locks to prevent edge cases

    2. Was hard enough that we abandoned it

    3. jrnieder: we’re viewing submodules as not a replacement for the
       monorepo, but as a separate thing for when components have an
       independent existence. Microsoft made the right choice by not using
       submodules artificially in the creation of the Windows monorepo.

 5. Jeff: do you want to support sub-sub-sub-submodules?

    1. Emily: we ruled that out.

    2. jrnieder: nested submodules already work well in Git, we’re not breaking
       that

       1. Philip Oakley: good; if that changes, please make docs + config clear
          about it

    3. As a matter of project hygiene, we encourage people to put their
       submodules in the top-level directly. That way, you know what code
       you’re pulling in.

    4. That said, there are unusual use cases e.g. around a build that pulls
       together multiple versions of the full Android codebase. So we actually
       do take advantage of nested submodules for those niche cases

 6. Please read the design doc, and expect lotsa patches over the next 3-6
    months