[PATCH 0/1] checkout: teach "git checkout" to checkout submodule branches

This series is based on gc/branch-recurse-submodules.

Since gc/branch-recurse-submodules is queued for next, I was hoping to get some feedback on a followup feature - teaching "git checkout --recurse-submodules" to checkout the branches created by "git branch --recurse-submodules".

When "git branch --recurse-submodules" was introduced [1], we noted that existing Git commands ignore submodule branches and that we would teach them to respect submodule branches if submodule.propagateBranches=true.

This series teaches "git checkout" to understand submodule.propagateBranches=true so that "git checkout" checks out the branches created by "git branch --recurse-submodules". Specifically, the behavior of "git checkout" follows the proposal laid out in the submodule branching RFC [2] (an abridged description can also be found in the commit message).

This series isn't ready for full review yet (see "Future work"), but the approach is unusual enough that I thought it would be useful to get early feedback and evaluate alternatives (instead of polishing a potentially bad approach).

I'd appreciate any feedback, though I'd like for reviewers to focus on fleshing out the design space. This could include answering the questions in "Questions", or providing additional context to "git checkout" that I missed.

(Cc-ed some people who have shown some interest in submodules in the past)

= Approach

Presently, "git checkout" updates the superproject/submodule HEAD and trees using different machinery:

- unpack-trees updates the superproject tree, submodule tree and
  submodule HEAD
- cmd_checkout() porcelain (specifically, update_refs_for_switch()),
  updates the superproject HEAD

This works well if we assume that "git checkout" always leaves submodule's HEAD pointing to a commit id, but this is no longer true with submodule.propagateBranches=true, so this series updates submodules differently:

- disable submodule updating in unpack-trees, so that neither the
  submodule HEAD nor submodule tree are updated
- update the submodules' HEAD and tree using a "git checkout" subprocess
  in the cmd_checkout() porcelain (specifically,
  update_refs_for_switch())

Doing work in the cmd_checkout() porcelain lets us support complicated, "git checkout" workflows that don't make sense in unpack-trees:

- updating the submodule's tree using the submodule's "topic" branch
  instead of the gitlink of the superproject's "topic" branch
- updating the submodule's HEAD to a ref instead of a commit id
- checking out the gitlink when the submodule's HEAD points to a branch
  pointing to the same commit (unpack-trees will ignore the submodule if
  it notices that the HEAD's oid does not change)
- preventing a checkout if a submodule is in detached HEAD and not
  pointing to the superproject gitlink (Future work, see RFC for more
  details [2])

The tradeoffs are:

- we lose atomicity guarantees because we run multiple subprocesses
  instead of a single unpack_trees()
- overhead of spawning child processes
- blurring of the lines between cmd_checkout() porcelain and
  unpack-trees - superproject trees and HEAD are still updated
  in different code locations (by cmd_checkout() porcelain and
  unpack-trees respectively), but submodule trees and HEAD are both
  updated by a single "git checkout" subprocess.

= Questions

- Is this an accurate summary of benefits and tradeoffs? Am I missing
  something?
- What alternatives exist and how viable are they?

= Future work

- Prevent a submodule checkout when the submodule's HEAD is detached and
  not pointing to the superproject gitlink
- Implement the checkout -b/-B flags i.e. create a new branch
  recursively and check it out.
- Refactor out the common bits between this series and "git branch
  --recurse-submodules".
- Check for checkout options that are incompatible with
  --recurse-submodules
- Parallelize the submodule checkout
- Handle the submodule.recurse config value
- Possibly refactor out the "find submodules in the index" logic from
  builtin/submodule--helper.

[1] 961b130d20 (branch: add --recurse-submodules option for branch creation, 2022-01-28) [2] https://lore.kernel.org/git/kl6lh7cjvpm3.fsf@chooglen-macbookpro.roam.corp.google.com

Glen Choo (1):
  checkout --recurse-submodules: checkout branches if configured

 builtin/checkout.c            | 142 +++++++++++++++++++++++++++++++++-
 submodule.c                   |   6 ++
 submodule.h                   |   9 +++
 t/t2013-checkout-submodule.sh |  90 +++++++++++++++++++++
 4 files changed, 243 insertions(+), 4 deletions(-)

base-commit: 679e3693aba0c17af60c031f7eef68f2296b8dad